May 15 23:15:20 client-18 syslogd 1.4.1: restart.
May 15 23:15:20 client-18 kernel: klogd 1.4.1, log source = /proc/kmsg started.
May 15 23:15:20 client-18 kernel: Linux version 2.6.18-194.17.1.el5 (mockbuild@builder10.centos.org) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-48)) #1 SMP Wed Sep 29 12:50:31 EDT 2010
May 15 23:15:20 client-18 kernel: Command line: ro root=LABEL=/ console=ttyS0,115200
May 15 23:15:20 client-18 kernel: BIOS-provided physical RAM map:
May 15 23:15:20 client-18 kernel:  BIOS-e820: 0000000000010000 - 000000000009cc00 (usable)
May 15 23:15:20 client-18 kernel:  BIOS-e820: 000000000009cc00 - 00000000000a0000 (reserved)
May 15 23:15:20 client-18 kernel:  BIOS-e820: 00000000000e6000 - 0000000000100000 (reserved)
May 15 23:15:20 client-18 kernel:  BIOS-e820: 0000000000100000 - 00000000bf760000 (usable)
May 15 23:15:20 client-18 kernel:  BIOS-e820: 00000000bf76e000 - 00000000bf770000 type 9
May 15 23:15:20 client-18 kernel:  BIOS-e820: 00000000bf770000 - 00000000bf77e000 (ACPI data)
May 15 23:15:20 client-18 kernel:  BIOS-e820: 00000000bf77e000 - 00000000bf7d0000 (ACPI NVS)
May 15 23:15:20 client-18 kernel:  BIOS-e820: 00000000bf7d0000 - 00000000bf7e0000 (reserved)
May 15 23:15:20 client-18 kernel:  BIOS-e820: 00000000bf7ec000 - 00000000c0000000 (reserved)
May 15 23:15:20 client-18 kernel:  BIOS-e820: 00000000e0000000 - 00000000f0000000 (reserved)
May 15 23:15:20 client-18 kernel:  BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
May 15 23:15:20 client-18 kernel:  BIOS-e820: 00000000ffc00000 - 0000000100000000 (reserved)
May 15 23:15:20 client-18 kernel:  BIOS-e820: 0000000100000000 - 0000000340000000 (usable)
May 15 23:15:20 client-18 kernel: DMI present.
May 15 23:15:20 client-18 kernel: No NUMA configuration found
May 15 23:15:20 client-18 kernel: Faking a node at 0000000000000000-0000000340000000
May 15 23:15:20 client-18 kernel: Bootmem setup node 0 0000000000000000-0000000340000000
May 15 23:15:20 client-18 kernel: Memory for crash kernel (0x0 to 0x0) notwithin permissible range
May 15 23:15:20 client-18 kernel: disabling kdump
May 15 23:15:20 client-18 kernel: ACPI: PM-Timer IO Port: 0x808
May 15 23:15:20 client-18 kernel: ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
May 15 23:15:20 client-18 kernel: Processor #0 7:10 APIC version 21
May 15 23:15:20 client-18 kernel: ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)
May 15 23:15:20 client-18 kernel: Processor #2 7:10 APIC version 21
May 15 23:15:20 client-18 kernel: ACPI: LAPIC (acpi_id[0x03] lapic_id[0x04] enabled)
May 15 23:15:20 client-18 kernel: Processor #4 7:10 APIC version 21
May 15 23:15:20 client-18 kernel: ACPI: LAPIC (acpi_id[0x04] lapic_id[0x06] enabled)
May 15 23:15:20 client-18 kernel: Processor #6 7:10 APIC version 21
May 15 23:15:20 client-18 kernel: ACPI: LAPIC (acpi_id[0x05] lapic_id[0x84] disabled)
May 15 23:15:20 client-18 kernel: ACPI: LAPIC (acpi_id[0x06] lapic_id[0x85] disabled)
May 15 23:15:20 client-18 kernel: ACPI: LAPIC (acpi_id[0x07] lapic_id[0x86] disabled)
May 15 23:15:20 client-18 kernel: ACPI: LAPIC (acpi_id[0x08] lapic_id[0x87] disabled)
May 15 23:15:20 client-18 kernel: ACPI: LAPIC (acpi_id[0x09] lapic_id[0x88] disabled)
May 15 23:15:20 client-18 kernel: ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x89] disabled)
May 15 23:15:20 client-18 kernel: ACPI: LAPIC (acpi_id[0x0b] lapic_id[0x8a] disabled)
May 15 23:15:20 client-18 kernel: ACPI: LAPIC (acpi_id[0x0c] lapic_id[0x8b] disabled)
May 15 23:15:20 client-18 kernel: ACPI: LAPIC (acpi_id[0x0d] lapic_id[0x8c] disabled)
May 15 23:15:20 client-18 kernel: ACPI: LAPIC (acpi_id[0x0e] lapic_id[0x8d] disabled)
May 15 23:15:20 client-18 kernel: ACPI: LAPIC (acpi_id[0x0f] lapic_id[0x8e] disabled)
May 15 23:15:20 client-18 kernel: ACPI: LAPIC (acpi_id[0x10] lapic_id[0x8f] disabled)
May 15 23:15:20 client-18 kernel: ACPI: LAPIC (acpi_id[0x11] lapic_id[0x90] disabled)
May 15 23:15:20 client-18 kernel: ACPI: LAPIC (acpi_id[0x12] lapic_id[0x91] disabled)
May 15 23:15:20 client-18 kernel: ACPI: LAPIC (acpi_id[0x13] lapic_id[0x92] disabled)
May 15 23:15:20 client-18 kernel: ACPI: LAPIC (acpi_id[0x14] lapic_id[0x93] disabled)
May 15 23:15:20 client-18 kernel: ACPI: LAPIC (acpi_id[0x15] lapic_id[0x94] disabled)
May 15 23:15:20 client-18 kernel: ACPI: LAPIC (acpi_id[0x16] lapic_id[0x95] disabled)
May 15 23:15:20 client-18 kernel: ACPI: LAPIC (acpi_id[0x17] lapic_id[0x96] disabled)
May 15 23:15:20 client-18 kernel: ACPI: LAPIC (acpi_id[0x18] lapic_id[0x97] disabled)
May 15 23:15:20 client-18 kernel: ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])
May 15 23:15:20 client-18 kernel: ACPI: IOAPIC (id[0x01] address[0xfec00000] gsi_base[0])
May 15 23:15:20 client-18 kernel: IOAPIC[0]: apic_id 1, version 32, address 0xfec00000, GSI 0-23
May 15 23:15:20 client-18 kernel: ACPI: IOAPIC (id[0x03] address[0xfec8a000] gsi_base[24])
May 15 23:15:20 client-18 kernel: IOAPIC[1]: apic_id 3, version 32, address 0xfec8a000, GSI 24-47
May 15 23:15:20 client-18 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
May 15 23:15:20 client-18 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
May 15 23:15:20 client-18 kernel: Setting APIC routing to physical flat
May 15 23:15:20 client-18 kernel: ACPI: HPET id: 0x8086a301 base: 0xfed00000
May 15 23:15:20 client-18 kernel: Using ACPI (MADT) for SMP configuration information
May 15 23:15:20 client-18 kernel: Nosave address range: 000000000009c000 - 000000000009d000
May 15 23:15:20 client-18 kernel: Nosave address range: 000000000009d000 - 00000000000a0000
May 15 23:15:20 client-18 kernel: Nosave address range: 00000000000a0000 - 00000000000e6000
May 15 23:15:20 client-18 kernel: Nosave address range: 00000000000e6000 - 0000000000100000
May 15 23:15:20 client-18 kernel: Nosave address range: 00000000bf760000 - 00000000bf76e000
May 15 23:15:20 client-18 kernel: Nosave address range: 00000000bf76e000 - 00000000bf770000
May 15 23:15:20 client-18 kernel: Nosave address range: 00000000bf770000 - 00000000bf77e000
May 15 23:15:20 client-18 kernel: Nosave address range: 00000000bf77e000 - 00000000bf7d0000
May 15 23:15:20 client-18 kernel: Nosave address range: 00000000bf7d0000 - 00000000bf7e0000
May 15 23:15:20 client-18 kernel: Nosave address range: 00000000bf7e0000 - 00000000bf7ec000
May 15 23:15:20 client-18 kernel: Nosave address range: 00000000bf7ec000 - 00000000c0000000
May 15 23:15:20 client-18 kernel: Nosave address range: 00000000c0000000 - 00000000e0000000
May 15 23:15:20 client-18 kernel: Nosave address range: 00000000e0000000 - 00000000f0000000
May 15 23:15:20 client-18 kernel: Nosave address range: 00000000f0000000 - 00000000fee00000
May 15 23:15:20 client-18 kernel: Nosave address range: 00000000fee00000 - 00000000fee01000
May 15 23:15:20 client-18 kernel: Nosave address range: 00000000fee01000 - 00000000ffc00000
May 15 23:15:20 client-18 kernel: Nosave address range: 00000000ffc00000 - 0000000100000000
May 15 23:15:20 client-18 kernel: Allocating PCI resources starting at c2000000 (gap: c0000000:20000000)
May 15 23:15:20 client-18 kernel: SMP: Allowing 24 CPUs, 20 hotplug CPUs
May 15 23:15:20 client-18 kernel: Built 1 zonelists.  Total pages: 3095502
May 15 23:15:20 client-18 kernel: Kernel command line: ro root=LABEL=/ console=ttyS0,115200
May 15 23:15:20 client-18 kernel: Initializing CPU#0
May 15 23:15:20 client-18 kernel: PID hash table entries: 4096 (order: 12, 32768 bytes)
May 15 23:15:20 client-18 kernel: Console: colour VGA+ 80x25
May 15 23:15:20 client-18 kernel: Dentry cache hash table entries: 2097152 (order: 12, 16777216 bytes)
May 15 23:15:20 client-18 kernel: Inode-cache hash table entries: 1048576 (order: 11, 8388608 bytes)
May 15 23:15:20 client-18 kernel: Checking aperture...
May 15 23:15:20 client-18 kernel: PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
May 15 23:15:20 client-18 kernel: Placing software IO TLB between 0x28e4000 - 0x68e4000
May 15 23:15:20 client-18 kernel: Memory: 12287648k/13631488k available (2574k kernel code, 285968k reserved, 1304k data, 212k init)
May 15 23:15:20 client-18 kernel: Calibrating delay loop (skipped), value calculated using timer frequency.. 4533.62 BogoMIPS (lpj=2266813)
May 15 23:15:20 client-18 kernel: Security Framework v1.0.0 initialized
May 15 23:15:20 client-18 kernel: SELinux:  Initializing.
May 15 23:15:20 client-18 kernel: selinux_register_security:  Registering secondary module capability
May 15 23:15:20 client-18 kernel: Capability LSM initialized as secondary
May 15 23:15:20 client-18 kernel: Mount-cache hash table entries: 256
May 15 23:15:20 client-18 kernel: CPU: L1 I cache: 32K, L1 D cache: 32K
May 15 23:15:20 client-18 kernel: CPU: L2 cache: 256K
May 15 23:15:20 client-18 kernel: CPU: L3 cache: 4096K
May 15 23:15:20 client-18 cpuspeed: Enabling ondemand cpu frequency scaling governor
May 15 23:15:20 client-18 kernel: using mwait in idle threads.
May 15 23:15:20 client-18 kernel: CPU: Physical Processor ID: 0
May 15 23:15:20 client-18 kernel: CPU: Processor Core ID: 0
May 15 23:15:20 client-18 kernel: CPU0: Thermal monitoring enabled (TM1)
May 15 23:15:20 client-18 kernel: SMP alternatives: switching to UP code
May 15 23:15:20 client-18 kernel: ACPI: Core revision 20060707
May 15 23:15:20 client-18 kernel: Using local APIC timer interrupts.
May 15 23:15:20 client-18 kernel: Detected 8.333 MHz APIC timer.
May 15 23:15:20 client-18 kernel: SMP alternatives: switching to SMP code
May 15 23:15:20 client-18 kernel: Booting processor 1/4 APIC 0x2
May 15 23:15:20 client-18 kernel: Initializing CPU#1
May 15 23:15:20 client-18 kernel: Calibrating delay using timer specific routine.. 4533.40 BogoMIPS (lpj=2266703)
May 15 23:15:20 client-18 kernel: CPU: L1 I cache: 32K, L1 D cache: 32K
May 15 23:15:20 client-18 kernel: CPU: L2 cache: 256K
May 15 23:15:20 client-18 kernel: CPU: L3 cache: 4096K
May 15 23:15:20 client-18 kernel: CPU: Physical Processor ID: 0
May 15 23:15:20 client-18 kernel: CPU: Processor Core ID: 1
May 15 23:15:20 client-18 kernel: CPU1: Thermal monitoring enabled (TM1)
May 15 23:15:20 client-18 kernel: Intel(R) Xeon(R) CPU           E5507  @ 2.27GHz stepping 05
May 15 23:15:20 client-18 kernel: SMP alternatives: switching to SMP code
May 15 23:15:20 client-18 kernel: Booting processor 2/4 APIC 0x4
May 15 23:15:20 client-18 kernel: Initializing CPU#2
May 15 23:15:20 client-18 kernel: Calibrating delay using timer specific routine.. 4533.66 BogoMIPS (lpj=2266833)
May 15 23:15:20 client-18 kernel: CPU: L1 I cache: 32K, L1 D cache: 32K
May 15 23:15:20 client-18 kernel: CPU: L2 cache: 256K
May 15 23:15:20 client-18 kernel: CPU: L3 cache: 4096K
May 15 23:15:20 client-18 kernel: CPU: Physical Processor ID: 0
May 15 23:15:20 client-18 kernel: CPU: Processor Core ID: 2
May 15 23:15:20 client-18 kernel: CPU2: Thermal monitoring enabled (TM1)
May 15 23:15:20 client-18 kernel: Intel(R) Xeon(R) CPU           E5507  @ 2.27GHz stepping 05
May 15 23:15:20 client-18 kernel: SMP alternatives: switching to SMP code
May 15 23:15:20 client-18 kernel: Booting processor 3/4 APIC 0x6
May 15 23:15:20 client-18 kernel: Initializing CPU#3
May 15 23:15:20 client-18 kernel: Calibrating delay using timer specific routine.. 4533.40 BogoMIPS (lpj=2266703)
May 15 23:15:20 client-18 kernel: CPU: L1 I cache: 32K, L1 D cache: 32K
May 15 23:15:20 client-18 kernel: CPU: L2 cache: 256K
May 15 23:15:20 client-18 kernel: CPU: L3 cache: 4096K
May 15 23:15:20 client-18 kernel: CPU: Physical Processor ID: 0
May 15 23:15:20 client-18 kernel: CPU: Processor Core ID: 3
May 15 23:15:20 client-18 kernel: CPU3: Thermal monitoring enabled (TM1)
May 15 23:15:20 client-18 kernel: Intel(R) Xeon(R) CPU           E5507  @ 2.27GHz stepping 05
May 15 23:15:20 client-18 kernel: Brought up 4 CPUs
May 15 23:15:20 client-18 kernel: testing NMI watchdog ... OK.
May 15 23:15:20 client-18 kernel: time.c: Using 14.318180 MHz WALL HPET GTOD HPET/TSC timer.
May 15 23:15:20 client-18 kernel: time.c: Detected 2266.813 MHz processor.
May 15 23:15:20 client-18 kernel: migration_cost=21
May 15 23:15:20 client-18 kernel: checking if image is initramfs... it is
May 15 23:15:20 client-18 kernel: Freeing initrd memory: 2611k freed
May 15 23:15:20 client-18 kernel: NET: Registered protocol family 16
May 15 23:15:20 client-18 kernel: ACPI: bus type pci registered
May 15 23:15:20 client-18 kernel: PCI: Using MMCONFIG at e0000000
May 15 23:15:20 client-18 kernel: ACPI: Interpreter enabled
May 15 23:15:20 client-18 kernel: ACPI: Using IOAPIC for interrupt routing
May 15 23:15:20 client-18 kernel: ACPI: No dock devices found.
May 15 23:15:20 client-18 kernel: ACPI: PCI Root Bridge [PCI0] (0000:00)
May 15 23:15:20 client-18 kernel: PCI: Transparent bridge - 0000:00:1e.0
May 15 23:15:20 client-18 kernel: ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 7 10 11 12 14 *15)
May 15 23:15:20 client-18 kernel: ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 *5 6 7 10 11 12 14 15)
May 15 23:15:20 client-18 kernel: ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 5 6 7 *10 11 12 14 15)
May 15 23:15:20 client-18 kernel: ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 6 7 10 *11 12 14 15)
May 15 23:15:20 client-18 kernel: ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 5 6 7 10 11 12 14 15) *0, disabled.
May 15 23:15:20 client-18 kernel: ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 5 6 7 10 11 12 *14 15)
May 15 23:15:20 client-18 kernel: ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 5 6 7 10 11 12 14 15) *0, disabled.
May 15 23:15:20 client-18 kernel: ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 5 6 *7 10 11 12 14 15)
May 15 23:15:20 client-18 kernel: Linux Plug and Play Support v0.97 (c) Adam Belay
May 15 23:15:20 client-18 kernel: pnp: PnP ACPI init
May 15 23:15:20 client-18 kernel: pnp: PnP ACPI: found 15 devices
May 15 23:15:20 client-18 kernel: usbcore: registered new driver usbfs
May 15 23:15:20 client-18 kernel: usbcore: registered new driver hub
May 15 23:15:20 client-18 kernel: PCI: Using ACPI for IRQ routing
May 15 23:15:20 client-18 kernel: PCI: If a device doesn't work, try "pci=routeirq".  If it helps, post a report
May 15 23:15:20 client-18 kernel: NetLabel: Initializing
May 15 23:15:20 client-18 kernel: NetLabel:  domain hash size = 128
May 15 23:15:20 client-18 kernel: NetLabel:  protocols = UNLABELED CIPSOv4
May 15 23:15:20 client-18 kernel: NetLabel:  unlabeled traffic allowed by default
May 15 23:15:20 client-18 kernel: hpet0: at MMIO 0xfed00000 (virtual 0xffffffffff5fe000), IRQs 2, 8, 0, 0
May 15 23:15:20 client-18 kernel: hpet0: 4 64-bit timers, 14318180 Hz
May 15 23:15:20 client-18 kernel: DMAR:Host address width 40
May 15 23:15:20 client-18 kernel: DMAR:DRHD base: 0x000000fbffe000 flags: 0x1
May 15 23:15:20 client-18 kernel: IOMMU fbffe000: ver 1:0 cap c90780106f0462 ecap f020f6
May 15 23:15:20 client-18 kernel: DMAR:RMRR base: 0x000000000e6000 end: 0x000000000e9fff
May 15 23:15:20 client-18 kernel: DMAR:RMRR base: 0x000000bf7ec000 end: 0x000000bf7fffff
May 15 23:15:20 client-18 kernel: DMAR:ATSR flags: 0x0
May 15 23:15:20 client-18 kernel: PCI-GART: No AMD northbridge found.
May 15 23:15:20 client-18 kernel: pnp: 00:08: ioport range 0x164e-0x164f has been reserved
May 15 23:15:20 client-18 kernel: pnp: 00:08: ioport range 0xa00-0xa0f has been reserved
May 15 23:15:20 client-18 kernel: pnp: 00:0c: ioport range 0xca2-0xca3 has been reserved
May 15 23:15:20 client-18 kernel: PCI: Bridge: 0000:00:01.0
May 15 23:15:20 client-18 kernel:   IO window: e000-efff
May 15 23:15:20 client-18 kernel:   MEM window: fbc00000-fbcfffff
May 15 23:15:20 client-18 kernel:   PREFETCH window: disabled.
May 15 23:15:20 client-18 kernel: PCI: Bridge: 0000:00:03.0
May 15 23:15:20 client-18 kernel:   IO window: disabled.
May 15 23:15:20 client-18 kernel:   MEM window: fbd00000-fbdfffff
May 15 23:15:20 client-18 kernel:   PREFETCH window 0x00000000f8800000-0x00000000f8ffffff
May 15 23:15:20 client-18 kernel: PCI: Bridge: 0000:00:05.0
May 15 23:15:20 client-18 kernel:   IO window: disabled.
May 15 23:15:20 client-18 kernel:   MEM window: disabled.
May 15 23:15:20 client-18 kernel:   PREFETCH window: disabled.
May 15 23:15:20 client-18 kernel: PCI: Bridge: 0000:00:07.0
May 15 23:15:20 client-18 kernel:   IO window: disabled.
May 15 23:15:20 client-18 kernel:   MEM window: disabled.
May 15 23:15:20 client-18 kernel:   PREFETCH window: disabled.
May 15 23:15:20 client-18 kernel: PCI: Bridge: 0000:00:1e.0
May 15 23:15:20 client-18 kernel:   IO window: disabled.
May 15 23:15:20 client-18 kernel:   MEM window: faf00000-fb7fffff
May 15 23:15:20 client-18 kernel:   PREFETCH window 0x00000000f9000000-0x00000000f9ffffff
May 15 23:15:20 client-18 kernel: NET: Registered protocol family 2
May 15 23:15:20 client-18 kernel: IP route cache hash table entries: 524288 (order: 10, 4194304 bytes)
May 15 23:15:20 client-18 kernel: TCP established hash table entries: 262144 (order: 10, 4194304 bytes)
May 15 23:15:20 client-18 kernel: TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
May 15 23:15:20 client-18 kernel: TCP: Hash tables configured (established 262144 bind 65536)
May 15 23:15:20 client-18 kernel: TCP reno registered
May 15 23:15:20 client-18 kernel: audit: initializing netlink socket (disabled)
May 15 23:15:20 client-18 kernel: type=2000 audit(1305501266.604:1): initialized
May 15 23:15:20 client-18 kernel: Total HugeTLB memory allocated, 0
May 15 23:15:20 client-18 kernel: VFS: Disk quotas dquot_6.5.1
May 15 23:15:20 client-18 kernel: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
May 15 23:15:20 client-18 kernel: Initializing Cryptographic API
May 15 23:15:20 client-18 kernel: alg: No test for crc32c (crc32c-generic)
May 15 23:15:20 client-18 kernel: ksign: Installing public key data
May 15 23:15:20 client-18 kernel: Loading keyring
May 15 23:15:20 client-18 kernel: - Added public key 565662CFF69DD0BE
May 15 23:15:20 client-18 kernel: - User ID: CentOS (Kernel Module GPG key)
May 15 23:15:20 client-18 kernel: io scheduler noop registered
May 15 23:15:20 client-18 kernel: io scheduler anticipatory registered
May 15 23:15:20 client-18 kernel: io scheduler deadline registered
May 15 23:15:20 client-18 kernel: io scheduler cfq registered (default)
May 15 23:15:20 client-18 kernel: pci_hotplug: PCI Hot Plug PCI Core version: 0.5
May 15 23:15:20 client-18 kernel: ACPI (exconfig-0456): Dynamic SSDT Load - OemId [DpgPmm] OemTableId [ P001Ist] [20060707]
May 15 23:15:20 client-18 kernel: ACPI (exconfig-0456): Dynamic SSDT Load - OemId [ PmRef] OemTableId [ P001Cst] [20060707]
May 15 23:15:20 client-18 kernel: ACPI: CPU0 (power states: C1[C1] C2[C2] C3[C3])
May 15 23:15:20 client-18 kernel: ACPI: CPU1 (power states: C1[C1] C2[C2] C3[C3])
May 15 23:15:20 client-18 kernel: ACPI: CPU2 (power states: C1[C1] C2[C2] C3[C3])
May 15 23:15:20 client-18 kernel: ACPI: CPU3 (power states: C1[C1] C2[C2] C3[C3])
May 15 23:15:20 client-18 kernel: Real Time Clock Driver v1.12ac
May 15 23:15:20 client-18 kernel: Non-volatile memory driver v1.2
May 15 23:15:20 client-18 kernel: Linux agpgart interface v0.101 (c) Dave Jones
May 15 23:15:20 client-18 kernel: Serial: 8250/16550 driver $Revision: 1.90 $ 4 ports, IRQ sharing enabled
May 15 23:15:20 client-18 kernel: serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
May 15 23:15:20 client-18 kernel: serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
May 15 23:15:20 client-18 kernel: 00:06: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
May 15 23:15:20 client-18 kernel: 00:07: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
May 15 23:15:20 client-18 kernel: brd: module loaded
May 15 23:15:20 client-18 kernel: Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
May 15 23:15:20 client-18 kernel: ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
May 15 23:15:20 client-18 kernel: ide-floppy driver 0.99.newide
May 15 23:15:20 client-18 kernel: usbcore: registered new driver hiddev
May 15 23:15:20 client-18 kernel: usbcore: registered new driver usbhid
May 15 23:15:20 client-18 kernel: drivers/usb/input/hid-core.c: v2.6:USB HID core driver
May 15 23:15:20 client-18 kernel: PNP: No PS/2 controller found. Probing ports directly.
May 15 23:15:20 client-18 kernel: serio: i8042 KBD port at 0x60,0x64 irq 1
May 15 23:15:20 client-18 kernel: serio: i8042 AUX port at 0x60,0x64 irq 12
May 15 23:15:20 client-18 kernel: mice: PS/2 mouse device common for all mice
May 15 23:15:20 client-18 kernel: md: md driver 0.90.3 MAX_MD_DEVS=256, MD_SB_DISKS=27
May 15 23:15:20 client-18 kernel: md: bitmap version 4.39
May 15 23:15:20 client-18 kernel: TCP bic registered
May 15 23:15:20 client-18 kernel: Initializing IPsec netlink socket
May 15 23:15:20 client-18 kernel: NET: Registered protocol family 1
May 15 23:15:20 client-18 kernel: NET: Registered protocol family 17
May 15 23:15:20 client-18 kernel: ACPI: (supports S0 S1 S4 S5)
May 15 23:15:20 client-18 kernel: Initalizing network drop monitor service
May 15 23:15:20 client-18 kernel: Freeing unused kernel memory: 212k freed
May 15 23:15:20 client-18 kernel: Write protecting the kernel read-only data: 504k
May 15 23:15:20 client-18 kernel: GSI 16 sharing vector 0xD1 and IRQ 16
May 15 23:15:20 client-18 kernel: ACPI: PCI Interrupt 0000:00:1a.7[C] -> GSI 18 (level, low) -> IRQ 209
May 15 23:15:20 client-18 kernel: ehci_hcd 0000:00:1a.7: EHCI Host Controller
May 15 23:15:20 client-18 kernel: ehci_hcd 0000:00:1a.7: new USB bus registered, assigned bus number 1
May 15 23:15:20 client-18 kernel: ehci_hcd 0000:00:1a.7: debug port 1
May 15 23:15:20 client-18 kernel: ehci_hcd 0000:00:1a.7: irq 209, io mem 0xfbed6000
May 15 23:15:20 client-18 kernel: ehci_hcd 0000:00:1a.7: USB 2.0 started, EHCI 1.00, driver 10 Dec 2004
May 15 23:15:20 client-18 kernel: usb usb1: configuration #1 chosen from 1 choice
May 15 23:15:20 client-18 kernel: hub 1-0:1.0: USB hub found
May 15 23:15:20 client-18 kernel: hub 1-0:1.0: 6 ports detected
May 15 23:15:20 client-18 kernel: GSI 17 sharing vector 0xD9 and IRQ 17
May 15 23:15:20 client-18 kernel: ACPI: PCI Interrupt 0000:00:1d.7[A] -> GSI 23 (level, low) -> IRQ 217
May 15 23:15:20 client-18 kernel: ehci_hcd 0000:00:1d.7: EHCI Host Controller
May 15 23:15:20 client-18 kernel: ehci_hcd 0000:00:1d.7: new USB bus registered, assigned bus number 2
May 15 23:15:20 client-18 kernel: ehci_hcd 0000:00:1d.7: debug port 1
May 15 23:15:20 client-18 kernel: ehci_hcd 0000:00:1d.7: irq 217, io mem 0xfbed4000
May 15 23:15:20 client-18 kernel: ehci_hcd 0000:00:1d.7: USB 2.0 started, EHCI 1.00, driver 10 Dec 2004
May 15 23:15:20 client-18 kernel: usb usb2: configuration #1 chosen from 1 choice
May 15 23:15:20 client-18 kernel: hub 2-0:1.0: USB hub found
May 15 23:15:20 client-18 kernel: hub 2-0:1.0: 6 ports detected
May 15 23:15:20 client-18 kernel: USB Universal Host Controller Interface driver v3.0
May 15 23:15:20 client-18 kernel: GSI 18 sharing vector 0xE1 and IRQ 18
May 15 23:15:20 client-18 kernel: ACPI: PCI Interrupt 0000:00:1a.0[A] -> GSI 16 (level, low) -> IRQ 225
May 15 23:15:20 client-18 kernel: uhci_hcd 0000:00:1a.0: UHCI Host Controller
May 15 23:15:20 client-18 kernel: uhci_hcd 0000:00:1a.0: new USB bus registered, assigned bus number 3
May 15 23:15:20 client-18 kernel: uhci_hcd 0000:00:1a.0: irq 225, io base 0x0000c880
May 15 23:15:20 client-18 kernel: usb usb3: configuration #1 chosen from 1 choice
May 15 23:15:20 client-18 kernel: hub 3-0:1.0: USB hub found
May 15 23:15:20 client-18 kernel: hub 3-0:1.0: 2 ports detected
May 15 23:15:20 client-18 kernel: GSI 19 sharing vector 0xE9 and IRQ 19
May 15 23:15:20 client-18 kernel: ACPI: PCI Interrupt 0000:00:1a.1[B] -> GSI 21 (level, low) -> IRQ 233
May 15 23:15:20 client-18 kernel: uhci_hcd 0000:00:1a.1: UHCI Host Controller
May 15 23:15:20 client-18 kernel: uhci_hcd 0000:00:1a.1: new USB bus registered, assigned bus number 4
May 15 23:15:20 client-18 kernel: uhci_hcd 0000:00:1a.1: irq 233, io base 0x0000cc00
May 15 23:15:20 client-18 kernel: usb usb4: configuration #1 chosen from 1 choice
May 15 23:15:20 client-18 kernel: hub 4-0:1.0: USB hub found
May 15 23:15:20 client-18 kernel: hub 4-0:1.0: 2 ports detected
May 15 23:15:20 client-18 kernel: GSI 20 sharing vector 0x32 and IRQ 20
May 15 23:15:20 client-18 kernel: ACPI: PCI Interrupt 0000:00:1a.2[D] -> GSI 19 (level, low) -> IRQ 50
May 15 23:15:20 client-18 kernel: uhci_hcd 0000:00:1a.2: UHCI Host Controller
May 15 23:15:20 client-18 kernel: uhci_hcd 0000:00:1a.2: new USB bus registered, assigned bus number 5
May 15 23:15:20 client-18 kernel: uhci_hcd 0000:00:1a.2: irq 50, io base 0x0000d000
May 15 23:15:20 client-18 kernel: usb usb5: configuration #1 chosen from 1 choice
May 15 23:15:20 client-18 kernel: hub 5-0:1.0: USB hub found
May 15 23:15:20 client-18 kernel: hub 5-0:1.0: 2 ports detected
May 15 23:15:20 client-18 kernel: ACPI: PCI Interrupt 0000:00:1d.0[A] -> GSI 23 (level, low) -> IRQ 217
May 15 23:15:20 client-18 kernel: uhci_hcd 0000:00:1d.0: UHCI Host Controller
May 15 23:15:20 client-18 kernel: uhci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 6
May 15 23:15:20 client-18 kernel: uhci_hcd 0000:00:1d.0: irq 217, io base 0x0000c400
May 15 23:15:20 client-18 kernel: usb usb6: configuration #1 chosen from 1 choice
May 15 23:15:20 client-18 kernel: hub 6-0:1.0: USB hub found
May 15 23:15:20 client-18 kernel: hub 6-0:1.0: 2 ports detected
May 15 23:15:20 client-18 kernel: usb 4-1: new full speed USB device using uhci_hcd and address 2
May 15 23:15:20 client-18 kernel: ACPI: PCI Interrupt 0000:00:1d.1[B] -> GSI 19 (level, low) -> IRQ 50
May 15 23:15:20 client-18 kernel: uhci_hcd 0000:00:1d.1: UHCI Host Controller
May 15 23:15:20 client-18 rpc.statd[3124]: Version 1.0.9 Starting
May 15 23:15:20 client-18 kernel: uhci_hcd 0000:00:1d.1: new USB bus registered, assigned bus number 7
May 15 23:15:20 client-18 kernel: uhci_hcd 0000:00:1d.1: irq 50, io base 0x0000c480
May 15 23:15:20 client-18 kernel: usb usb7: configuration #1 chosen from 1 choice
May 15 23:15:20 client-18 kernel: hub 7-0:1.0: USB hub found
May 15 23:15:20 client-18 kernel: hub 7-0:1.0: 2 ports detected
May 15 23:15:20 client-18 kernel: usb 4-1: configuration #1 chosen from 1 choice
May 15 23:15:20 client-18 kernel: ACPI: PCI Interrupt 0000:00:1d.2[C] -> GSI 18 (level, low) -> IRQ 209
May 15 23:15:20 client-18 kernel: uhci_hcd 0000:00:1d.2: UHCI Host Controller
May 15 23:15:20 client-18 kernel: uhci_hcd 0000:00:1d.2: new USB bus registered, assigned bus number 8
May 15 23:15:20 client-18 kernel: input: American Megatrends Inc. Virtual Keyboard and Mouse as /class/input/input0
May 15 23:15:20 client-18 kernel: input: USB HID v1.10 Keyboard [American Megatrends Inc. Virtual Keyboard and Mouse] on usb-0000:00:1a.1-1
May 15 23:15:20 client-18 kernel: input: American Megatrends Inc. Virtual Keyboard and Mouse as /class/input/input1
May 15 23:15:20 client-18 kernel: input: USB HID v1.10 Mouse [American Megatrends Inc. Virtual Keyboard and Mouse] on usb-0000:00:1a.1-1
May 15 23:15:20 client-18 kernel: uhci_hcd 0000:00:1d.2: irq 209, io base 0x0000c800
May 15 23:15:20 client-18 kernel: usb usb8: configuration #1 chosen from 1 choice
May 15 23:15:20 client-18 kernel: hub 8-0:1.0: USB hub found
May 15 23:15:20 client-18 kernel: hub 8-0:1.0: 2 ports detected
May 15 23:15:20 client-18 kernel: SCSI subsystem initialized
May 15 23:15:20 client-18 kernel: ACPI: PCI Interrupt 0000:00:1f.2[B] -> GSI 19 (level, low) -> IRQ 50
May 15 23:15:20 client-18 kernel: ahci 0000:00:1f.2: AHCI 0001.0200 32 slots 6 ports 3 Gbps 0x3f impl SATA mode
May 15 23:15:20 client-18 kernel: ahci 0000:00:1f.2: flags: 64bit ncq sntf stag pm led clo pio slum part ems 
May 15 23:15:20 client-18 rpc.statd[3124]: statd running as root. chown /var/lib/nfs/statd/sm to choose different user 
May 15 23:15:20 client-18 kernel: scsi0 : ahci
May 15 23:15:20 client-18 kernel: scsi1 : ahci
May 15 23:15:20 client-18 kernel: scsi2 : ahci
May 15 23:15:20 client-18 kernel: scsi3 : ahci
May 15 23:15:20 client-18 kernel: scsi4 : ahci
May 15 23:15:20 client-18 kernel: scsi5 : ahci
May 15 23:15:20 client-18 kernel: ata1: SATA max UDMA/133 abar m2048@0xfbefa000 port 0xfbefa100 irq 58
May 15 23:15:20 client-18 kernel: ata2: SATA max UDMA/133 abar m2048@0xfbefa000 port 0xfbefa180 irq 58
May 15 23:15:20 client-18 kernel: ata3: SATA max UDMA/133 abar m2048@0xfbefa000 port 0xfbefa200 irq 58
May 15 23:15:20 client-18 kernel: ata4: SATA max UDMA/133 abar m2048@0xfbefa000 port 0xfbefa280 irq 58
May 15 23:15:20 client-18 kernel: ata5: SATA max UDMA/133 abar m2048@0xfbefa000 port 0xfbefa300 irq 58
May 15 23:15:20 client-18 kernel: ata6: SATA max UDMA/133 abar m2048@0xfbefa000 port 0xfbefa380 irq 58
May 15 23:15:20 client-18 kernel: ata1: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
May 15 23:15:20 client-18 kernel: ata1.00: ATA-8: WDC WD2502ABYS-02B7A0, 02.03B03, max UDMA/133
May 15 23:15:20 client-18 kernel: ata1.00: 490350672 sectors, multi 0: LBA48 NCQ (depth 31/32)
May 15 23:15:20 client-18 kernel: ata1.00: configured for UDMA/133
May 15 23:15:20 client-18 kernel: ata2: SATA link down (SStatus 0 SControl 300)
May 15 23:15:20 client-18 kernel: ata3: SATA link down (SStatus 0 SControl 300)
May 15 23:15:20 client-18 kernel: ata4: SATA link down (SStatus 0 SControl 300)
May 15 23:15:20 client-18 kernel: ata5: SATA link down (SStatus 0 SControl 300)
May 15 23:15:20 client-18 kernel: ata6: SATA link down (SStatus 0 SControl 300)
May 15 23:15:20 client-18 kernel:   Vendor: ATA       Model: WDC WD2502ABYS-0  Rev: 02.0
May 15 23:15:20 client-18 kernel:   Type:   Direct-Access                      ANSI SCSI revision: 05
May 15 23:15:20 client-18 kernel: SCSI device sda: 490350672 512-byte hdwr sectors (251060 MB)
May 15 23:15:20 client-18 kernel: sda: Write Protect is off
May 15 23:15:20 client-18 kernel: SCSI device sda: drive cache: write back
May 15 23:15:20 client-18 kernel: SCSI device sda: 490350672 512-byte hdwr sectors (251060 MB)
May 15 23:15:20 client-18 kernel: sda: Write Protect is off
May 15 23:15:20 client-18 kernel: SCSI device sda: drive cache: write back
May 15 23:15:20 client-18 kernel:  sda: sda1 sda2
May 15 23:15:20 client-18 kernel: sd 0:0:0:0: Attached scsi disk sda
May 15 23:15:20 client-18 kernel: device-mapper: uevent: version 1.0.3
May 15 23:15:20 client-18 kernel: device-mapper: ioctl: 4.11.5-ioctl (2007-12-12) initialised: dm-devel@redhat.com
May 15 23:15:20 client-18 kernel: device-mapper: dm-raid45: initialized v0.2594l
May 15 23:15:20 client-18 kernel: kjournald starting.  Commit interval 5 seconds
May 15 23:15:20 client-18 kernel: EXT3-fs: mounted filesystem with ordered data mode.
May 15 23:15:20 client-18 kernel: SELinux:  Disabled at runtime.
May 15 23:15:20 client-18 kernel: type=1404 audit(1305501293.111:2): selinux=0 auid=4294967295 ses=4294967295
May 15 23:15:20 client-18 kernel: input: PC Speaker as /class/input/input2
May 15 23:15:20 client-18 kernel: mlx4_core: Mellanox ConnectX core driver v1.0-ofed1.5.3 (January 19, 2011)
May 15 23:15:20 client-18 kernel: mlx4_core: Initializing 0000:02:00.0
May 15 23:15:20 client-18 kernel: GSI 21 sharing vector 0x42 and IRQ 21
May 15 23:15:20 client-18 kernel: ACPI: PCI Interrupt 0000:02:00.0[A] -> GSI 24 (level, low) -> IRQ 66
May 15 23:15:20 client-18 kernel: dca service started, version 1.4
May 15 23:15:20 client-18 kernel: sd 0:0:0:0: Attached scsi generic sg0 type 0
May 15 23:15:20 client-18 kernel: 802.1Q VLAN Support v1.8 Ben Greear <greearb@candelatech.com>
May 15 23:15:20 client-18 kernel: All bugs added by David S. Miller <davem@redhat.com>
May 15 23:15:20 client-18 kernel: Intel(R) Gigabit Ethernet Network Driver - version 2.1.0-k2
May 15 23:15:20 client-18 kernel: Copyright (c) 2007-2009 Intel Corporation.
May 15 23:15:20 client-18 kernel: shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
May 15 23:15:20 client-18 kernel: GSI 22 sharing vector 0x53 and IRQ 22
May 15 23:15:20 client-18 kernel: ACPI: PCI Interrupt 0000:01:00.0[A] -> GSI 28 (level, low) -> IRQ 83
May 15 23:15:20 client-18 kernel: mlx4_ib: Mellanox ConnectX InfiniBand driver v1.0-ofed1.5.3 (January 19, 2011)
May 15 23:15:20 client-18 kernel: igb 0000:01:00.0: Intel(R) Gigabit Ethernet Network Connection
May 15 23:15:20 client-18 kernel: igb 0000:01:00.0: eth0: (PCIe:2.5Gb/s:Width x4) 00:25:90:1d:3e:22
May 15 23:15:20 client-18 kernel: igb 0000:01:00.0: eth0: PBA No: ffffff-0ff
May 15 23:15:20 client-18 kernel: igb 0000:01:00.0: Using MSI-X interrupts. 4 rx queue(s), 1 tx queue(s)
May 15 23:15:20 client-18 kernel: GSI 23 sharing vector 0x8B and IRQ 23
May 15 23:15:20 client-18 kernel: ACPI: PCI Interrupt 0000:01:00.1[B] -> GSI 40 (level, low) -> IRQ 139
May 15 23:15:20 client-18 kernel: igb 0000:01:00.1: Intel(R) Gigabit Ethernet Network Connection
May 15 23:15:20 client-18 kernel: igb 0000:01:00.1: eth1: (PCIe:2.5Gb/s:Width x4) 00:25:90:1d:3e:23
May 15 23:15:20 client-18 kernel: igb 0000:01:00.1: eth1: PBA No: ffffff-0ff
May 15 23:15:20 client-18 kernel: igb 0000:01:00.1: Using MSI-X interrupts. 4 rx queue(s), 1 tx queue(s)
May 15 23:15:20 client-18 kernel: ACPI: PCI Interrupt 0000:00:1f.3[C] -> GSI 18 (level, low) -> IRQ 209
May 15 23:15:20 client-18 kernel: floppy0: no floppy controllers found
May 15 23:15:20 client-18 kernel: lp: driver loaded but no devices found
May 15 23:15:20 client-18 kernel: ACPI: Power Button (FF) [PWRF]
May 15 23:15:20 client-18 kernel: ACPI: Power Button (CM) [PWRB]
May 15 23:15:20 client-18 kernel: ACPI: Mapper loaded
May 15 23:15:20 client-18 kernel: dell-wmi: No known WMI GUID found
May 15 23:15:20 client-18 kernel: md: Autodetecting RAID arrays.
May 15 23:15:20 client-18 kernel: md: autorun ...
May 15 23:15:20 client-18 kernel: md: ... autorun DONE.
May 15 23:15:20 client-18 kernel: device-mapper: multipath: version 1.0.5 loaded
May 15 23:15:20 client-18 kernel: loop: loaded (max 8 devices)
May 15 23:15:20 client-18 kernel: EXT3 FS on sda1, internal journal
May 15 23:15:20 client-18 kernel: Adding 14346036k swap on /dev/sda2.  Priority:-1 extents:1 across:14346036k
May 15 23:15:20 client-18 kernel: IA-32 Microcode Update Driver: v1.14a <tigran@veritas.com>
May 15 23:15:20 client-18 kernel: mlx4_en: Mellanox ConnectX HCA Ethernet driver v1.5.3 (Jan 2011)
May 15 23:15:20 client-18 kernel: mlx4_en 0000:02:00.0: UDP RSS is not supported on this device.
May 15 23:15:20 client-18 kernel: NET: Registered protocol family 10
May 15 23:15:20 client-18 kernel: lo: Disabled Privacy Extensions
May 15 23:15:20 client-18 kernel: IPv6 over IPv4 tunneling driver
May 15 23:15:20 client-18 kernel: ADDRCONF(NETDEV_UP): ib0: link is not ready
May 15 23:15:20 client-18 kernel: ib0: enabling connected mode will cause multicast packet drops
May 15 23:15:20 client-18 kernel: ib0: mtu > 2044 will cause multicast packet drops.
May 15 23:15:20 client-18 kernel: ib0: mtu > 2044 will cause multicast packet drops.
May 15 23:15:20 client-18 kernel: NET: Registered protocol family 27
May 15 23:15:20 client-18 kernel: NET: Registered protocol family 28
May 15 23:15:20 client-18 kernel: QLGC_VNIC: Initializing QLogic Corp. Virtual NIC (VNIC) driver version 1.3.0.0.4
May 15 23:15:20 client-18 kernel: Loading iSCSI transport class v2.0-871.
May 15 23:15:20 client-18 kernel: cxgb3i: disagrees about version of symbol cxgb3_register_client
May 15 23:15:20 client-18 kernel: cxgb3i: Unknown symbol cxgb3_register_client
May 15 23:15:20 client-18 kernel: cxgb3i: disagrees about version of symbol cxgb3_alloc_atid
May 15 23:15:20 client-18 kernel: cxgb3i: Unknown symbol cxgb3_alloc_atid
May 15 23:15:20 client-18 kernel: cxgb3i: disagrees about version of symbol t3_l2t_get
May 15 23:15:20 client-18 kernel: cxgb3i: Unknown symbol t3_l2t_get
May 15 23:15:20 client-18 kernel: cxgb3i: disagrees about version of symbol cxgb3_insert_tid
May 15 23:15:20 client-18 kernel: cxgb3i: Unknown symbol cxgb3_insert_tid
May 15 23:15:20 client-18 kernel: cxgb3i: disagrees about version of symbol t3_l2e_free
May 15 23:15:20 client-18 kernel: cxgb3i: Unknown symbol t3_l2e_free
May 15 23:15:20 client-18 kernel: cxgb3i: disagrees about version of symbol t3_l2t_send_slow
May 15 23:15:20 client-18 kernel: cxgb3i: Unknown symbol t3_l2t_send_slow
May 15 23:15:20 client-18 kernel: cxgb3i: disagrees about version of symbol cxgb3_unregister_client
May 15 23:15:20 client-18 kernel: cxgb3i: Unknown symbol cxgb3_unregister_client
May 15 23:15:20 client-18 kernel: Broadcom NetXtreme II CNIC Driver cnic v2.1.0 (Oct 10, 2009)
May 15 23:15:20 client-18 kernel: Broadcom NetXtreme II iSCSI Driver bnx2i v2.1.0 (Dec 06, 2009)
May 15 23:15:20 client-18 kernel: iscsi: registered transport (bnx2i)
May 15 23:15:20 client-18 kernel: iscsi: registered transport (tcp)
May 15 23:15:20 client-18 kernel: ib_iser: disagrees about version of symbol ib_fmr_pool_unmap
May 15 23:15:20 client-18 kernel: ib_iser: Unknown symbol ib_fmr_pool_unmap
May 15 23:15:20 client-18 kernel: ib_iser: disagrees about version of symbol ib_create_cq
May 15 23:15:20 client-18 kernel: ib_iser: Unknown symbol ib_create_cq
May 15 23:15:20 client-18 kernel: ib_iser: disagrees about version of symbol rdma_resolve_addr
May 15 23:15:20 client-18 kernel: ib_iser: Unknown symbol rdma_resolve_addr
May 15 23:15:20 client-18 kernel: ib_iser: disagrees about version of symbol ib_create_fmr_pool
May 15 23:15:20 client-18 kernel: ib_iser: Unknown symbol ib_create_fmr_pool
May 15 23:15:20 client-18 kernel: ib_iser: disagrees about version of symbol ib_dereg_mr
May 15 23:15:20 client-18 kernel: ib_iser: Unknown symbol ib_dereg_mr
May 15 23:15:20 client-18 kernel: ib_iser: disagrees about version of symbol rdma_disconnect
May 15 23:15:20 client-18 kernel: ib_iser: Unknown symbol rdma_disconnect
May 15 23:15:20 client-18 kernel: ib_iser: disagrees about version of symbol rdma_resolve_route
May 15 23:15:20 client-18 kernel: ib_iser: Unknown symbol rdma_resolve_route
May 15 23:15:20 client-18 kernel: ib_iser: disagrees about version of symbol rdma_create_qp
May 15 23:15:20 client-18 kernel: ib_iser: Unknown symbol rdma_create_qp
May 15 23:15:20 client-18 kernel: ib_iser: disagrees about version of symbol ib_destroy_cq
May 15 23:15:20 client-18 kernel: ib_iser: Unknown symbol ib_destroy_cq
May 15 23:15:20 client-18 kernel: ib_iser: disagrees about version of symbol rdma_create_id
May 15 23:15:20 client-18 kernel: ib_iser: Unknown symbol rdma_create_id
May 15 23:15:20 client-18 kernel: ib_iser: disagrees about version of symbol rdma_destroy_qp
May 15 23:15:20 client-18 kernel: ib_iser: Unknown symbol rdma_destroy_qp
May 15 23:15:20 client-18 kernel: ib_iser: disagrees about version of symbol ib_get_dma_mr
May 15 23:15:20 client-18 kernel: ib_iser: Unknown symbol ib_get_dma_mr
May 15 23:15:20 client-18 kernel: ib_iser: disagrees about version of symbol ib_alloc_pd
May 15 23:15:20 client-18 kernel: ib_iser: Unknown symbol ib_alloc_pd
May 15 23:15:20 client-18 kernel: ib_iser: disagrees about version of symbol rdma_connect
May 15 23:15:20 client-18 kernel: ib_iser: Unknown symbol rdma_connect
May 15 23:15:20 client-18 kernel: ib_iser: disagrees about version of symbol rdma_destroy_id
May 15 23:15:20 client-18 kernel: ib_iser: Unknown symbol rdma_destroy_id
May 15 23:15:20 client-18 kernel: ib_iser: disagrees about version of symbol ib_dealloc_pd
May 15 23:15:20 client-18 kernel: ib_iser: Unknown symbol ib_dealloc_pd
May 15 23:15:20 client-18 kernel: ib_iser: disagrees about version of symbol ib_fmr_pool_map_phys
May 15 23:15:20 client-18 kernel: ib_iser: Unknown symbol ib_fmr_pool_map_phys
May 15 23:15:20 client-18 kernel: iscsi: registered transport (be2iscsi)
May 15 23:15:20 client-18 kernel: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
May 15 23:15:20 client-18 kernel: ADDRCONF(NETDEV_UP): eth1: link is not ready
May 15 23:15:20 client-18 kernel: ADDRCONF(NETDEV_CHANGE): ib0: link becomes ready
May 15 23:15:21 client-18 hcid[3167]: Bluetooth HCI daemon
May 15 23:15:21 client-18 kernel: Bluetooth: Core ver 2.10
May 15 23:15:21 client-18 kernel: NET: Registered protocol family 31
May 15 23:15:21 client-18 kernel: Bluetooth: HCI device and connection manager initialized
May 15 23:15:21 client-18 kernel: Bluetooth: HCI socket layer initialized
May 15 23:15:21 client-18 hcid[3167]: Register path:/org/bluez fallback:1
May 15 23:15:21 client-18 kernel: Bluetooth: L2CAP ver 2.8
May 15 23:15:21 client-18 kernel: Bluetooth: L2CAP socket layer initialized
May 15 23:15:21 client-18 sdpd[3173]: Bluetooth SDP daemon 
May 15 23:15:21 client-18 kernel: Bluetooth: RFCOMM socket layer initialized
May 15 23:15:21 client-18 kernel: Bluetooth: RFCOMM TTY layer initialized
May 15 23:15:21 client-18 kernel: Bluetooth: RFCOMM ver 1.8
May 15 23:15:21 client-18 pcscd: pcscdaemon.c:507:main() pcsc-lite 1.4.4 daemon ready.
May 15 23:15:21 client-18 pcscd: hotplug_libusb.c:402:HPEstablishUSBNotifications() Driver ifd-egate.bundle does not support IFD_GENERATE_HOTPLUG. Using active polling instead.
May 15 23:15:21 client-18 pcscd: hotplug_libusb.c:411:HPEstablishUSBNotifications() Polling forced every 1 second(s)
May 15 23:15:22 client-18 kernel: Bluetooth: HIDP (Human Interface Emulation) ver 1.1
May 15 23:15:22 client-18 hidd[3297]: Bluetooth HID daemon
May 15 23:15:23 client-18 automount[3326]: lookup_read_master: lookup(nisplus): couldn't locate nis+ table auto.master
May 15 23:15:23 client-18 automount[3326]: lookup_read_map: lookup(nisplus): couldn't locate nis+ table auto.direct
May 15 23:15:23 client-18 sssd: Starting up
May 15 23:15:24 client-18 sssd[be[LAB.WHAMCLOUD.COM]]: Starting up
May 15 23:15:24 client-18 sssd[nss]: Starting up
May 15 23:15:24 client-18 sssd[pam]: Starting up
May 15 23:15:25 client-18 ntpd[3398]: ntpd 4.2.2p1@1.1570-o Sat Dec 19 00:56:13 UTC 2009 (1)
May 15 23:15:25 client-18 ntpd[3399]: precision = 1.000 usec
May 15 23:15:25 client-18 ntpd[3399]: Listening on interface wildcard, 0.0.0.0#123 Disabled
May 15 23:15:25 client-18 ntpd[3399]: Listening on interface wildcard, ::#123 Disabled
May 15 23:15:25 client-18 ntpd[3399]: Listening on interface lo, ::1#123 Enabled
May 15 23:15:25 client-18 ntpd[3399]: Listening on interface eth0, fe80::225:90ff:fe1d:3e22#123 Enabled
May 15 23:15:25 client-18 ntpd[3399]: Listening on interface ib0, fe80::225:90ff:ff1b:6709#123 Enabled
May 15 23:15:25 client-18 ntpd[3399]: Listening on interface lo, 127.0.0.1#123 Enabled
May 15 23:15:25 client-18 ntpd[3399]: Listening on interface eth0, 10.10.4.18#123 Enabled
May 15 23:15:25 client-18 ntpd[3399]: Listening on interface eth1, 172.16.8.18#123 Enabled
May 15 23:15:25 client-18 ntpd[3399]: Listening on interface ib0, 192.168.4.18#123 Enabled
May 15 23:15:25 client-18 ntpd[3399]: kernel time sync status 0040
May 15 23:15:25 client-18 xinetd[3388]: xinetd Version 2.3.14 started with libwrap loadavg labeled-networking options compiled in.
May 15 23:15:25 client-18 xinetd[3388]: Started working: 2 available services
May 15 23:15:25 client-18 gpm[3434]: *** info [startup.c(95)]: 
May 15 23:15:25 client-18 gpm[3434]: Started gpm successfully. Entered daemon mode.
May 15 23:15:25 client-18 smartd[3503]: smartd version 5.38 [x86_64-redhat-linux-gnu] Copyright (C) 2002-8 Bruce Allen 
May 15 23:15:25 client-18 smartd[3503]: Home page is http://smartmontools.sourceforge.net/  
May 15 23:15:25 client-18 smartd[3503]: Opened configuration file /etc/smartd.conf 
May 15 23:15:25 client-18 smartd[3503]: Configuration file /etc/smartd.conf was parsed, found DEVICESCAN, scanning devices 
May 15 23:15:25 client-18 smartd[3503]: Problem creating device name scan list 
May 15 23:15:25 client-18 smartd[3503]: Device: /dev/sda, opened 
May 15 23:15:25 client-18 smartd[3503]: Device /dev/sda: using '-d sat' for ATA disk behind SAT layer. 
May 15 23:15:25 client-18 smartd[3503]: Device: /dev/sda, opened 
May 15 23:15:25 client-18 smartd[3503]: Device: /dev/sda, not found in smartd database. 
May 15 23:15:26 client-18 smartd[3503]: Device: /dev/sda, is SMART capable. Adding to "monitor" list. 
May 15 23:15:26 client-18 smartd[3503]: Monitoring 0 ATA and 1 SCSI devices 
May 15 23:15:26 client-18 smartd[3506]: smartd has fork()ed into background mode. New PID=3506. 
May 15 23:19:43 client-18 ntpd[3399]: synchronized to 10.10.0.1, stratum 2
May 17 00:19:43 client-18 ntpd[3399]: time reset +89999.999316 s
May 17 00:24:33 client-18 ntpd[3399]: synchronized to 10.10.0.2, stratum 3
May 17 00:27:45 client-18 ntpd[3399]: synchronized to 10.10.0.1, stratum 2
May 17 00:41:03 client-18 ntpd[3399]: kernel time sync enabled 0001
May 17 01:06:45 client-18 xinetd[3388]: START: shell pid=3748 from=10.10.0.1
May 17 01:06:45 client-18 rshd[3749]: sarah@brent.lab.whamcloud.com as root: cmd='cp /home/sarah/.maloorc /root'
May 17 01:06:45 client-18 kernel: FS-Cache: Loaded
May 17 01:06:45 client-18 xinetd[3388]: EXIT: shell status=0 pid=3748 duration=0(sec)
May 17 01:07:02 client-18 xinetd[3388]: START: shell pid=3795 from=10.10.0.1
May 17 01:07:02 client-18 rshd[3796]: sarah@brent.lab.whamcloud.com as root: cmd='cp /home/sarah/backup/sarah.sh /usr/lib64/lustre/tests/cfg'
May 17 01:07:02 client-18 xinetd[3388]: EXIT: shell status=0 pid=3795 duration=0(sec)
May 17 01:07:20 client-18 xinetd[3388]: START: shell pid=3828 from=10.10.0.1
May 17 01:07:20 client-18 rshd[3829]: sarah@brent.lab.whamcloud.com as root: cmd='cp /home/sarah/backup/modprobe.conf /etc/'
May 17 01:07:20 client-18 xinetd[3388]: EXIT: shell status=0 pid=3828 duration=0(sec)
May 17 01:08:38 client-18 shutdown[3914]: shutting down for system halt
May 17 01:08:38 client-18 init: Switching to runlevel: 0
May 17 01:08:39 client-18 smartd[3506]: smartd received signal 15: Terminated 
May 17 01:08:39 client-18 smartd[3506]: smartd is exiting (exit status 0) 
May 17 01:08:40 client-18 xinetd[3388]: Exiting...
May 16 00:11:24 client-18 syslogd 1.4.1: restart.
May 16 00:11:24 client-18 kernel: klogd 1.4.1, log source = /proc/kmsg started.
May 16 00:11:24 client-18 kernel: Linux version 2.6.18-194.17.1.el5 (mockbuild@builder10.centos.org) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-48)) #1 SMP Wed Sep 29 12:50:31 EDT 2010
May 16 00:11:24 client-18 kernel: Command line: ro root=LABEL=/ console=ttyS0,115200
May 16 00:11:24 client-18 kernel: BIOS-provided physical RAM map:
May 16 00:11:24 client-18 kernel:  BIOS-e820: 0000000000010000 - 000000000009cc00 (usable)
May 16 00:11:24 client-18 kernel:  BIOS-e820: 000000000009cc00 - 00000000000a0000 (reserved)
May 16 00:11:24 client-18 kernel:  BIOS-e820: 00000000000e6000 - 0000000000100000 (reserved)
May 16 00:11:24 client-18 kernel:  BIOS-e820: 0000000000100000 - 00000000bf760000 (usable)
May 16 00:11:24 client-18 kernel:  BIOS-e820: 00000000bf76e000 - 00000000bf770000 type 9
May 16 00:11:24 client-18 kernel:  BIOS-e820: 00000000bf770000 - 00000000bf77e000 (ACPI data)
May 16 00:11:24 client-18 kernel:  BIOS-e820: 00000000bf77e000 - 00000000bf7d0000 (ACPI NVS)
May 16 00:11:24 client-18 kernel:  BIOS-e820: 00000000bf7d0000 - 00000000bf7e0000 (reserved)
May 16 00:11:24 client-18 kernel:  BIOS-e820: 00000000bf7ec000 - 00000000c0000000 (reserved)
May 16 00:11:24 client-18 kernel:  BIOS-e820: 00000000e0000000 - 00000000f0000000 (reserved)
May 16 00:11:24 client-18 kernel:  BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
May 16 00:11:24 client-18 kernel:  BIOS-e820: 00000000ffc00000 - 0000000100000000 (reserved)
May 16 00:11:24 client-18 kernel:  BIOS-e820: 0000000100000000 - 0000000340000000 (usable)
May 16 00:11:24 client-18 kernel: DMI present.
May 16 00:11:24 client-18 kernel: No NUMA configuration found
May 16 00:11:24 client-18 kernel: Faking a node at 0000000000000000-0000000340000000
May 16 00:11:24 client-18 kernel: Bootmem setup node 0 0000000000000000-0000000340000000
May 16 00:11:24 client-18 kernel: Memory for crash kernel (0x0 to 0x0) notwithin permissible range
May 16 00:11:24 client-18 kernel: disabling kdump
May 16 00:11:24 client-18 kernel: ACPI: PM-Timer IO Port: 0x808
May 16 00:11:24 client-18 kernel: ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
May 16 00:11:24 client-18 kernel: Processor #0 7:10 APIC version 21
May 16 00:11:24 client-18 kernel: ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)
May 16 00:11:24 client-18 kernel: Processor #2 7:10 APIC version 21
May 16 00:11:24 client-18 kernel: ACPI: LAPIC (acpi_id[0x03] lapic_id[0x04] enabled)
May 16 00:11:24 client-18 kernel: Processor #4 7:10 APIC version 21
May 16 00:11:24 client-18 kernel: ACPI: LAPIC (acpi_id[0x04] lapic_id[0x06] enabled)
May 16 00:11:24 client-18 kernel: Processor #6 7:10 APIC version 21
May 16 00:11:24 client-18 kernel: ACPI: LAPIC (acpi_id[0x05] lapic_id[0x84] disabled)
May 16 00:11:24 client-18 kernel: ACPI: LAPIC (acpi_id[0x06] lapic_id[0x85] disabled)
May 16 00:11:24 client-18 kernel: ACPI: LAPIC (acpi_id[0x07] lapic_id[0x86] disabled)
May 16 00:11:24 client-18 kernel: ACPI: LAPIC (acpi_id[0x08] lapic_id[0x87] disabled)
May 16 00:11:24 client-18 kernel: ACPI: LAPIC (acpi_id[0x09] lapic_id[0x88] disabled)
May 16 00:11:24 client-18 kernel: ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x89] disabled)
May 16 00:11:24 client-18 kernel: ACPI: LAPIC (acpi_id[0x0b] lapic_id[0x8a] disabled)
May 16 00:11:24 client-18 kernel: ACPI: LAPIC (acpi_id[0x0c] lapic_id[0x8b] disabled)
May 16 00:11:24 client-18 kernel: ACPI: LAPIC (acpi_id[0x0d] lapic_id[0x8c] disabled)
May 16 00:11:24 client-18 kernel: ACPI: LAPIC (acpi_id[0x0e] lapic_id[0x8d] disabled)
May 16 00:11:24 client-18 kernel: ACPI: LAPIC (acpi_id[0x0f] lapic_id[0x8e] disabled)
May 16 00:11:24 client-18 kernel: ACPI: LAPIC (acpi_id[0x10] lapic_id[0x8f] disabled)
May 16 00:11:24 client-18 kernel: ACPI: LAPIC (acpi_id[0x11] lapic_id[0x90] disabled)
May 16 00:11:24 client-18 kernel: ACPI: LAPIC (acpi_id[0x12] lapic_id[0x91] disabled)
May 16 00:11:24 client-18 kernel: ACPI: LAPIC (acpi_id[0x13] lapic_id[0x92] disabled)
May 16 00:11:24 client-18 kernel: ACPI: LAPIC (acpi_id[0x14] lapic_id[0x93] disabled)
May 16 00:11:24 client-18 kernel: ACPI: LAPIC (acpi_id[0x15] lapic_id[0x94] disabled)
May 16 00:11:24 client-18 kernel: ACPI: LAPIC (acpi_id[0x16] lapic_id[0x95] disabled)
May 16 00:11:24 client-18 kernel: ACPI: LAPIC (acpi_id[0x17] lapic_id[0x96] disabled)
May 16 00:11:24 client-18 kernel: ACPI: LAPIC (acpi_id[0x18] lapic_id[0x97] disabled)
May 16 00:11:24 client-18 kernel: ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])
May 16 00:11:24 client-18 kernel: ACPI: IOAPIC (id[0x01] address[0xfec00000] gsi_base[0])
May 16 00:11:24 client-18 kernel: IOAPIC[0]: apic_id 1, version 32, address 0xfec00000, GSI 0-23
May 16 00:11:24 client-18 kernel: ACPI: IOAPIC (id[0x03] address[0xfec8a000] gsi_base[24])
May 16 00:11:24 client-18 kernel: IOAPIC[1]: apic_id 3, version 32, address 0xfec8a000, GSI 24-47
May 16 00:11:24 client-18 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
May 16 00:11:24 client-18 cpuspeed: Enabling ondemand cpu frequency scaling governor
May 16 00:11:24 client-18 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
May 16 00:11:24 client-18 kernel: Setting APIC routing to physical flat
May 16 00:11:24 client-18 kernel: ACPI: HPET id: 0x8086a301 base: 0xfed00000
May 16 00:11:24 client-18 kernel: Using ACPI (MADT) for SMP configuration information
May 16 00:11:24 client-18 kernel: Nosave address range: 000000000009c000 - 000000000009d000
May 16 00:11:24 client-18 kernel: Nosave address range: 000000000009d000 - 00000000000a0000
May 16 00:11:24 client-18 kernel: Nosave address range: 00000000000a0000 - 00000000000e6000
May 16 00:11:24 client-18 kernel: Nosave address range: 00000000000e6000 - 0000000000100000
May 16 00:11:24 client-18 kernel: Nosave address range: 00000000bf760000 - 00000000bf76e000
May 16 00:11:24 client-18 kernel: Nosave address range: 00000000bf76e000 - 00000000bf770000
May 16 00:11:24 client-18 kernel: Nosave address range: 00000000bf770000 - 00000000bf77e000
May 16 00:11:24 client-18 kernel: Nosave address range: 00000000bf77e000 - 00000000bf7d0000
May 16 00:11:24 client-18 kernel: Nosave address range: 00000000bf7d0000 - 00000000bf7e0000
May 16 00:11:24 client-18 kernel: Nosave address range: 00000000bf7e0000 - 00000000bf7ec000
May 16 00:11:24 client-18 kernel: Nosave address range: 00000000bf7ec000 - 00000000c0000000
May 16 00:11:24 client-18 kernel: Nosave address range: 00000000c0000000 - 00000000e0000000
May 16 00:11:24 client-18 kernel: Nosave address range: 00000000e0000000 - 00000000f0000000
May 16 00:11:24 client-18 kernel: Nosave address range: 00000000f0000000 - 00000000fee00000
May 16 00:11:24 client-18 kernel: Nosave address range: 00000000fee00000 - 00000000fee01000
May 16 00:11:24 client-18 kernel: Nosave address range: 00000000fee01000 - 00000000ffc00000
May 16 00:11:24 client-18 kernel: Nosave address range: 00000000ffc00000 - 0000000100000000
May 16 00:11:24 client-18 kernel: Allocating PCI resources starting at c2000000 (gap: c0000000:20000000)
May 16 00:11:24 client-18 kernel: SMP: Allowing 24 CPUs, 20 hotplug CPUs
May 16 00:11:24 client-18 kernel: Built 1 zonelists.  Total pages: 3095502
May 16 00:11:24 client-18 kernel: Kernel command line: ro root=LABEL=/ console=ttyS0,115200
May 16 00:11:24 client-18 kernel: Initializing CPU#0
May 16 00:11:24 client-18 kernel: PID hash table entries: 4096 (order: 12, 32768 bytes)
May 16 00:11:24 client-18 kernel: Console: colour VGA+ 80x25
May 16 00:11:24 client-18 kernel: Dentry cache hash table entries: 2097152 (order: 12, 16777216 bytes)
May 16 00:11:24 client-18 kernel: Inode-cache hash table entries: 1048576 (order: 11, 8388608 bytes)
May 16 00:11:24 client-18 kernel: Checking aperture...
May 16 00:11:24 client-18 kernel: PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
May 16 00:11:24 client-18 kernel: Placing software IO TLB between 0x28e4000 - 0x68e4000
May 16 00:11:24 client-18 kernel: Memory: 12287648k/13631488k available (2574k kernel code, 285968k reserved, 1304k data, 212k init)
May 16 00:11:24 client-18 kernel: Calibrating delay loop (skipped), value calculated using timer frequency.. 4533.60 BogoMIPS (lpj=2266804)
May 16 00:11:24 client-18 kernel: Security Framework v1.0.0 initialized
May 16 00:11:24 client-18 kernel: SELinux:  Initializing.
May 16 00:11:24 client-18 kernel: selinux_register_security:  Registering secondary module capability
May 16 00:11:24 client-18 kernel: Capability LSM initialized as secondary
May 16 00:11:24 client-18 kernel: Mount-cache hash table entries: 256
May 16 00:11:24 client-18 kernel: CPU: L1 I cache: 32K, L1 D cache: 32K
May 16 00:11:24 client-18 kernel: CPU: L2 cache: 256K
May 16 00:11:24 client-18 kernel: CPU: L3 cache: 4096K
May 16 00:11:24 client-18 kernel: using mwait in idle threads.
May 16 00:11:24 client-18 kernel: CPU: Physical Processor ID: 0
May 16 00:11:24 client-18 kernel: CPU: Processor Core ID: 0
May 16 00:11:24 client-18 kernel: CPU0: Thermal monitoring enabled (TM1)
May 16 00:11:24 client-18 kernel: SMP alternatives: switching to UP code
May 16 00:11:24 client-18 kernel: ACPI: Core revision 20060707
May 16 00:11:24 client-18 kernel: Using local APIC timer interrupts.
May 16 00:11:24 client-18 kernel: Detected 8.333 MHz APIC timer.
May 16 00:11:24 client-18 kernel: SMP alternatives: switching to SMP code
May 16 00:11:24 client-18 kernel: Booting processor 1/4 APIC 0x2
May 16 00:11:24 client-18 kernel: Initializing CPU#1
May 16 00:11:24 client-18 kernel: Calibrating delay using timer specific routine.. 4533.46 BogoMIPS (lpj=2266734)
May 16 00:11:24 client-18 kernel: CPU: L1 I cache: 32K, L1 D cache: 32K
May 16 00:11:24 client-18 kernel: CPU: L2 cache: 256K
May 16 00:11:24 client-18 kernel: CPU: L3 cache: 4096K
May 16 00:11:24 client-18 kernel: CPU: Physical Processor ID: 0
May 16 00:11:24 client-18 kernel: CPU: Processor Core ID: 1
May 16 00:11:24 client-18 kernel: CPU1: Thermal monitoring enabled (TM1)
May 16 00:11:24 client-18 kernel: Intel(R) Xeon(R) CPU           E5507  @ 2.27GHz stepping 05
May 16 00:11:24 client-18 kernel: SMP alternatives: switching to SMP code
May 16 00:11:24 client-18 kernel: Booting processor 2/4 APIC 0x4
May 16 00:11:24 client-18 kernel: Initializing CPU#2
May 16 00:11:24 client-18 kernel: Calibrating delay using timer specific routine.. 4533.40 BogoMIPS (lpj=2266703)
May 16 00:11:24 client-18 kernel: CPU: L1 I cache: 32K, L1 D cache: 32K
May 16 00:11:24 client-18 kernel: CPU: L2 cache: 256K
May 16 00:11:24 client-18 kernel: CPU: L3 cache: 4096K
May 16 00:11:24 client-18 kernel: CPU: Physical Processor ID: 0
May 16 00:11:24 client-18 kernel: CPU: Processor Core ID: 2
May 16 00:11:24 client-18 kernel: CPU2: Thermal monitoring enabled (TM1)
May 16 00:11:24 client-18 kernel: Intel(R) Xeon(R) CPU           E5507  @ 2.27GHz stepping 05
May 16 00:11:24 client-18 kernel: SMP alternatives: switching to SMP code
May 16 00:11:24 client-18 kernel: Booting processor 3/4 APIC 0x6
May 16 00:11:24 client-18 kernel: Initializing CPU#3
May 16 00:11:24 client-18 kernel: Calibrating delay using timer specific routine.. 4533.40 BogoMIPS (lpj=2266704)
May 16 00:11:24 client-18 kernel: CPU: L1 I cache: 32K, L1 D cache: 32K
May 16 00:11:24 client-18 kernel: CPU: L2 cache: 256K
May 16 00:11:24 client-18 kernel: CPU: L3 cache: 4096K
May 16 00:11:24 client-18 kernel: CPU: Physical Processor ID: 0
May 16 00:11:24 client-18 kernel: CPU: Processor Core ID: 3
May 16 00:11:24 client-18 kernel: CPU3: Thermal monitoring enabled (TM1)
May 16 00:11:24 client-18 kernel: Intel(R) Xeon(R) CPU           E5507  @ 2.27GHz stepping 05
May 16 00:11:24 client-18 kernel: Brought up 4 CPUs
May 16 00:11:24 client-18 kernel: testing NMI watchdog ... OK.
May 16 00:11:24 client-18 kernel: time.c: Using 14.318180 MHz WALL HPET GTOD HPET/TSC timer.
May 16 00:11:24 client-18 kernel: time.c: Detected 2266.804 MHz processor.
May 16 00:11:24 client-18 kernel: migration_cost=20
May 16 00:11:24 client-18 kernel: checking if image is initramfs... it is
May 16 00:11:24 client-18 kernel: Freeing initrd memory: 2611k freed
May 16 00:11:24 client-18 kernel: NET: Registered protocol family 16
May 16 00:11:24 client-18 kernel: ACPI: bus type pci registered
May 16 00:11:24 client-18 kernel: PCI: Using MMCONFIG at e0000000
May 16 00:11:24 client-18 kernel: ACPI: Interpreter enabled
May 16 00:11:24 client-18 kernel: ACPI: Using IOAPIC for interrupt routing
May 16 00:11:24 client-18 kernel: ACPI: No dock devices found.
May 16 00:11:24 client-18 kernel: ACPI: PCI Root Bridge [PCI0] (0000:00)
May 16 00:11:24 client-18 kernel: PCI: Transparent bridge - 0000:00:1e.0
May 16 00:11:24 client-18 kernel: ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 7 10 11 12 14 *15)
May 16 00:11:24 client-18 kernel: ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 *5 6 7 10 11 12 14 15)
May 16 00:11:24 client-18 kernel: ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 5 6 7 *10 11 12 14 15)
May 16 00:11:24 client-18 kernel: ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 6 7 10 *11 12 14 15)
May 16 00:11:24 client-18 kernel: ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 5 6 7 10 11 12 14 15) *0, disabled.
May 16 00:11:24 client-18 kernel: ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 5 6 7 10 11 12 *14 15)
May 16 00:11:24 client-18 kernel: ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 5 6 7 10 11 12 14 15) *0, disabled.
May 16 00:11:24 client-18 kernel: ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 5 6 *7 10 11 12 14 15)
May 16 00:11:24 client-18 kernel: Linux Plug and Play Support v0.97 (c) Adam Belay
May 16 00:11:24 client-18 kernel: pnp: PnP ACPI init
May 16 00:11:24 client-18 kernel: pnp: PnP ACPI: found 15 devices
May 16 00:11:24 client-18 kernel: usbcore: registered new driver usbfs
May 16 00:11:24 client-18 kernel: usbcore: registered new driver hub
May 16 00:11:24 client-18 kernel: PCI: Using ACPI for IRQ routing
May 16 00:11:24 client-18 kernel: PCI: If a device doesn't work, try "pci=routeirq".  If it helps, post a report
May 16 00:11:24 client-18 kernel: NetLabel: Initializing
May 16 00:11:24 client-18 kernel: NetLabel:  domain hash size = 128
May 16 00:11:24 client-18 kernel: NetLabel:  protocols = UNLABELED CIPSOv4
May 16 00:11:24 client-18 kernel: NetLabel:  unlabeled traffic allowed by default
May 16 00:11:24 client-18 kernel: hpet0: at MMIO 0xfed00000 (virtual 0xffffffffff5fe000), IRQs 2, 8, 0, 0
May 16 00:11:24 client-18 kernel: hpet0: 4 64-bit timers, 14318180 Hz
May 16 00:11:24 client-18 kernel: DMAR:Host address width 40
May 16 00:11:24 client-18 kernel: DMAR:DRHD base: 0x000000fbffe000 flags: 0x1
May 16 00:11:24 client-18 kernel: IOMMU fbffe000: ver 1:0 cap c90780106f0462 ecap f020f6
May 16 00:11:24 client-18 kernel: DMAR:RMRR base: 0x000000000e6000 end: 0x000000000e9fff
May 16 00:11:24 client-18 kernel: DMAR:RMRR base: 0x000000bf7ec000 end: 0x000000bf7fffff
May 16 00:11:24 client-18 kernel: DMAR:ATSR flags: 0x0
May 16 00:11:24 client-18 kernel: PCI-GART: No AMD northbridge found.
May 16 00:11:24 client-18 kernel: pnp: 00:08: ioport range 0x164e-0x164f has been reserved
May 16 00:11:24 client-18 kernel: pnp: 00:08: ioport range 0xa00-0xa0f has been reserved
May 16 00:11:24 client-18 kernel: pnp: 00:0c: ioport range 0xca2-0xca3 has been reserved
May 16 00:11:24 client-18 kernel: PCI: Bridge: 0000:00:01.0
May 16 00:11:24 client-18 kernel:   IO window: e000-efff
May 16 00:11:24 client-18 kernel:   MEM window: fbc00000-fbcfffff
May 16 00:11:24 client-18 kernel:   PREFETCH window: disabled.
May 16 00:11:24 client-18 kernel: PCI: Bridge: 0000:00:03.0
May 16 00:11:24 client-18 kernel:   IO window: disabled.
May 16 00:11:24 client-18 kernel:   MEM window: fbd00000-fbdfffff
May 16 00:11:24 client-18 kernel:   PREFETCH window 0x00000000f8800000-0x00000000f8ffffff
May 16 00:11:24 client-18 kernel: PCI: Bridge: 0000:00:05.0
May 16 00:11:24 client-18 kernel:   IO window: disabled.
May 16 00:11:24 client-18 kernel:   MEM window: disabled.
May 16 00:11:24 client-18 kernel:   PREFETCH window: disabled.
May 16 00:11:24 client-18 kernel: PCI: Bridge: 0000:00:07.0
May 16 00:11:24 client-18 kernel:   IO window: disabled.
May 16 00:11:24 client-18 kernel:   MEM window: disabled.
May 16 00:11:24 client-18 kernel:   PREFETCH window: disabled.
May 16 00:11:24 client-18 kernel: PCI: Bridge: 0000:00:1e.0
May 16 00:11:24 client-18 kernel:   IO window: disabled.
May 16 00:11:24 client-18 kernel:   MEM window: faf00000-fb7fffff
May 16 00:11:24 client-18 rpc.statd[3141]: Version 1.0.9 Starting
May 16 00:11:24 client-18 kernel:   PREFETCH window 0x00000000f9000000-0x00000000f9ffffff
May 16 00:11:24 client-18 kernel: NET: Registered protocol family 2
May 16 00:11:24 client-18 kernel: IP route cache hash table entries: 524288 (order: 10, 4194304 bytes)
May 16 00:11:24 client-18 kernel: TCP established hash table entries: 262144 (order: 10, 4194304 bytes)
May 16 00:11:24 client-18 kernel: TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
May 16 00:11:24 client-18 kernel: TCP: Hash tables configured (established 262144 bind 65536)
May 16 00:11:24 client-18 kernel: TCP reno registered
May 16 00:11:24 client-18 kernel: audit: initializing netlink socket (disabled)
May 16 00:11:24 client-18 kernel: type=2000 audit(1305504630.604:1): initialized
May 16 00:11:24 client-18 kernel: Total HugeTLB memory allocated, 0
May 16 00:11:24 client-18 kernel: VFS: Disk quotas dquot_6.5.1
May 16 00:11:24 client-18 kernel: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
May 16 00:11:24 client-18 kernel: Initializing Cryptographic API
May 16 00:11:24 client-18 kernel: alg: No test for crc32c (crc32c-generic)
May 16 00:11:24 client-18 kernel: ksign: Installing public key data
May 16 00:11:24 client-18 kernel: Loading keyring
May 16 00:11:24 client-18 kernel: - Added public key 565662CFF69DD0BE
May 16 00:11:24 client-18 kernel: - User ID: CentOS (Kernel Module GPG key)
May 16 00:11:24 client-18 kernel: io scheduler noop registered
May 16 00:11:24 client-18 kernel: io scheduler anticipatory registered
May 16 00:11:24 client-18 rpc.statd[3141]: statd running as root. chown /var/lib/nfs/statd/sm to choose different user 
May 16 00:11:24 client-18 kernel: io scheduler deadline registered
May 16 00:11:24 client-18 kernel: io scheduler cfq registered (default)
May 16 00:11:24 client-18 kernel: pci_hotplug: PCI Hot Plug PCI Core version: 0.5
May 16 00:11:24 client-18 kernel: ACPI (exconfig-0456): Dynamic SSDT Load - OemId [DpgPmm] OemTableId [ P001Ist] [20060707]
May 16 00:11:24 client-18 kernel: ACPI (exconfig-0456): Dynamic SSDT Load - OemId [ PmRef] OemTableId [ P001Cst] [20060707]
May 16 00:11:24 client-18 kernel: ACPI: CPU0 (power states: C1[C1] C2[C2] C3[C3])
May 16 00:11:24 client-18 kernel: ACPI: CPU1 (power states: C1[C1] C2[C2] C3[C3])
May 16 00:11:24 client-18 kernel: ACPI: CPU2 (power states: C1[C1] C2[C2] C3[C3])
May 16 00:11:24 client-18 kernel: ACPI: CPU3 (power states: C1[C1] C2[C2] C3[C3])
May 16 00:11:24 client-18 kernel: Real Time Clock Driver v1.12ac
May 16 00:11:24 client-18 kernel: Non-volatile memory driver v1.2
May 16 00:11:24 client-18 kernel: Linux agpgart interface v0.101 (c) Dave Jones
May 16 00:11:24 client-18 kernel: Serial: 8250/16550 driver $Revision: 1.90 $ 4 ports, IRQ sharing enabled
May 16 00:11:24 client-18 kernel: serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
May 16 00:11:24 client-18 kernel: serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
May 16 00:11:24 client-18 kernel: 00:06: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
May 16 00:11:24 client-18 kernel: 00:07: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
May 16 00:11:24 client-18 kernel: brd: module loaded
May 16 00:11:24 client-18 kernel: Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
May 16 00:11:24 client-18 kernel: ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
May 16 00:11:24 client-18 kernel: ide-floppy driver 0.99.newide
May 16 00:11:24 client-18 kernel: usbcore: registered new driver hiddev
May 16 00:11:24 client-18 kernel: usbcore: registered new driver usbhid
May 16 00:11:24 client-18 kernel: drivers/usb/input/hid-core.c: v2.6:USB HID core driver
May 16 00:11:24 client-18 kernel: PNP: No PS/2 controller found. Probing ports directly.
May 16 00:11:24 client-18 kernel: serio: i8042 KBD port at 0x60,0x64 irq 1
May 16 00:11:24 client-18 kernel: serio: i8042 AUX port at 0x60,0x64 irq 12
May 16 00:11:24 client-18 kernel: mice: PS/2 mouse device common for all mice
May 16 00:11:24 client-18 kernel: md: md driver 0.90.3 MAX_MD_DEVS=256, MD_SB_DISKS=27
May 16 00:11:24 client-18 kernel: md: bitmap version 4.39
May 16 00:11:24 client-18 kernel: TCP bic registered
May 16 00:11:24 client-18 kernel: Initializing IPsec netlink socket
May 16 00:11:24 client-18 kernel: NET: Registered protocol family 1
May 16 00:11:24 client-18 kernel: NET: Registered protocol family 17
May 16 00:11:24 client-18 kernel: ACPI: (supports S0 S1 S4 S5)
May 16 00:11:24 client-18 kernel: Initalizing network drop monitor service
May 16 00:11:24 client-18 kernel: Freeing unused kernel memory: 212k freed
May 16 00:11:24 client-18 kernel: Write protecting the kernel read-only data: 504k
May 16 00:11:24 client-18 kernel: GSI 16 sharing vector 0xD1 and IRQ 16
May 16 00:11:24 client-18 kernel: ACPI: PCI Interrupt 0000:00:1a.7[C] -> GSI 18 (level, low) -> IRQ 209
May 16 00:11:24 client-18 kernel: ehci_hcd 0000:00:1a.7: EHCI Host Controller
May 16 00:11:24 client-18 kernel: ehci_hcd 0000:00:1a.7: new USB bus registered, assigned bus number 1
May 16 00:11:24 client-18 kernel: ehci_hcd 0000:00:1a.7: debug port 1
May 16 00:11:24 client-18 kernel: ehci_hcd 0000:00:1a.7: irq 209, io mem 0xfbed6000
May 16 00:11:24 client-18 kernel: ehci_hcd 0000:00:1a.7: USB 2.0 started, EHCI 1.00, driver 10 Dec 2004
May 16 00:11:24 client-18 kernel: usb usb1: configuration #1 chosen from 1 choice
May 16 00:11:24 client-18 kernel: hub 1-0:1.0: USB hub found
May 16 00:11:24 client-18 kernel: hub 1-0:1.0: 6 ports detected
May 16 00:11:24 client-18 kernel: GSI 17 sharing vector 0xD9 and IRQ 17
May 16 00:11:24 client-18 kernel: ACPI: PCI Interrupt 0000:00:1d.7[A] -> GSI 23 (level, low) -> IRQ 217
May 16 00:11:24 client-18 kernel: ehci_hcd 0000:00:1d.7: EHCI Host Controller
May 16 00:11:24 client-18 kernel: ehci_hcd 0000:00:1d.7: new USB bus registered, assigned bus number 2
May 16 00:11:24 client-18 kernel: ehci_hcd 0000:00:1d.7: debug port 1
May 16 00:11:24 client-18 kernel: ehci_hcd 0000:00:1d.7: irq 217, io mem 0xfbed4000
May 16 00:11:24 client-18 kernel: ehci_hcd 0000:00:1d.7: USB 2.0 started, EHCI 1.00, driver 10 Dec 2004
May 16 00:11:24 client-18 kernel: usb usb2: configuration #1 chosen from 1 choice
May 16 00:11:24 client-18 kernel: hub 2-0:1.0: USB hub found
May 16 00:11:24 client-18 kernel: hub 2-0:1.0: 6 ports detected
May 16 00:11:24 client-18 kernel: USB Universal Host Controller Interface driver v3.0
May 16 00:11:24 client-18 kernel: GSI 18 sharing vector 0xE1 and IRQ 18
May 16 00:11:24 client-18 kernel: ACPI: PCI Interrupt 0000:00:1a.0[A] -> GSI 16 (level, low) -> IRQ 225
May 16 00:11:24 client-18 kernel: uhci_hcd 0000:00:1a.0: UHCI Host Controller
May 16 00:11:24 client-18 kernel: uhci_hcd 0000:00:1a.0: new USB bus registered, assigned bus number 3
May 16 00:11:24 client-18 kernel: uhci_hcd 0000:00:1a.0: irq 225, io base 0x0000c880
May 16 00:11:24 client-18 kernel: usb usb3: configuration #1 chosen from 1 choice
May 16 00:11:24 client-18 kernel: hub 3-0:1.0: USB hub found
May 16 00:11:24 client-18 kernel: hub 3-0:1.0: 2 ports detected
May 16 00:11:24 client-18 kernel: GSI 19 sharing vector 0xE9 and IRQ 19
May 16 00:11:24 client-18 kernel: ACPI: PCI Interrupt 0000:00:1a.1[B] -> GSI 21 (level, low) -> IRQ 233
May 16 00:11:24 client-18 kernel: uhci_hcd 0000:00:1a.1: UHCI Host Controller
May 16 00:11:24 client-18 kernel: uhci_hcd 0000:00:1a.1: new USB bus registered, assigned bus number 4
May 16 00:11:24 client-18 kernel: uhci_hcd 0000:00:1a.1: irq 233, io base 0x0000cc00
May 16 00:11:24 client-18 kernel: usb usb4: configuration #1 chosen from 1 choice
May 16 00:11:24 client-18 kernel: hub 4-0:1.0: USB hub found
May 16 00:11:24 client-18 kernel: hub 4-0:1.0: 2 ports detected
May 16 00:11:24 client-18 kernel: GSI 20 sharing vector 0x32 and IRQ 20
May 16 00:11:24 client-18 kernel: ACPI: PCI Interrupt 0000:00:1a.2[D] -> GSI 19 (level, low) -> IRQ 50
May 16 00:11:24 client-18 kernel: uhci_hcd 0000:00:1a.2: UHCI Host Controller
May 16 00:11:24 client-18 kernel: uhci_hcd 0000:00:1a.2: new USB bus registered, assigned bus number 5
May 16 00:11:24 client-18 kernel: uhci_hcd 0000:00:1a.2: irq 50, io base 0x0000d000
May 16 00:11:24 client-18 kernel: usb usb5: configuration #1 chosen from 1 choice
May 16 00:11:24 client-18 kernel: hub 5-0:1.0: USB hub found
May 16 00:11:24 client-18 kernel: hub 5-0:1.0: 2 ports detected
May 16 00:11:24 client-18 kernel: ACPI: PCI Interrupt 0000:00:1d.0[A] -> GSI 23 (level, low) -> IRQ 217
May 16 00:11:24 client-18 kernel: uhci_hcd 0000:00:1d.0: UHCI Host Controller
May 16 00:11:24 client-18 kernel: uhci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 6
May 16 00:11:24 client-18 kernel: uhci_hcd 0000:00:1d.0: irq 217, io base 0x0000c400
May 16 00:11:24 client-18 kernel: usb usb6: configuration #1 chosen from 1 choice
May 16 00:11:24 client-18 kernel: hub 6-0:1.0: USB hub found
May 16 00:11:24 client-18 kernel: hub 6-0:1.0: 2 ports detected
May 16 00:11:25 client-18 kernel: usb 4-1: new full speed USB device using uhci_hcd and address 2
May 16 00:11:25 client-18 kernel: ACPI: PCI Interrupt 0000:00:1d.1[B] -> GSI 19 (level, low) -> IRQ 50
May 16 00:11:25 client-18 kernel: uhci_hcd 0000:00:1d.1: UHCI Host Controller
May 16 00:11:25 client-18 kernel: uhci_hcd 0000:00:1d.1: new USB bus registered, assigned bus number 7
May 16 00:11:25 client-18 kernel: uhci_hcd 0000:00:1d.1: irq 50, io base 0x0000c480
May 16 00:11:25 client-18 kernel: usb usb7: configuration #1 chosen from 1 choice
May 16 00:11:25 client-18 kernel: hub 7-0:1.0: USB hub found
May 16 00:11:25 client-18 kernel: hub 7-0:1.0: 2 ports detected
May 16 00:11:25 client-18 kernel: usb 4-1: configuration #1 chosen from 1 choice
May 16 00:11:25 client-18 kernel: ACPI: PCI Interrupt 0000:00:1d.2[C] -> GSI 18 (level, low) -> IRQ 209
May 16 00:11:25 client-18 kernel: uhci_hcd 0000:00:1d.2: UHCI Host Controller
May 16 00:11:25 client-18 kernel: uhci_hcd 0000:00:1d.2: new USB bus registered, assigned bus number 8
May 16 00:11:25 client-18 kernel: input: American Megatrends Inc. Virtual Keyboard and Mouse as /class/input/input0
May 16 00:11:25 client-18 kernel: input: USB HID v1.10 Keyboard [American Megatrends Inc. Virtual Keyboard and Mouse] on usb-0000:00:1a.1-1
May 16 00:11:25 client-18 kernel: input: American Megatrends Inc. Virtual Keyboard and Mouse as /class/input/input1
May 16 00:11:25 client-18 kernel: input: USB HID v1.10 Mouse [American Megatrends Inc. Virtual Keyboard and Mouse] on usb-0000:00:1a.1-1
May 16 00:11:25 client-18 kernel: uhci_hcd 0000:00:1d.2: irq 209, io base 0x0000c800
May 16 00:11:25 client-18 kernel: usb usb8: configuration #1 chosen from 1 choice
May 16 00:11:25 client-18 kernel: hub 8-0:1.0: USB hub found
May 16 00:11:25 client-18 kernel: hub 8-0:1.0: 2 ports detected
May 16 00:11:25 client-18 kernel: SCSI subsystem initialized
May 16 00:11:25 client-18 kernel: ACPI: PCI Interrupt 0000:00:1f.2[B] -> GSI 19 (level, low) -> IRQ 50
May 16 00:11:25 client-18 kernel: ahci 0000:00:1f.2: AHCI 0001.0200 32 slots 6 ports 3 Gbps 0x3f impl SATA mode
May 16 00:11:25 client-18 kernel: ahci 0000:00:1f.2: flags: 64bit ncq sntf stag pm led clo pio slum part ems 
May 16 00:11:25 client-18 kernel: scsi0 : ahci
May 16 00:11:25 client-18 kernel: scsi1 : ahci
May 16 00:11:25 client-18 kernel: scsi2 : ahci
May 16 00:11:25 client-18 kernel: scsi3 : ahci
May 16 00:11:25 client-18 kernel: scsi4 : ahci
May 16 00:11:25 client-18 kernel: scsi5 : ahci
May 16 00:11:25 client-18 kernel: ata1: SATA max UDMA/133 abar m2048@0xfbefa000 port 0xfbefa100 irq 58
May 16 00:11:25 client-18 kernel: ata2: SATA max UDMA/133 abar m2048@0xfbefa000 port 0xfbefa180 irq 58
May 16 00:11:25 client-18 kernel: ata3: SATA max UDMA/133 abar m2048@0xfbefa000 port 0xfbefa200 irq 58
May 16 00:11:25 client-18 kernel: ata4: SATA max UDMA/133 abar m2048@0xfbefa000 port 0xfbefa280 irq 58
May 16 00:11:25 client-18 kernel: ata5: SATA max UDMA/133 abar m2048@0xfbefa000 port 0xfbefa300 irq 58
May 16 00:11:25 client-18 kernel: ata6: SATA max UDMA/133 abar m2048@0xfbefa000 port 0xfbefa380 irq 58
May 16 00:11:25 client-18 kernel: ata1: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
May 16 00:11:25 client-18 kernel: ata1.00: ATA-8: WDC WD2502ABYS-02B7A0, 02.03B03, max UDMA/133
May 16 00:11:25 client-18 kernel: ata1.00: 490350672 sectors, multi 0: LBA48 NCQ (depth 31/32)
May 16 00:11:25 client-18 kernel: ata1.00: configured for UDMA/133
May 16 00:11:25 client-18 kernel: ata2: SATA link down (SStatus 0 SControl 300)
May 16 00:11:25 client-18 kernel: ata3: SATA link down (SStatus 0 SControl 300)
May 16 00:11:25 client-18 kernel: ata4: SATA link down (SStatus 0 SControl 300)
May 16 00:11:25 client-18 kernel: ata5: SATA link down (SStatus 0 SControl 300)
May 16 00:11:25 client-18 kernel: ata6: SATA link down (SStatus 0 SControl 300)
May 16 00:11:25 client-18 kernel:   Vendor: ATA       Model: WDC WD2502ABYS-0  Rev: 02.0
May 16 00:11:25 client-18 kernel:   Type:   Direct-Access                      ANSI SCSI revision: 05
May 16 00:11:25 client-18 kernel: SCSI device sda: 490350672 512-byte hdwr sectors (251060 MB)
May 16 00:11:25 client-18 kernel: sda: Write Protect is off
May 16 00:11:25 client-18 kernel: SCSI device sda: drive cache: write back
May 16 00:11:25 client-18 kernel: SCSI device sda: 490350672 512-byte hdwr sectors (251060 MB)
May 16 00:11:25 client-18 kernel: sda: Write Protect is off
May 16 00:11:25 client-18 kernel: SCSI device sda: drive cache: write back
May 16 00:11:25 client-18 kernel:  sda: sda1 sda2
May 16 00:11:25 client-18 kernel: sd 0:0:0:0: Attached scsi disk sda
May 16 00:11:25 client-18 kernel: device-mapper: uevent: version 1.0.3
May 16 00:11:25 client-18 kernel: device-mapper: ioctl: 4.11.5-ioctl (2007-12-12) initialised: dm-devel@redhat.com
May 16 00:11:25 client-18 kernel: device-mapper: dm-raid45: initialized v0.2594l
May 16 00:11:25 client-18 kernel: EXT3-fs: INFO: recovery required on readonly filesystem.
May 16 00:11:25 client-18 kernel: EXT3-fs: write access will be enabled during recovery.
May 16 00:11:25 client-18 kernel: kjournald starting.  Commit interval 5 seconds
May 16 00:11:25 client-18 kernel: EXT3-fs: recovery complete.
May 16 00:11:25 client-18 kernel: EXT3-fs: mounted filesystem with ordered data mode.
May 16 00:11:25 client-18 kernel: SELinux:  Disabled at runtime.
May 16 00:11:25 client-18 kernel: type=1404 audit(1305504657.503:2): selinux=0 auid=4294967295 ses=4294967295
May 16 00:11:25 client-18 kernel: input: PC Speaker as /class/input/input2
May 16 00:11:25 client-18 kernel: shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
May 16 00:11:25 client-18 kernel: dca service started, version 1.4
May 16 00:11:25 client-18 kernel: mlx4_core: Mellanox ConnectX core driver v1.0-ofed1.5.3 (January 19, 2011)
May 16 00:11:25 client-18 kernel: mlx4_core: Initializing 0000:02:00.0
May 16 00:11:25 client-18 kernel: GSI 21 sharing vector 0x42 and IRQ 21
May 16 00:11:25 client-18 kernel: ACPI: PCI Interrupt 0000:02:00.0[A] -> GSI 24 (level, low) -> IRQ 66
May 16 00:11:25 client-18 kernel: 802.1Q VLAN Support v1.8 Ben Greear <greearb@candelatech.com>
May 16 00:11:25 client-18 kernel: All bugs added by David S. Miller <davem@redhat.com>
May 16 00:11:25 client-18 kernel: sd 0:0:0:0: Attached scsi generic sg0 type 0
May 16 00:11:25 client-18 kernel: Intel(R) Gigabit Ethernet Network Driver - version 2.1.0-k2
May 16 00:11:25 client-18 kernel: Copyright (c) 2007-2009 Intel Corporation.
May 16 00:11:25 client-18 kernel: ACPI: PCI Interrupt 0000:00:1f.3[C] -> GSI 18 (level, low) -> IRQ 209
May 16 00:11:25 client-18 kernel: GSI 22 sharing vector 0x53 and IRQ 22
May 16 00:11:25 client-18 kernel: ACPI: PCI Interrupt 0000:01:00.0[A] -> GSI 28 (level, low) -> IRQ 83
May 16 00:11:25 client-18 kernel: mlx4_ib: Mellanox ConnectX InfiniBand driver v1.0-ofed1.5.3 (January 19, 2011)
May 16 00:11:25 client-18 kernel: igb 0000:01:00.0: Intel(R) Gigabit Ethernet Network Connection
May 16 00:11:25 client-18 kernel: igb 0000:01:00.0: eth0: (PCIe:2.5Gb/s:Width x4) 00:25:90:1d:3e:22
May 16 00:11:25 client-18 kernel: igb 0000:01:00.0: eth0: PBA No: ffffff-0ff
May 16 00:11:25 client-18 kernel: igb 0000:01:00.0: Using MSI-X interrupts. 4 rx queue(s), 1 tx queue(s)
May 16 00:11:25 client-18 kernel: GSI 23 sharing vector 0x8B and IRQ 23
May 16 00:11:25 client-18 kernel: ACPI: PCI Interrupt 0000:01:00.1[B] -> GSI 40 (level, low) -> IRQ 139
May 16 00:11:25 client-18 kernel: igb 0000:01:00.1: Intel(R) Gigabit Ethernet Network Connection
May 16 00:11:25 client-18 kernel: igb 0000:01:00.1: eth1: (PCIe:2.5Gb/s:Width x4) 00:25:90:1d:3e:23
May 16 00:11:25 client-18 kernel: igb 0000:01:00.1: eth1: PBA No: ffffff-0ff
May 16 00:11:25 client-18 kernel: igb 0000:01:00.1: Using MSI-X interrupts. 4 rx queue(s), 1 tx queue(s)
May 16 00:11:25 client-18 kernel: floppy0: no floppy controllers found
May 16 00:11:25 client-18 kernel: lp: driver loaded but no devices found
May 16 00:11:25 client-18 kernel: ACPI: Power Button (FF) [PWRF]
May 16 00:11:25 client-18 kernel: ACPI: Power Button (CM) [PWRB]
May 16 00:11:25 client-18 kernel: ACPI: Mapper loaded
May 16 00:11:25 client-18 kernel: dell-wmi: No known WMI GUID found
May 16 00:11:25 client-18 kernel: md: Autodetecting RAID arrays.
May 16 00:11:25 client-18 kernel: md: autorun ...
May 16 00:11:25 client-18 kernel: md: ... autorun DONE.
May 16 00:11:25 client-18 kernel: device-mapper: multipath: version 1.0.5 loaded
May 16 00:11:25 client-18 kernel: loop: loaded (max 8 devices)
May 16 00:11:25 client-18 kernel: EXT3 FS on sda1, internal journal
May 16 00:11:25 client-18 kernel: Adding 14346036k swap on /dev/sda2.  Priority:-1 extents:1 across:14346036k
May 16 00:11:25 client-18 kernel: IA-32 Microcode Update Driver: v1.14a <tigran@veritas.com>
May 16 00:11:25 client-18 kernel: mlx4_en: Mellanox ConnectX HCA Ethernet driver v1.5.3 (Jan 2011)
May 16 00:11:25 client-18 kernel: mlx4_en 0000:02:00.0: UDP RSS is not supported on this device.
May 16 00:11:25 client-18 kernel: NET: Registered protocol family 10
May 16 00:11:25 client-18 kernel: lo: Disabled Privacy Extensions
May 16 00:11:25 client-18 kernel: IPv6 over IPv4 tunneling driver
May 16 00:11:25 client-18 kernel: ADDRCONF(NETDEV_UP): ib0: link is not ready
May 16 00:11:25 client-18 kernel: ib0: enabling connected mode will cause multicast packet drops
May 16 00:11:25 client-18 kernel: ib0: mtu > 2044 will cause multicast packet drops.
May 16 00:11:25 client-18 kernel: ib0: mtu > 2044 will cause multicast packet drops.
May 16 00:11:25 client-18 kernel: NET: Registered protocol family 27
May 16 00:11:25 client-18 kernel: NET: Registered protocol family 28
May 16 00:11:25 client-18 kernel: QLGC_VNIC: Initializing QLogic Corp. Virtual NIC (VNIC) driver version 1.3.0.0.4
May 16 00:11:25 client-18 kernel: Loading iSCSI transport class v2.0-871.
May 16 00:11:25 client-18 kernel: cxgb3i: disagrees about version of symbol cxgb3_register_client
May 16 00:11:25 client-18 kernel: cxgb3i: Unknown symbol cxgb3_register_client
May 16 00:11:25 client-18 kernel: cxgb3i: disagrees about version of symbol cxgb3_alloc_atid
May 16 00:11:25 client-18 kernel: cxgb3i: Unknown symbol cxgb3_alloc_atid
May 16 00:11:25 client-18 kernel: cxgb3i: disagrees about version of symbol t3_l2t_get
May 16 00:11:25 client-18 kernel: cxgb3i: Unknown symbol t3_l2t_get
May 16 00:11:25 client-18 kernel: cxgb3i: disagrees about version of symbol cxgb3_insert_tid
May 16 00:11:25 client-18 kernel: cxgb3i: Unknown symbol cxgb3_insert_tid
May 16 00:11:25 client-18 kernel: cxgb3i: disagrees about version of symbol t3_l2e_free
May 16 00:11:25 client-18 kernel: cxgb3i: Unknown symbol t3_l2e_free
May 16 00:11:25 client-18 kernel: cxgb3i: disagrees about version of symbol t3_l2t_send_slow
May 16 00:11:25 client-18 kernel: cxgb3i: Unknown symbol t3_l2t_send_slow
May 16 00:11:25 client-18 kernel: cxgb3i: disagrees about version of symbol cxgb3_unregister_client
May 16 00:11:25 client-18 kernel: cxgb3i: Unknown symbol cxgb3_unregister_client
May 16 00:11:25 client-18 kernel: Broadcom NetXtreme II CNIC Driver cnic v2.1.0 (Oct 10, 2009)
May 16 00:11:25 client-18 kernel: Broadcom NetXtreme II iSCSI Driver bnx2i v2.1.0 (Dec 06, 2009)
May 16 00:11:25 client-18 kernel: iscsi: registered transport (bnx2i)
May 16 00:11:25 client-18 kernel: iscsi: registered transport (tcp)
May 16 00:11:25 client-18 hcid[3184]: Bluetooth HCI daemon
May 16 00:11:25 client-18 kernel: ib_iser: disagrees about version of symbol ib_fmr_pool_unmap
May 16 00:11:25 client-18 kernel: ib_iser: Unknown symbol ib_fmr_pool_unmap
May 16 00:11:25 client-18 kernel: ib_iser: disagrees about version of symbol ib_create_cq
May 16 00:11:25 client-18 kernel: ib_iser: Unknown symbol ib_create_cq
May 16 00:11:25 client-18 kernel: ib_iser: disagrees about version of symbol rdma_resolve_addr
May 16 00:11:25 client-18 kernel: ib_iser: Unknown symbol rdma_resolve_addr
May 16 00:11:25 client-18 kernel: ib_iser: disagrees about version of symbol ib_create_fmr_pool
May 16 00:11:25 client-18 kernel: ib_iser: Unknown symbol ib_create_fmr_pool
May 16 00:11:25 client-18 kernel: ib_iser: disagrees about version of symbol ib_dereg_mr
May 16 00:11:25 client-18 kernel: ib_iser: Unknown symbol ib_dereg_mr
May 16 00:11:25 client-18 kernel: ib_iser: disagrees about version of symbol rdma_disconnect
May 16 00:11:25 client-18 kernel: ib_iser: Unknown symbol rdma_disconnect
May 16 00:11:25 client-18 kernel: ib_iser: disagrees about version of symbol rdma_resolve_route
May 16 00:11:25 client-18 kernel: ib_iser: Unknown symbol rdma_resolve_route
May 16 00:11:25 client-18 kernel: ib_iser: disagrees about version of symbol rdma_create_qp
May 16 00:11:25 client-18 kernel: ib_iser: Unknown symbol rdma_create_qp
May 16 00:11:25 client-18 kernel: ib_iser: disagrees about version of symbol ib_destroy_cq
May 16 00:11:25 client-18 kernel: ib_iser: Unknown symbol ib_destroy_cq
May 16 00:11:25 client-18 kernel: ib_iser: disagrees about version of symbol rdma_create_id
May 16 00:11:25 client-18 kernel: ib_iser: Unknown symbol rdma_create_id
May 16 00:11:25 client-18 kernel: ib_iser: disagrees about version of symbol rdma_destroy_qp
May 16 00:11:25 client-18 kernel: ib_iser: Unknown symbol rdma_destroy_qp
May 16 00:11:25 client-18 kernel: ib_iser: disagrees about version of symbol ib_get_dma_mr
May 16 00:11:25 client-18 kernel: ib_iser: Unknown symbol ib_get_dma_mr
May 16 00:11:25 client-18 kernel: ib_iser: disagrees about version of symbol ib_alloc_pd
May 16 00:11:25 client-18 kernel: ib_iser: Unknown symbol ib_alloc_pd
May 16 00:11:25 client-18 sdpd[3190]: Bluetooth SDP daemon 
May 16 00:11:25 client-18 kernel: ib_iser: disagrees about version of symbol rdma_connect
May 16 00:11:25 client-18 hcid[3184]: Register path:/org/bluez fallback:1
May 16 00:11:25 client-18 kernel: ib_iser: Unknown symbol rdma_connect
May 16 00:11:25 client-18 kernel: ib_iser: disagrees about version of symbol rdma_destroy_id
May 16 00:11:25 client-18 kernel: ib_iser: Unknown symbol rdma_destroy_id
May 16 00:11:25 client-18 kernel: ib_iser: disagrees about version of symbol ib_dealloc_pd
May 16 00:11:25 client-18 kernel: ib_iser: Unknown symbol ib_dealloc_pd
May 16 00:11:25 client-18 kernel: ib_iser: disagrees about version of symbol ib_fmr_pool_map_phys
May 16 00:11:25 client-18 kernel: ib_iser: Unknown symbol ib_fmr_pool_map_phys
May 16 00:11:25 client-18 kernel: iscsi: registered transport (be2iscsi)
May 16 00:11:25 client-18 kernel: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
May 16 00:11:25 client-18 kernel: ADDRCONF(NETDEV_UP): eth1: link is not ready
May 16 00:11:25 client-18 kernel: ADDRCONF(NETDEV_CHANGE): ib0: link becomes ready
May 16 00:11:25 client-18 kernel: Bluetooth: Core ver 2.10
May 16 00:11:25 client-18 kernel: NET: Registered protocol family 31
May 16 00:11:25 client-18 kernel: Bluetooth: HCI device and connection manager initialized
May 16 00:11:25 client-18 kernel: Bluetooth: HCI socket layer initialized
May 16 00:11:25 client-18 kernel: Bluetooth: L2CAP ver 2.8
May 16 00:11:25 client-18 kernel: Bluetooth: L2CAP socket layer initialized
May 16 00:11:25 client-18 kernel: Bluetooth: RFCOMM socket layer initialized
May 16 00:11:25 client-18 kernel: Bluetooth: RFCOMM TTY layer initialized
May 16 00:11:25 client-18 kernel: Bluetooth: RFCOMM ver 1.8
May 16 00:11:25 client-18 pcscd: pcscdaemon.c:507:main() pcsc-lite 1.4.4 daemon ready.
May 16 00:11:25 client-18 pcscd: hotplug_libusb.c:402:HPEstablishUSBNotifications() Driver ifd-egate.bundle does not support IFD_GENERATE_HOTPLUG. Using active polling instead.
May 16 00:11:25 client-18 pcscd: hotplug_libusb.c:411:HPEstablishUSBNotifications() Polling forced every 1 second(s)
May 16 00:11:26 client-18 kernel: Bluetooth: HIDP (Human Interface Emulation) ver 1.1
May 16 00:11:26 client-18 hidd[3314]: Bluetooth HID daemon
May 16 00:11:27 client-18 automount[3343]: lookup_read_master: lookup(nisplus): couldn't locate nis+ table auto.master
May 16 00:11:27 client-18 automount[3343]: lookup_read_map: lookup(nisplus): couldn't locate nis+ table auto.direct
May 16 00:11:27 client-18 sssd: Starting up
May 16 00:11:27 client-18 sssd[be[LAB.WHAMCLOUD.COM]]: Starting up
May 16 00:11:28 client-18 sssd[nss]: Starting up
May 16 00:11:28 client-18 sssd[pam]: Starting up
May 16 00:11:28 client-18 ntpd[3400]: ntpd 4.2.2p1@1.1570-o Sat Dec 19 00:56:13 UTC 2009 (1)
May 16 00:11:28 client-18 ntpd[3401]: precision = 1.000 usec
May 16 00:11:28 client-18 ntpd[3401]: Listening on interface wildcard, 0.0.0.0#123 Disabled
May 16 00:11:28 client-18 ntpd[3401]: Listening on interface wildcard, ::#123 Disabled
May 16 00:11:28 client-18 ntpd[3401]: Listening on interface lo, ::1#123 Enabled
May 16 00:11:28 client-18 ntpd[3401]: Listening on interface eth0, fe80::225:90ff:fe1d:3e22#123 Enabled
May 16 00:11:28 client-18 ntpd[3401]: Listening on interface ib0, fe80::225:90ff:ff1b:6709#123 Enabled
May 16 00:11:28 client-18 ntpd[3401]: Listening on interface lo, 127.0.0.1#123 Enabled
May 16 00:11:28 client-18 ntpd[3401]: Listening on interface eth0, 10.10.4.18#123 Enabled
May 16 00:11:28 client-18 ntpd[3401]: Listening on interface eth1, 172.16.8.18#123 Enabled
May 16 00:11:28 client-18 ntpd[3401]: Listening on interface ib0, 192.168.4.18#123 Enabled
May 16 00:11:28 client-18 ntpd[3401]: kernel time sync status 0040
May 16 00:11:28 client-18 xinetd[3390]: xinetd Version 2.3.14 started with libwrap loadavg labeled-networking options compiled in.
May 16 00:11:28 client-18 xinetd[3390]: Started working: 2 available services
May 16 00:11:28 client-18 gpm[3436]: *** info [startup.c(95)]: 
May 16 00:11:28 client-18 gpm[3436]: Started gpm successfully. Entered daemon mode.
May 16 00:11:28 client-18 smartd[3502]: smartd version 5.38 [x86_64-redhat-linux-gnu] Copyright (C) 2002-8 Bruce Allen 
May 16 00:11:28 client-18 smartd[3502]: Home page is http://smartmontools.sourceforge.net/  
May 16 00:11:28 client-18 smartd[3502]: Opened configuration file /etc/smartd.conf 
May 16 00:11:28 client-18 smartd[3502]: Configuration file /etc/smartd.conf was parsed, found DEVICESCAN, scanning devices 
May 16 00:11:28 client-18 smartd[3502]: Problem creating device name scan list 
May 16 00:11:28 client-18 smartd[3502]: Device: /dev/sda, opened 
May 16 00:11:28 client-18 smartd[3502]: Device /dev/sda: using '-d sat' for ATA disk behind SAT layer. 
May 16 00:11:28 client-18 smartd[3502]: Device: /dev/sda, opened 
May 16 00:11:28 client-18 smartd[3502]: Device: /dev/sda, not found in smartd database. 
May 16 00:11:28 client-18 smartd[3502]: Device: /dev/sda, is SMART capable. Adding to "monitor" list. 
May 16 00:11:28 client-18 smartd[3502]: Monitoring 0 ATA and 1 SCSI devices 
May 16 00:11:29 client-18 smartd[3508]: smartd has fork()ed into background mode. New PID=3508. 
May 16 00:13:01 client-18 xinetd[3390]: START: shell pid=3583 from=192.168.4.5
May 16 00:13:02 client-18 rshd[3584]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:; lsof -t /mnt/lustre || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount  /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre is still busy, wait one second && sleep 1;     umount  /mnt/lustre; fi fi");echo XXRETCODE:$?'
May 16 00:13:02 client-18 xinetd[3390]: EXIT: shell status=0 pid=3583 duration=1(sec)
May 16 00:13:02 client-18 xinetd[3390]: START: shell pid=3620 from=192.168.4.5
May 16 00:13:02 client-18 rshd[3621]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:; lsof -t /mnt/lustre2 || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre2 | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount  /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre2 is still busy, wait one second && sleep 1;     umount  /mnt/lustre2; fi fi");echo XXRETCODE:$?'
May 16 00:13:02 client-18 xinetd[3390]: EXIT: shell status=0 pid=3620 duration=0(sec)
May 16 00:14:04 client-18 xinetd[3390]: START: shell pid=3660 from=192.168.4.5
May 16 00:14:04 client-18 rshd[3661]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); mpts=\$(mount | grep -w -c /mnt/lustre); if [ \$running -ne \$mpts ]; then     echo \$(hostname) env are INSANE!;     exit 1; fi");echo XXRETCODE:$?'
May 16 00:14:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=3660 duration=0(sec)
May 16 00:14:04 client-18 xinetd[3390]: START: shell pid=3700 from=192.168.4.5
May 16 00:14:04 client-18 rshd[3701]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); mpts=\$(mount | grep -w -c /mnt/lustre2); if [ \$running -ne \$mpts ]; then     echo \$(hostname) env are INSANE!;     exit 1; fi");echo XXRETCODE:$?'
May 16 00:14:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=3700 duration=0(sec)
May 16 00:14:17 client-18 xinetd[3390]: START: shell pid=3741 from=192.168.4.5
May 16 00:14:17 client-18 rshd[3742]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c " running=\$(mount | grep -c /mnt/lustre' '); rc=0; if [ \$running -eq 0 ] ; then     mkdir -p /mnt/lustre;     mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre;     rc=\$?; fi; exit \$rc");echo XXRETCODE:$?'
May 16 00:14:17 client-18 kernel: Lustre: OBD class driver, http://www.lustre.org/
May 16 00:14:17 client-18 kernel: Lustre:         Lustre Version: 2.0.60
May 16 00:14:17 client-18 kernel: Lustre:         Build Version: jenkins-g985ac5b-PRISTINE-2.6.18-194.17.1.el5
May 16 00:14:17 client-18 kernel: Lustre: Lustre LU module (ffffffff888c11c0).
May 16 00:14:17 client-18 kernel: Lustre: Register global MR array, MR size: 0xffffffffffffffff, array size: 1
May 16 00:14:17 client-18 kernel: Lustre: Added LNI 192.168.4.18@o2ib [8/64/0/180]
May 16 00:14:17 client-18 kernel: Lustre: Added LNI 10.10.4.18@tcp [8/256/0/180]
May 16 00:14:17 client-18 kernel: Lustre: Accept all, port 7988
May 16 00:14:18 client-18 kernel: Lustre: Lustre OSC module (ffffffff88b006c0).
May 16 00:14:18 client-18 kernel: Lustre: Lustre LOV module (ffffffff88b6f540).
May 16 00:14:18 client-18 kernel: Lustre: Lustre client module (ffffffff88c0f3a0).
May 16 00:14:18 client-18 kernel: Lustre: 3781:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null
May 16 00:14:18 client-18 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'.
May 16 00:14:18 client-18 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import
May 16 00:14:18 client-18 kernel: Lustre: 3781:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff81031ffac400->192.168.4.128@o2ib netid 50000: select flavor null
May 16 00:14:18 client-18 kernel: Lustre: Client lustre-client has started
May 16 00:14:18 client-18 xinetd[3390]: EXIT: shell status=0 pid=3741 duration=1(sec)
May 16 00:14:18 client-18 xinetd[3390]: START: shell pid=3933 from=192.168.4.5
May 16 00:14:18 client-18 rshd[3934]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "mount | grep -w /mnt/lustre");echo XXRETCODE:$?'
May 16 00:14:18 client-18 xinetd[3390]: EXIT: shell status=0 pid=3933 duration=0(sec)
May 16 00:14:18 client-18 xinetd[3390]: START: shell pid=3970 from=192.168.4.5
May 16 00:14:18 client-18 rshd[3971]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param debug=\"0x33f0406\";         lctl set_param subsystem_debug=\"0xffb7e3ff\";         lctl set_param debug_mb=32;");echo XXRETCODE:$?'
May 16 00:14:18 client-18 kernel: Lustre: 4005:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release.
May 16 00:14:18 client-18 kernel: Lustre: 4006:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release.
May 16 00:14:18 client-18 xinetd[3390]: EXIT: shell status=0 pid=3970 duration=0(sec)
May 16 00:14:19 client-18 xinetd[3390]: START: shell pid=4007 from=192.168.4.5
May 16 00:14:19 client-18 rshd[4008]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark Using TIMEOUT=20");echo XXRETCODE:$?'
May 16 00:14:19 client-18 kernel: Lustre: DEBUG MARKER: Using TIMEOUT=20
May 16 00:14:19 client-18 xinetd[3390]: EXIT: shell status=0 pid=4007 duration=0(sec)
May 16 00:14:24 client-18 xinetd[3390]: START: shell pid=4042 from=192.168.4.5
May 16 00:14:24 client-18 rshd[4043]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/tests:/usr/lib64/openmpi/1.4-gcc/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/usr/lib64/lustre/utils:/usr/lib64/lustre/utils/gss:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey::/usr/lib64/lustre/tests/mpi NAME=sarah sh rpc.sh check_logdir /tmp/test_logs/2011-05-17/001301 ");echo XXRETCODE:$?'
May 16 00:14:24 client-18 xinetd[3390]: EXIT: shell status=0 pid=4042 duration=0(sec)
May 16 00:14:24 client-18 xinetd[3390]: START: shell pid=4127 from=192.168.4.5
May 16 00:14:25 client-18 rshd[4128]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/tests:/usr/lib64/openmpi/1.4-gcc/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/usr/lib64/lustre/utils:/usr/lib64/lustre/utils/gss:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey::/usr/lib64/lustre/tests/mpi NAME=sarah sh rpc.sh yml_node ");echo XXRETCODE:$?'
May 16 00:14:25 client-18 xinetd[3390]: EXIT: shell status=0 pid=4127 duration=1(sec)
May 16 00:14:25 client-18 xinetd[3390]: START: shell pid=4233 from=192.168.4.5
May 16 00:14:25 client-18 rshd[4234]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark -----============= acceptance-small: sanity ============----- Tue May 17 00:14:25 PDT 2011");echo XXRETCODE:$?'
May 16 00:14:25 client-18 kernel: Lustre: DEBUG MARKER: -----============= acceptance-small: sanity ============----- Tue May 17 00:14:25 PDT 2011
May 16 00:14:25 client-18 xinetd[3390]: EXIT: shell status=0 pid=4233 duration=0(sec)
May 16 00:14:25 client-18 xinetd[3390]: START: shell pid=4268 from=192.168.4.5
May 16 00:14:25 client-18 rshd[4269]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/tests:/usr/lib64/openmpi/1.4-gcc/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/usr/lib64/lustre/utils:/usr/lib64/lustre/utils/gss:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey::/usr/lib64/lustre/tests/mpi:/sbin NAME=sarah sh rpc.sh check_config_client /mnt/lustre ");echo XXRETCODE:$?'
May 16 00:14:25 client-18 xinetd[3390]: EXIT: shell status=0 pid=4268 duration=0(sec)
May 16 00:14:26 client-18 xinetd[3390]: START: shell pid=4362 from=192.168.4.5
May 16 00:14:26 client-18 rshd[4363]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); mpts=\$(mount | grep -w -c /mnt/lustre); if [ \$running -ne \$mpts ]; then     echo \$(hostname) env are INSANE!;     exit 1; fi");echo XXRETCODE:$?'
May 16 00:14:26 client-18 xinetd[3390]: EXIT: shell status=0 pid=4362 duration=0(sec)
May 16 00:14:26 client-18 xinetd[3390]: START: shell pid=4402 from=192.168.4.5
May 16 00:14:26 client-18 rshd[4403]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); mpts=\$(mount | grep -w -c /mnt/lustre2); if [ \$running -ne \$mpts ]; then     echo \$(hostname) env are INSANE!;     exit 1; fi");echo XXRETCODE:$?'
May 16 00:14:26 client-18 xinetd[3390]: EXIT: shell status=0 pid=4402 duration=0(sec)
May 16 00:14:28 client-18 xinetd[3390]: START: shell pid=4442 from=192.168.4.5
May 16 00:14:28 client-18 rshd[4443]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark Using TIMEOUT=20");echo XXRETCODE:$?'
May 16 00:14:28 client-18 kernel: Lustre: DEBUG MARKER: Using TIMEOUT=20
May 16 00:14:28 client-18 xinetd[3390]: EXIT: shell status=0 pid=4442 duration=0(sec)
May 16 00:14:29 client-18 xinetd[3390]: START: shell pid=4477 from=192.168.4.5
May 16 00:14:29 client-18 rshd[4478]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param debug=\"0x33f0406\";             lctl set_param subsystem_debug=\"0xffb7e3ff\";             lctl set_param debug_mb=32;             sync");echo XXRETCODE:$?'
May 16 00:14:29 client-18 kernel: Lustre: 4512:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release.
May 16 00:14:29 client-18 xinetd[3390]: EXIT: shell status=0 pid=4477 duration=0(sec)
May 16 00:14:29 client-18 xinetd[3390]: START: shell pid=4515 from=192.168.4.5
May 16 00:14:29 client-18 rshd[4516]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "cat /proc/mounts");echo XXRETCODE:$?'
May 16 00:14:29 client-18 xinetd[3390]: EXIT: shell status=0 pid=4515 duration=0(sec)
May 16 00:14:30 client-18 xinetd[3390]: START: shell pid=4550 from=192.168.4.5
May 16 00:14:30 client-18 rshd[4551]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl get_param -n osc.*OST*-osc-[^M][^D][^T]*.srpc_info");echo XXRETCODE:$?'
May 16 00:14:30 client-18 xinetd[3390]: EXIT: shell status=0 pid=4550 duration=0(sec)
May 16 00:14:30 client-18 xinetd[3390]: START: shell pid=4585 from=192.168.4.5
May 16 00:14:30 client-18 rshd[4586]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl get_param -n mdc.*-MDT*-mdc-*.srpc_info");echo XXRETCODE:$?'
May 16 00:14:30 client-18 xinetd[3390]: EXIT: shell status=0 pid=4585 duration=0(sec)
May 16 00:14:30 client-18 xinetd[3390]: START: shell pid=4620 from=192.168.4.5
May 16 00:14:30 client-18 rshd[4621]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark excepting tests: 27u 42a 42b 42c 42d 45 51d 65a 65e 68b");echo XXRETCODE:$?'
May 16 00:14:30 client-18 kernel: Lustre: DEBUG MARKER: excepting tests: 27u 42a 42b 42c 42d 45 51d 65a 65e 68b
May 16 00:14:30 client-18 xinetd[3390]: EXIT: shell status=0 pid=4620 duration=0(sec)
May 16 00:14:30 client-18 xinetd[3390]: START: shell pid=4655 from=192.168.4.5
May 16 00:14:30 client-18 rshd[4656]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 0: touch ...\/ \; rm ...\/ =============================================================== 00:14:30 \(1305616470\)");echo XXRETCODE:$?'
May 16 00:14:30 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 0: touch .../ ; rm .../ =============================================================== 00:14:30 (1305616470)
May 16 00:14:30 client-18 xinetd[3390]: EXIT: shell status=0 pid=4655 duration=0(sec)
May 16 00:14:30 client-18 xinetd[3390]: START: shell pid=4690 from=192.168.4.5
May 16 00:14:30 client-18 rshd[4691]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 16 00:14:30 client-18 xinetd[3390]: EXIT: shell status=0 pid=4690 duration=0(sec)
May 16 00:14:31 client-18 xinetd[3390]: START: shell pid=4726 from=192.168.4.5
May 16 00:14:31 client-18 rshd[4727]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 16 00:14:31 client-18 xinetd[3390]: EXIT: shell status=0 pid=4726 duration=0(sec)
May 16 00:14:31 client-18 xinetd[3390]: START: shell pid=4763 from=192.168.4.5
May 16 00:14:31 client-18 rshd[4764]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 0b: chmod 0755 \/mnt\/lustre ============================================================ 00:14:31 \(1305616471\)");echo XXRETCODE:$?'
May 16 00:14:31 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 0b: chmod 0755 /mnt/lustre ============================================================ 00:14:31 (1305616471)
May 16 00:14:31 client-18 xinetd[3390]: EXIT: shell status=0 pid=4763 duration=0(sec)
May 16 00:14:31 client-18 xinetd[3390]: START: shell pid=4798 from=192.168.4.5
May 16 00:14:31 client-18 rshd[4799]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 16 00:14:31 client-18 xinetd[3390]: EXIT: shell status=0 pid=4798 duration=0(sec)
May 16 00:14:31 client-18 xinetd[3390]: START: shell pid=4834 from=192.168.4.5
May 16 00:14:31 client-18 rshd[4835]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 16 00:14:31 client-18 xinetd[3390]: EXIT: shell status=0 pid=4834 duration=0(sec)
May 16 00:14:31 client-18 xinetd[3390]: START: shell pid=4871 from=192.168.4.5
May 16 00:14:31 client-18 rshd[4872]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 0c: check import proc ================================================================= 00:14:31 \(1305616471\)");echo XXRETCODE:$?'
May 16 00:14:31 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 0c: check import proc ================================================================= 00:14:31 (1305616471)
May 16 00:14:31 client-18 xinetd[3390]: EXIT: shell status=0 pid=4871 duration=0(sec)
May 16 00:14:31 client-18 xinetd[3390]: START: shell pid=4906 from=192.168.4.5
May 16 00:14:31 client-18 rshd[4907]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 16 00:14:31 client-18 xinetd[3390]: EXIT: shell status=0 pid=4906 duration=0(sec)
May 16 00:14:32 client-18 xinetd[3390]: START: shell pid=4942 from=192.168.4.5
May 16 00:14:32 client-18 rshd[4943]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 16 00:14:32 client-18 xinetd[3390]: EXIT: shell status=0 pid=4942 duration=0(sec)
May 16 00:14:32 client-18 xinetd[3390]: START: shell pid=4979 from=192.168.4.5
May 16 00:14:32 client-18 rshd[4980]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 1a: mkdir ...\/d1\; mkdir ...\/d1\/d2 ===================================================== 00:14:32 \(1305616472\)");echo XXRETCODE:$?'
May 16 00:14:32 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 1a: mkdir .../d1; mkdir .../d1/d2 ===================================================== 00:14:32 (1305616472)
May 16 00:14:32 client-18 xinetd[3390]: EXIT: shell status=0 pid=4979 duration=0(sec)
May 16 00:14:32 client-18 xinetd[3390]: START: shell pid=5014 from=192.168.4.5
May 16 00:14:32 client-18 rshd[5015]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 16 00:14:32 client-18 xinetd[3390]: EXIT: shell status=0 pid=5014 duration=0(sec)
May 16 00:14:32 client-18 xinetd[3390]: START: shell pid=5050 from=192.168.4.5
May 16 00:14:32 client-18 rshd[5051]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 16 00:14:32 client-18 xinetd[3390]: EXIT: shell status=0 pid=5050 duration=0(sec)
May 16 00:14:32 client-18 xinetd[3390]: START: shell pid=5087 from=192.168.4.5
May 16 00:14:32 client-18 rshd[5088]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 1b: rmdir ...\/d1\/d2\; rmdir ...\/d1 ===================================================== 00:14:32 \(1305616472\)");echo XXRETCODE:$?'
May 16 00:14:32 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 1b: rmdir .../d1/d2; rmdir .../d1 ===================================================== 00:14:32 (1305616472)
May 16 00:14:32 client-18 xinetd[3390]: EXIT: shell status=0 pid=5087 duration=0(sec)
May 16 00:14:32 client-18 xinetd[3390]: START: shell pid=5122 from=192.168.4.5
May 16 00:14:32 client-18 rshd[5123]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 16 00:14:32 client-18 xinetd[3390]: EXIT: shell status=0 pid=5122 duration=0(sec)
May 16 00:14:33 client-18 xinetd[3390]: START: shell pid=5158 from=192.168.4.5
May 16 00:14:33 client-18 rshd[5159]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 16 00:14:33 client-18 xinetd[3390]: EXIT: shell status=0 pid=5158 duration=0(sec)
May 16 00:14:33 client-18 xinetd[3390]: START: shell pid=5195 from=192.168.4.5
May 16 00:14:33 client-18 rshd[5196]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 2a: mkdir ...\/d2\; touch ...\/d2\/f ====================================================== 00:14:33 \(1305616473\)");echo XXRETCODE:$?'
May 16 00:14:33 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 2a: mkdir .../d2; touch .../d2/f ====================================================== 00:14:33 (1305616473)
May 16 00:14:33 client-18 xinetd[3390]: EXIT: shell status=0 pid=5195 duration=0(sec)
May 16 00:14:33 client-18 xinetd[3390]: START: shell pid=5230 from=192.168.4.5
May 16 00:14:33 client-18 rshd[5231]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 16 00:14:33 client-18 xinetd[3390]: EXIT: shell status=0 pid=5230 duration=0(sec)
May 16 00:14:33 client-18 xinetd[3390]: START: shell pid=5266 from=192.168.4.5
May 16 00:14:33 client-18 rshd[5267]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 16 00:14:33 client-18 xinetd[3390]: EXIT: shell status=0 pid=5266 duration=0(sec)
May 16 00:14:33 client-18 xinetd[3390]: START: shell pid=5303 from=192.168.4.5
May 16 00:14:33 client-18 rshd[5304]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 2b: rm -r ...\/d2\; checkstat ...\/d2\/f ================================================== 00:14:33 \(1305616473\)");echo XXRETCODE:$?'
May 16 00:14:33 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 2b: rm -r .../d2; checkstat .../d2/f ================================================== 00:14:33 (1305616473)
May 16 00:14:33 client-18 xinetd[3390]: EXIT: shell status=0 pid=5303 duration=0(sec)
May 16 00:14:33 client-18 xinetd[3390]: START: shell pid=5338 from=192.168.4.5
May 16 00:14:33 client-18 rshd[5339]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 16 00:14:33 client-18 xinetd[3390]: EXIT: shell status=0 pid=5338 duration=0(sec)
May 16 00:14:34 client-18 xinetd[3390]: START: shell pid=5374 from=192.168.4.5
May 16 00:14:34 client-18 rshd[5375]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 16 00:14:34 client-18 xinetd[3390]: EXIT: shell status=0 pid=5374 duration=0(sec)
May 16 00:14:34 client-18 xinetd[3390]: START: shell pid=5411 from=192.168.4.5
May 16 00:14:34 client-18 rshd[5412]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 3a: mkdir ...\/d3 ====================================================================== 00:14:34 \(1305616474\)");echo XXRETCODE:$?'
May 16 00:14:34 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 3a: mkdir .../d3 ====================================================================== 00:14:34 (1305616474)
May 16 00:14:34 client-18 xinetd[3390]: EXIT: shell status=0 pid=5411 duration=0(sec)
May 16 00:14:34 client-18 xinetd[3390]: START: shell pid=5446 from=192.168.4.5
May 16 00:14:34 client-18 rshd[5447]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 16 00:14:34 client-18 xinetd[3390]: EXIT: shell status=0 pid=5446 duration=0(sec)
May 16 00:14:34 client-18 xinetd[3390]: START: shell pid=5482 from=192.168.4.5
May 16 00:14:34 client-18 rshd[5483]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 16 00:14:34 client-18 xinetd[3390]: EXIT: shell status=0 pid=5482 duration=0(sec)
May 16 00:14:34 client-18 xinetd[3390]: START: shell pid=5519 from=192.168.4.5
May 16 00:14:34 client-18 rshd[5520]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 3b: touch ...\/d3\/f ==================================================================== 00:14:34 \(1305616474\)");echo XXRETCODE:$?'
May 16 00:14:34 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 3b: touch .../d3/f ==================================================================== 00:14:34 (1305616474)
May 16 00:14:34 client-18 xinetd[3390]: EXIT: shell status=0 pid=5519 duration=0(sec)
May 16 00:14:34 client-18 xinetd[3390]: START: shell pid=5554 from=192.168.4.5
May 16 00:14:34 client-18 rshd[5555]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 16 00:14:34 client-18 xinetd[3390]: EXIT: shell status=0 pid=5554 duration=0(sec)
May 16 00:14:35 client-18 xinetd[3390]: START: shell pid=5590 from=192.168.4.5
May 16 00:14:35 client-18 rshd[5591]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 16 00:14:35 client-18 xinetd[3390]: EXIT: shell status=0 pid=5590 duration=0(sec)
May 16 00:14:35 client-18 xinetd[3390]: START: shell pid=5627 from=192.168.4.5
May 16 00:14:35 client-18 rshd[5628]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 3c: rm -r ...\/d3 ====================================================================== 00:14:35 \(1305616475\)");echo XXRETCODE:$?'
May 16 00:14:35 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 3c: rm -r .../d3 ====================================================================== 00:14:35 (1305616475)
May 16 00:14:35 client-18 xinetd[3390]: EXIT: shell status=0 pid=5627 duration=0(sec)
May 16 00:14:35 client-18 xinetd[3390]: START: shell pid=5662 from=192.168.4.5
May 16 00:14:35 client-18 rshd[5663]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 16 00:14:35 client-18 xinetd[3390]: EXIT: shell status=0 pid=5662 duration=0(sec)
May 16 00:14:35 client-18 xinetd[3390]: START: shell pid=5698 from=192.168.4.5
May 16 00:14:35 client-18 rshd[5699]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 16 00:14:35 client-18 xinetd[3390]: EXIT: shell status=0 pid=5698 duration=0(sec)
May 16 00:14:35 client-18 xinetd[3390]: START: shell pid=5735 from=192.168.4.5
May 16 00:14:35 client-18 rshd[5736]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 4a: mkdir ...\/d4 ====================================================================== 00:14:35 \(1305616475\)");echo XXRETCODE:$?'
May 16 00:14:35 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 4a: mkdir .../d4 ====================================================================== 00:14:35 (1305616475)
May 16 00:14:35 client-18 xinetd[3390]: EXIT: shell status=0 pid=5735 duration=0(sec)
May 16 00:14:35 client-18 xinetd[3390]: START: shell pid=5770 from=192.168.4.5
May 16 00:14:35 client-18 rshd[5771]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 16 00:14:35 client-18 xinetd[3390]: EXIT: shell status=0 pid=5770 duration=0(sec)
May 16 00:14:36 client-18 xinetd[3390]: START: shell pid=5806 from=192.168.4.5
May 16 00:14:36 client-18 rshd[5807]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 16 00:14:36 client-18 xinetd[3390]: EXIT: shell status=0 pid=5806 duration=0(sec)
May 16 00:14:36 client-18 xinetd[3390]: START: shell pid=5843 from=192.168.4.5
May 16 00:14:36 client-18 rshd[5844]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 4b: mkdir ...\/d4\/d2 =================================================================== 00:14:36 \(1305616476\)");echo XXRETCODE:$?'
May 16 00:14:36 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 4b: mkdir .../d4/d2 =================================================================== 00:14:36 (1305616476)
May 16 00:14:36 client-18 xinetd[3390]: EXIT: shell status=0 pid=5843 duration=0(sec)
May 16 00:14:36 client-18 xinetd[3390]: START: shell pid=5878 from=192.168.4.5
May 16 00:14:36 client-18 rshd[5879]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 16 00:14:36 client-18 xinetd[3390]: EXIT: shell status=0 pid=5878 duration=0(sec)
May 16 00:14:36 client-18 xinetd[3390]: START: shell pid=5914 from=192.168.4.5
May 16 00:14:36 client-18 rshd[5915]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 16 00:14:36 client-18 xinetd[3390]: EXIT: shell status=0 pid=5914 duration=0(sec)
May 16 00:14:36 client-18 xinetd[3390]: START: shell pid=5951 from=192.168.4.5
May 16 00:14:36 client-18 rshd[5952]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 5: mkdir ...\/d5 ...\/d5\/d2\; chmod ...\/d5\/d2 ============================================ 00:14:36 \(1305616476\)");echo XXRETCODE:$?'
May 16 00:14:36 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 5: mkdir .../d5 .../d5/d2; chmod .../d5/d2 ============================================ 00:14:36 (1305616476)
May 16 00:14:36 client-18 xinetd[3390]: EXIT: shell status=0 pid=5951 duration=0(sec)
May 16 00:14:36 client-18 xinetd[3390]: START: shell pid=5986 from=192.168.4.5
May 16 00:14:36 client-18 rshd[5987]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 16 00:14:36 client-18 xinetd[3390]: EXIT: shell status=0 pid=5986 duration=0(sec)
May 16 00:14:37 client-18 xinetd[3390]: START: shell pid=6022 from=192.168.4.5
May 16 00:14:37 client-18 rshd[6023]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 16 00:14:37 client-18 xinetd[3390]: EXIT: shell status=0 pid=6022 duration=0(sec)
May 16 00:14:37 client-18 xinetd[3390]: START: shell pid=6059 from=192.168.4.5
May 16 00:14:37 client-18 rshd[6060]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 6a: touch ...\/f6a\; chmod ...\/f6a ====================================================== 00:14:37 \(1305616477\)");echo XXRETCODE:$?'
May 16 00:14:37 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 6a: touch .../f6a; chmod .../f6a ====================================================== 00:14:37 (1305616477)
May 16 00:14:37 client-18 xinetd[3390]: EXIT: shell status=0 pid=6059 duration=0(sec)
May 16 00:14:37 client-18 xinetd[3390]: START: shell pid=6094 from=192.168.4.5
May 16 00:14:37 client-18 rshd[6095]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 16 00:14:37 client-18 xinetd[3390]: EXIT: shell status=0 pid=6094 duration=0(sec)
May 16 00:14:37 client-18 xinetd[3390]: START: shell pid=6130 from=192.168.4.5
May 16 00:14:37 client-18 rshd[6131]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 16 00:14:37 client-18 xinetd[3390]: EXIT: shell status=0 pid=6130 duration=0(sec)
May 16 00:14:37 client-18 xinetd[3390]: START: shell pid=6167 from=192.168.4.5
May 16 00:14:37 client-18 rshd[6168]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 6b: runas -u 500 chmod ...\/f6a \(should return error\) ================================== 00:14:37 \(1305616477\)");echo XXRETCODE:$?'
May 16 00:14:37 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 6b: runas -u 500 chmod .../f6a (should return error) ================================== 00:14:37 (1305616477)
May 16 00:14:37 client-18 xinetd[3390]: EXIT: shell status=0 pid=6167 duration=0(sec)
May 16 00:14:37 client-18 xinetd[3390]: START: shell pid=6202 from=192.168.4.5
May 16 00:14:37 client-18 rshd[6203]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 16 00:14:37 client-18 xinetd[3390]: EXIT: shell status=0 pid=6202 duration=0(sec)
May 16 00:14:38 client-18 xinetd[3390]: START: shell pid=6238 from=192.168.4.5
May 16 00:14:38 client-18 rshd[6239]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 16 00:14:38 client-18 xinetd[3390]: EXIT: shell status=0 pid=6238 duration=0(sec)
May 16 00:14:38 client-18 xinetd[3390]: START: shell pid=6275 from=192.168.4.5
May 16 00:14:38 client-18 rshd[6276]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 6c: touch ...\/f6c\; chown ...\/f6c ====================================================== 00:14:38 \(1305616478\)");echo XXRETCODE:$?'
May 16 00:14:38 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 6c: touch .../f6c; chown .../f6c ====================================================== 00:14:38 (1305616478)
May 16 00:14:38 client-18 xinetd[3390]: EXIT: shell status=0 pid=6275 duration=0(sec)
May 16 00:14:38 client-18 xinetd[3390]: START: shell pid=6310 from=192.168.4.5
May 16 00:14:38 client-18 rshd[6311]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 16 00:14:38 client-18 xinetd[3390]: EXIT: shell status=0 pid=6310 duration=0(sec)
May 16 00:14:38 client-18 xinetd[3390]: START: shell pid=6346 from=192.168.4.5
May 16 00:14:38 client-18 rshd[6347]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 16 00:14:38 client-18 xinetd[3390]: EXIT: shell status=0 pid=6346 duration=0(sec)
May 16 00:14:38 client-18 xinetd[3390]: START: shell pid=6383 from=192.168.4.5
May 16 00:14:38 client-18 rshd[6384]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 6d: runas -u 500 chown ...\/f6c \(should return error\) ================================== 00:14:38 \(1305616478\)");echo XXRETCODE:$?'
May 16 00:14:38 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 6d: runas -u 500 chown .../f6c (should return error) ================================== 00:14:38 (1305616478)
May 16 00:14:38 client-18 xinetd[3390]: EXIT: shell status=0 pid=6383 duration=0(sec)
May 16 00:14:38 client-18 xinetd[3390]: START: shell pid=6418 from=192.168.4.5
May 16 00:14:38 client-18 rshd[6419]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 16 00:14:38 client-18 xinetd[3390]: EXIT: shell status=0 pid=6418 duration=0(sec)
May 16 00:14:39 client-18 xinetd[3390]: START: shell pid=6454 from=192.168.4.5
May 16 00:14:39 client-18 rshd[6455]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 16 00:14:39 client-18 xinetd[3390]: EXIT: shell status=0 pid=6454 duration=0(sec)
May 16 00:14:39 client-18 xinetd[3390]: START: shell pid=6491 from=192.168.4.5
May 16 00:14:39 client-18 rshd[6492]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 6e: touch ...\/f6e\; chgrp ...\/f6e ====================================================== 00:14:39 \(1305616479\)");echo XXRETCODE:$?'
May 16 00:14:39 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 6e: touch .../f6e; chgrp .../f6e ====================================================== 00:14:39 (1305616479)
May 16 00:14:39 client-18 xinetd[3390]: EXIT: shell status=0 pid=6491 duration=0(sec)
May 16 00:14:39 client-18 xinetd[3390]: START: shell pid=6526 from=192.168.4.5
May 16 00:14:39 client-18 rshd[6527]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 16 00:14:39 client-18 xinetd[3390]: EXIT: shell status=0 pid=6526 duration=0(sec)
May 16 00:14:39 client-18 xinetd[3390]: START: shell pid=6562 from=192.168.4.5
May 16 00:14:39 client-18 rshd[6563]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 16 00:14:39 client-18 xinetd[3390]: EXIT: shell status=0 pid=6562 duration=0(sec)
May 16 00:14:39 client-18 xinetd[3390]: START: shell pid=6599 from=192.168.4.5
May 16 00:14:39 client-18 rshd[6600]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 6f: runas -u 500 chgrp ...\/f6e \(should return error\) ================================== 00:14:39 \(1305616479\)");echo XXRETCODE:$?'
May 16 00:14:39 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 6f: runas -u 500 chgrp .../f6e (should return error) ================================== 00:14:39 (1305616479)
May 16 00:14:39 client-18 xinetd[3390]: EXIT: shell status=0 pid=6599 duration=0(sec)
May 16 00:14:39 client-18 xinetd[3390]: START: shell pid=6634 from=192.168.4.5
May 16 00:14:39 client-18 rshd[6635]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 16 00:14:39 client-18 xinetd[3390]: EXIT: shell status=0 pid=6634 duration=0(sec)
May 16 00:14:40 client-18 xinetd[3390]: START: shell pid=6670 from=192.168.4.5
May 16 00:14:40 client-18 rshd[6671]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 16 00:14:40 client-18 xinetd[3390]: EXIT: shell status=0 pid=6670 duration=0(sec)
May 16 00:14:40 client-18 xinetd[3390]: START: shell pid=6707 from=192.168.4.5
May 16 00:14:40 client-18 rshd[6708]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 6g: Is new dir in sgid dir inheriting group? ========================================== 00:14:40 \(1305616480\)");echo XXRETCODE:$?'
May 16 00:14:40 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 6g: Is new dir in sgid dir inheriting group? ========================================== 00:14:40 (1305616480)
May 16 00:14:40 client-18 xinetd[3390]: EXIT: shell status=0 pid=6707 duration=0(sec)
May 16 00:14:40 client-18 xinetd[3390]: START: shell pid=6742 from=192.168.4.5
May 16 00:14:40 client-18 rshd[6743]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 16 00:14:40 client-18 xinetd[3390]: EXIT: shell status=0 pid=6742 duration=0(sec)
May 16 00:14:40 client-18 xinetd[3390]: START: shell pid=6778 from=192.168.4.5
May 16 00:14:40 client-18 rshd[6779]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 16 00:14:40 client-18 xinetd[3390]: EXIT: shell status=0 pid=6778 duration=0(sec)
May 16 00:14:40 client-18 xinetd[3390]: START: shell pid=6815 from=192.168.4.5
May 16 00:14:40 client-18 rshd[6816]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 6h: runas -u 500 chown RUNAS_ID.0 ...\/f6h \(should return error\) ======================= 00:14:40 \(1305616480\)");echo XXRETCODE:$?'
May 16 00:14:40 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 6h: runas -u 500 chown RUNAS_ID.0 .../f6h (should return error) ======================= 00:14:40 (1305616480)
May 16 00:14:40 client-18 xinetd[3390]: EXIT: shell status=0 pid=6815 duration=0(sec)
May 16 00:14:40 client-18 xinetd[3390]: START: shell pid=6850 from=192.168.4.5
May 16 00:14:40 client-18 rshd[6851]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 16 00:14:41 client-18 xinetd[3390]: EXIT: shell status=0 pid=6850 duration=1(sec)
May 16 00:14:41 client-18 xinetd[3390]: START: shell pid=6886 from=192.168.4.5
May 16 00:14:41 client-18 rshd[6887]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 16 00:14:41 client-18 xinetd[3390]: EXIT: shell status=0 pid=6886 duration=0(sec)
May 16 00:14:41 client-18 xinetd[3390]: START: shell pid=6923 from=192.168.4.5
May 16 00:14:41 client-18 rshd[6924]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 7a: mkdir ...\/d7\; mcreate ...\/d7\/f\; chmod ...\/d7\/f ==================================== 00:14:41 \(1305616481\)");echo XXRETCODE:$?'
May 16 00:14:41 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 7a: mkdir .../d7; mcreate .../d7/f; chmod .../d7/f ==================================== 00:14:41 (1305616481)
May 16 00:14:41 client-18 xinetd[3390]: EXIT: shell status=0 pid=6923 duration=0(sec)
May 16 00:14:41 client-18 xinetd[3390]: START: shell pid=6958 from=192.168.4.5
May 16 00:14:41 client-18 rshd[6959]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 16 00:14:41 client-18 xinetd[3390]: EXIT: shell status=0 pid=6958 duration=0(sec)
May 16 00:14:41 client-18 xinetd[3390]: START: shell pid=6994 from=192.168.4.5
May 16 00:14:41 client-18 rshd[6995]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 16 00:14:41 client-18 xinetd[3390]: EXIT: shell status=0 pid=6994 duration=0(sec)
May 16 00:14:41 client-18 xinetd[3390]: START: shell pid=7031 from=192.168.4.5
May 16 00:14:41 client-18 rshd[7032]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 7b: mkdir ...\/d7\; mcreate d7\/f2\; echo foo \> d7\/f2 ===================================== 00:14:41 \(1305616481\)");echo XXRETCODE:$?'
May 16 00:14:41 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 7b: mkdir .../d7; mcreate d7/f2; echo foo > d7/f2 ===================================== 00:14:41 (1305616481)
May 16 00:14:41 client-18 xinetd[3390]: EXIT: shell status=0 pid=7031 duration=0(sec)
May 16 00:14:41 client-18 xinetd[3390]: START: shell pid=7066 from=192.168.4.5
May 16 00:14:41 client-18 rshd[7067]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 16 00:14:42 client-18 xinetd[3390]: EXIT: shell status=0 pid=7066 duration=1(sec)
May 16 00:14:42 client-18 xinetd[3390]: START: shell pid=7102 from=192.168.4.5
May 16 00:14:42 client-18 rshd[7103]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 16 00:14:42 client-18 xinetd[3390]: EXIT: shell status=0 pid=7102 duration=0(sec)
May 16 00:14:42 client-18 xinetd[3390]: START: shell pid=7139 from=192.168.4.5
May 16 00:14:42 client-18 rshd[7140]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 8: mkdir ...\/d8\; touch ...\/d8\/f\; chmod ...\/d8\/f ======================================= 00:14:42 \(1305616482\)");echo XXRETCODE:$?'
May 16 00:14:42 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 8: mkdir .../d8; touch .../d8/f; chmod .../d8/f ======================================= 00:14:42 (1305616482)
May 16 00:14:42 client-18 xinetd[3390]: EXIT: shell status=0 pid=7139 duration=0(sec)
May 16 00:14:42 client-18 xinetd[3390]: START: shell pid=7174 from=192.168.4.5
May 16 00:14:42 client-18 rshd[7175]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 16 00:14:42 client-18 xinetd[3390]: EXIT: shell status=0 pid=7174 duration=0(sec)
May 16 00:14:42 client-18 xinetd[3390]: START: shell pid=7210 from=192.168.4.5
May 16 00:14:42 client-18 rshd[7211]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 16 00:14:42 client-18 xinetd[3390]: EXIT: shell status=0 pid=7210 duration=0(sec)
May 16 00:14:42 client-18 xinetd[3390]: START: shell pid=7247 from=192.168.4.5
May 16 00:14:42 client-18 rshd[7248]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 9: mkdir ...\/d9 ...\/d9\/d2 ...\/d9\/d2\/d3 ================================================ 00:14:42 \(1305616482\)");echo XXRETCODE:$?'
May 16 00:14:42 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 9: mkdir .../d9 .../d9/d2 .../d9/d2/d3 ================================================ 00:14:42 (1305616482)
May 16 00:14:42 client-18 xinetd[3390]: EXIT: shell status=0 pid=7247 duration=0(sec)
May 16 00:14:42 client-18 xinetd[3390]: START: shell pid=7282 from=192.168.4.5
May 16 00:14:42 client-18 rshd[7283]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 16 00:14:43 client-18 xinetd[3390]: EXIT: shell status=0 pid=7282 duration=1(sec)
May 16 00:14:43 client-18 xinetd[3390]: START: shell pid=7318 from=192.168.4.5
May 16 00:14:43 client-18 rshd[7319]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 16 00:14:43 client-18 xinetd[3390]: EXIT: shell status=0 pid=7318 duration=0(sec)
May 16 00:14:43 client-18 xinetd[3390]: START: shell pid=7355 from=192.168.4.5
May 16 00:14:43 client-18 rshd[7356]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 10: mkdir ...\/d10 ...\/d10\/d2\; touch ...\/d10\/d2\/f ====================================== 00:14:43 \(1305616483\)");echo XXRETCODE:$?'
May 16 00:14:43 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 10: mkdir .../d10 .../d10/d2; touch .../d10/d2/f ====================================== 00:14:43 (1305616483)
May 16 00:14:43 client-18 xinetd[3390]: EXIT: shell status=0 pid=7355 duration=0(sec)
May 16 00:14:43 client-18 xinetd[3390]: START: shell pid=7390 from=192.168.4.5
May 16 00:14:43 client-18 rshd[7391]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 16 00:14:43 client-18 xinetd[3390]: EXIT: shell status=0 pid=7390 duration=0(sec)
May 16 00:14:43 client-18 xinetd[3390]: START: shell pid=7426 from=192.168.4.5
May 16 00:14:43 client-18 rshd[7427]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 16 00:14:43 client-18 xinetd[3390]: EXIT: shell status=0 pid=7426 duration=0(sec)
May 16 00:14:43 client-18 xinetd[3390]: START: shell pid=7463 from=192.168.4.5
May 16 00:14:43 client-18 rshd[7464]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 11: mkdir ...\/d11 d11\/d2\; chmod ...\/d11\/d2 ============================================ 00:14:43 \(1305616483\)");echo XXRETCODE:$?'
May 16 00:14:43 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 11: mkdir .../d11 d11/d2; chmod .../d11/d2 ============================================ 00:14:43 (1305616483)
May 16 00:14:43 client-18 xinetd[3390]: EXIT: shell status=0 pid=7463 duration=0(sec)
May 16 00:14:43 client-18 xinetd[3390]: START: shell pid=7498 from=192.168.4.5
May 16 00:14:43 client-18 rshd[7499]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 16 00:14:44 client-18 xinetd[3390]: EXIT: shell status=0 pid=7498 duration=1(sec)
May 16 00:14:44 client-18 xinetd[3390]: START: shell pid=7534 from=192.168.4.5
May 16 00:14:44 client-18 rshd[7535]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 16 00:14:44 client-18 xinetd[3390]: EXIT: shell status=0 pid=7534 duration=0(sec)
May 16 00:14:44 client-18 xinetd[3390]: START: shell pid=7571 from=192.168.4.5
May 16 00:14:44 client-18 rshd[7572]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 12: touch ...\/d12\/f\; chmod ...\/d12\/f ...\/d12\/f ======================================== 00:14:44 \(1305616484\)");echo XXRETCODE:$?'
May 16 00:14:44 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 12: touch .../d12/f; chmod .../d12/f .../d12/f ======================================== 00:14:44 (1305616484)
May 16 00:14:44 client-18 xinetd[3390]: EXIT: shell status=0 pid=7571 duration=0(sec)
May 16 00:14:44 client-18 xinetd[3390]: START: shell pid=7606 from=192.168.4.5
May 16 00:14:44 client-18 rshd[7607]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 16 00:14:44 client-18 xinetd[3390]: EXIT: shell status=0 pid=7606 duration=0(sec)
May 16 00:14:44 client-18 xinetd[3390]: START: shell pid=7642 from=192.168.4.5
May 16 00:14:44 client-18 rshd[7643]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 16 00:14:44 client-18 xinetd[3390]: EXIT: shell status=0 pid=7642 duration=0(sec)
May 16 00:14:44 client-18 xinetd[3390]: START: shell pid=7679 from=192.168.4.5
May 16 00:14:44 client-18 rshd[7680]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 13: creat ...\/d13\/f\; dd ...\/d13\/f\; \> ...\/d13\/f ======================================== 00:14:44 \(1305616484\)");echo XXRETCODE:$?'
May 16 00:14:44 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 13: creat .../d13/f; dd .../d13/f; > .../d13/f ======================================== 00:14:44 (1305616484)
May 16 00:14:44 client-18 xinetd[3390]: EXIT: shell status=0 pid=7679 duration=0(sec)
May 16 00:14:44 client-18 xinetd[3390]: START: shell pid=7714 from=192.168.4.5
May 16 00:14:44 client-18 rshd[7715]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 16 00:14:45 client-18 xinetd[3390]: EXIT: shell status=0 pid=7714 duration=1(sec)
May 16 00:14:45 client-18 xinetd[3390]: START: shell pid=7750 from=192.168.4.5
May 16 00:14:45 client-18 rshd[7751]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 16 00:14:45 client-18 xinetd[3390]: EXIT: shell status=0 pid=7750 duration=0(sec)
May 16 00:14:45 client-18 xinetd[3390]: START: shell pid=7787 from=192.168.4.5
May 16 00:14:45 client-18 rshd[7788]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 14: touch ...\/d14\/f\; rm ...\/d14\/f\; rm ...\/d14\/f ======================================= 00:14:45 \(1305616485\)");echo XXRETCODE:$?'
May 16 00:14:45 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 14: touch .../d14/f; rm .../d14/f; rm .../d14/f ======================================= 00:14:45 (1305616485)
May 16 00:14:45 client-18 xinetd[3390]: EXIT: shell status=0 pid=7787 duration=0(sec)
May 16 00:14:45 client-18 xinetd[3390]: START: shell pid=7822 from=192.168.4.5
May 16 00:14:45 client-18 rshd[7823]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 16 00:14:45 client-18 xinetd[3390]: EXIT: shell status=0 pid=7822 duration=0(sec)
May 16 00:14:45 client-18 xinetd[3390]: START: shell pid=7858 from=192.168.4.5
May 16 00:14:45 client-18 rshd[7859]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 16 00:14:45 client-18 xinetd[3390]: EXIT: shell status=0 pid=7858 duration=0(sec)
May 16 00:14:45 client-18 xinetd[3390]: START: shell pid=7895 from=192.168.4.5
May 16 00:14:45 client-18 rshd[7896]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 15: touch ...\/d15\/f\; mv ...\/d15\/f ...\/d15\/f2 ========================================== 00:14:45 \(1305616485\)");echo XXRETCODE:$?'
May 16 00:14:45 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 15: touch .../d15/f; mv .../d15/f .../d15/f2 ========================================== 00:14:45 (1305616485)
May 16 00:14:45 client-18 xinetd[3390]: EXIT: shell status=0 pid=7895 duration=0(sec)
May 16 00:14:45 client-18 xinetd[3390]: START: shell pid=7930 from=192.168.4.5
May 16 00:14:46 client-18 rshd[7931]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 16 00:14:46 client-18 xinetd[3390]: EXIT: shell status=0 pid=7930 duration=1(sec)
May 16 00:14:46 client-18 xinetd[3390]: START: shell pid=7966 from=192.168.4.5
May 16 00:14:46 client-18 rshd[7967]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 16 00:14:46 client-18 xinetd[3390]: EXIT: shell status=0 pid=7966 duration=0(sec)
May 16 00:14:46 client-18 xinetd[3390]: START: shell pid=8003 from=192.168.4.5
May 16 00:14:46 client-18 rshd[8004]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 16: touch ...\/d16\/f\; rm -rf ...\/d16\/f ================================================= 00:14:46 \(1305616486\)");echo XXRETCODE:$?'
May 16 00:14:46 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 16: touch .../d16/f; rm -rf .../d16/f ================================================= 00:14:46 (1305616486)
May 16 00:14:46 client-18 xinetd[3390]: EXIT: shell status=0 pid=8003 duration=0(sec)
May 16 00:14:46 client-18 xinetd[3390]: START: shell pid=8038 from=192.168.4.5
May 16 00:14:46 client-18 rshd[8039]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 16 00:14:46 client-18 xinetd[3390]: EXIT: shell status=0 pid=8038 duration=0(sec)
May 16 00:14:46 client-18 xinetd[3390]: START: shell pid=8074 from=192.168.4.5
May 16 00:14:46 client-18 rshd[8075]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 16 00:14:46 client-18 xinetd[3390]: EXIT: shell status=0 pid=8074 duration=0(sec)
May 16 00:14:46 client-18 xinetd[3390]: START: shell pid=8111 from=192.168.4.5
May 16 00:14:46 client-18 rshd[8112]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 17a: symlinks: create, remove \(real\) ================================================== 00:14:46 \(1305616486\)");echo XXRETCODE:$?'
May 16 00:14:46 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 17a: symlinks: create, remove (real) ================================================== 00:14:46 (1305616486)
May 16 00:14:46 client-18 xinetd[3390]: EXIT: shell status=0 pid=8111 duration=0(sec)
May 16 00:14:47 client-18 xinetd[3390]: START: shell pid=8146 from=192.168.4.5
May 16 00:14:47 client-18 rshd[8147]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 16 00:14:47 client-18 xinetd[3390]: EXIT: shell status=0 pid=8146 duration=0(sec)
May 16 00:14:47 client-18 xinetd[3390]: START: shell pid=8182 from=192.168.4.5
May 16 00:14:47 client-18 rshd[8183]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 16 00:14:47 client-18 xinetd[3390]: EXIT: shell status=0 pid=8182 duration=0(sec)
May 16 00:14:47 client-18 xinetd[3390]: START: shell pid=8219 from=192.168.4.5
May 16 00:14:47 client-18 rshd[8220]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 17b: symlinks: create, remove \(dangling\) ============================================== 00:14:47 \(1305616487\)");echo XXRETCODE:$?'
May 16 00:14:47 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 17b: symlinks: create, remove (dangling) ============================================== 00:14:47 (1305616487)
May 16 00:14:47 client-18 xinetd[3390]: EXIT: shell status=0 pid=8219 duration=0(sec)
May 16 00:14:47 client-18 xinetd[3390]: START: shell pid=8254 from=192.168.4.5
May 16 00:14:47 client-18 rshd[8255]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 16 00:14:47 client-18 xinetd[3390]: EXIT: shell status=0 pid=8254 duration=0(sec)
May 16 00:14:47 client-18 xinetd[3390]: START: shell pid=8290 from=192.168.4.5
May 16 00:14:47 client-18 rshd[8291]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 16 00:14:47 client-18 xinetd[3390]: EXIT: shell status=0 pid=8290 duration=0(sec)
May 16 00:14:47 client-18 xinetd[3390]: START: shell pid=8328 from=192.168.4.5
May 16 00:14:47 client-18 rshd[8329]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 17c: symlinks: open dangling \(should return error\) ==================================== 00:14:47 \(1305616487\)");echo XXRETCODE:$?'
May 16 00:14:47 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 17c: symlinks: open dangling (should return error) ==================================== 00:14:47 (1305616487)
May 16 00:14:47 client-18 xinetd[3390]: EXIT: shell status=0 pid=8328 duration=0(sec)
May 16 00:14:48 client-18 xinetd[3390]: START: shell pid=8363 from=192.168.4.5
May 16 00:14:48 client-18 rshd[8364]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 16 00:14:48 client-18 xinetd[3390]: EXIT: shell status=0 pid=8363 duration=0(sec)
May 16 00:14:48 client-18 xinetd[3390]: START: shell pid=8399 from=192.168.4.5
May 16 00:14:48 client-18 rshd[8400]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 16 00:14:48 client-18 xinetd[3390]: EXIT: shell status=0 pid=8399 duration=0(sec)
May 16 00:14:48 client-18 xinetd[3390]: START: shell pid=8436 from=192.168.4.5
May 16 00:14:48 client-18 rshd[8437]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 17d: symlinks: create dangling ======================================================== 00:14:48 \(1305616488\)");echo XXRETCODE:$?'
May 16 00:14:48 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 17d: symlinks: create dangling ======================================================== 00:14:48 (1305616488)
May 16 00:14:48 client-18 xinetd[3390]: EXIT: shell status=0 pid=8436 duration=0(sec)
May 16 00:14:48 client-18 xinetd[3390]: START: shell pid=8471 from=192.168.4.5
May 16 00:14:48 client-18 rshd[8472]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 16 00:14:48 client-18 xinetd[3390]: EXIT: shell status=0 pid=8471 duration=0(sec)
May 16 00:14:48 client-18 xinetd[3390]: START: shell pid=8507 from=192.168.4.5
May 16 00:14:48 client-18 rshd[8508]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 16 00:14:48 client-18 xinetd[3390]: EXIT: shell status=0 pid=8507 duration=0(sec)
May 16 00:14:48 client-18 xinetd[3390]: START: shell pid=8544 from=192.168.4.5
May 16 00:14:48 client-18 rshd[8545]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 17e: symlinks: create recursive symlink \(should return error\) ========================= 00:14:48 \(1305616488\)");echo XXRETCODE:$?'
May 16 00:14:48 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 17e: symlinks: create recursive symlink (should return error) ========================= 00:14:48 (1305616488)
May 16 00:14:48 client-18 xinetd[3390]: EXIT: shell status=0 pid=8544 duration=0(sec)
May 16 00:14:49 client-18 xinetd[3390]: START: shell pid=8579 from=192.168.4.5
May 16 00:14:49 client-18 rshd[8580]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 16 00:14:49 client-18 xinetd[3390]: EXIT: shell status=0 pid=8579 duration=0(sec)
May 16 00:14:49 client-18 xinetd[3390]: START: shell pid=8615 from=192.168.4.5
May 16 00:14:49 client-18 rshd[8616]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 16 00:14:49 client-18 xinetd[3390]: EXIT: shell status=0 pid=8615 duration=0(sec)
May 16 00:14:49 client-18 xinetd[3390]: START: shell pid=8652 from=192.168.4.5
May 16 00:14:49 client-18 rshd[8653]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 17f: symlinks: long and very long symlink name ======================================== 00:14:49 \(1305616489\)");echo XXRETCODE:$?'
May 16 00:14:49 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 17f: symlinks: long and very long symlink name ======================================== 00:14:49 (1305616489)
May 16 00:14:49 client-18 xinetd[3390]: EXIT: shell status=0 pid=8652 duration=0(sec)
May 16 00:14:49 client-18 xinetd[3390]: START: shell pid=8687 from=192.168.4.5
May 16 00:14:49 client-18 rshd[8688]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 16 00:14:49 client-18 xinetd[3390]: EXIT: shell status=0 pid=8687 duration=0(sec)
May 16 00:14:49 client-18 xinetd[3390]: START: shell pid=8723 from=192.168.4.5
May 16 00:14:49 client-18 rshd[8724]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 16 00:14:49 client-18 xinetd[3390]: EXIT: shell status=0 pid=8723 duration=0(sec)
May 16 00:14:49 client-18 xinetd[3390]: START: shell pid=8760 from=192.168.4.5
May 16 00:14:49 client-18 rshd[8761]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 17g: symlinks: really long symlink name =============================================== 00:14:49 \(1305616489\)");echo XXRETCODE:$?'
May 16 00:14:50 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 17g: symlinks: really long symlink name =============================================== 00:14:49 (1305616489)
May 16 00:14:50 client-18 xinetd[3390]: EXIT: shell status=0 pid=8760 duration=1(sec)
May 16 00:14:50 client-18 xinetd[3390]: START: shell pid=8795 from=192.168.4.5
May 16 00:14:50 client-18 rshd[8796]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 16 00:14:50 client-18 xinetd[3390]: EXIT: shell status=0 pid=8795 duration=0(sec)
May 16 00:14:50 client-18 xinetd[3390]: START: shell pid=8831 from=192.168.4.5
May 16 00:14:50 client-18 rshd[8832]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 16 00:14:50 client-18 xinetd[3390]: EXIT: shell status=0 pid=8831 duration=0(sec)
May 16 00:14:50 client-18 xinetd[3390]: START: shell pid=8868 from=192.168.4.5
May 16 00:14:50 client-18 rshd[8869]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 17h: create objects: lov_free_memmd\(\) doesn\'t lbug ==================================== 00:14:50 \(1305616490\)");echo XXRETCODE:$?'
May 16 00:14:50 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 17h: create objects: lov_free_memmd() doesn't lbug ==================================== 00:14:50 (1305616490)
May 16 00:14:50 client-18 xinetd[3390]: EXIT: shell status=0 pid=8868 duration=0(sec)
May 16 00:14:50 client-18 xinetd[3390]: START: shell pid=8904 from=192.168.4.5
May 16 00:14:50 client-18 rshd[8905]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 16 00:14:50 client-18 xinetd[3390]: EXIT: shell status=0 pid=8904 duration=0(sec)
May 16 00:14:50 client-18 xinetd[3390]: START: shell pid=8940 from=192.168.4.5
May 16 00:14:50 client-18 rshd[8941]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 16 00:14:51 client-18 xinetd[3390]: EXIT: shell status=0 pid=8940 duration=1(sec)
May 16 00:14:51 client-18 xinetd[3390]: START: shell pid=8977 from=192.168.4.5
May 16 00:14:51 client-18 rshd[8978]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 17i: don\'t panic on short symlink ===================================================== 00:14:51 \(1305616491\)");echo XXRETCODE:$?'
May 16 00:14:51 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 17i: don't panic on short symlink ===================================================== 00:14:51 (1305616491)
May 16 00:14:51 client-18 xinetd[3390]: EXIT: shell status=0 pid=8977 duration=0(sec)
May 16 00:14:51 client-18 xinetd[3390]: START: shell pid=9012 from=192.168.4.5
May 16 00:14:51 client-18 rshd[9013]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 16 00:14:51 client-18 xinetd[3390]: EXIT: shell status=0 pid=9012 duration=0(sec)
May 16 00:14:51 client-18 xinetd[3390]: START: shell pid=9048 from=192.168.4.5
May 16 00:14:51 client-18 rshd[9049]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 16 00:14:51 client-18 xinetd[3390]: EXIT: shell status=0 pid=9048 duration=0(sec)
May 16 00:14:51 client-18 xinetd[3390]: START: shell pid=9085 from=192.168.4.5
May 16 00:14:51 client-18 rshd[9086]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 17k: symlinks: rsync with xattrs enabled ============================================== 00:14:51 \(1305616491\)");echo XXRETCODE:$?'
May 16 00:14:51 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 17k: symlinks: rsync with xattrs enabled ============================================== 00:14:51 (1305616491)
May 16 00:14:51 client-18 xinetd[3390]: EXIT: shell status=0 pid=9085 duration=0(sec)
May 16 00:14:52 client-18 xinetd[3390]: START: shell pid=9120 from=192.168.4.5
May 16 00:14:52 client-18 rshd[9121]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 16 00:14:52 client-18 xinetd[3390]: EXIT: shell status=0 pid=9120 duration=0(sec)
May 16 00:14:52 client-18 xinetd[3390]: START: shell pid=9156 from=192.168.4.5
May 16 00:14:52 client-18 rshd[9157]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 16 00:14:52 client-18 xinetd[3390]: EXIT: shell status=0 pid=9156 duration=0(sec)
May 16 00:14:52 client-18 xinetd[3390]: START: shell pid=9193 from=192.168.4.5
May 16 00:14:52 client-18 rshd[9194]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 18: touch ...\/f \; ls ... ============================================================== 00:14:52 \(1305616492\)");echo XXRETCODE:$?'
May 16 00:14:52 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 18: touch .../f ; ls ... ============================================================== 00:14:52 (1305616492)
May 16 00:14:52 client-18 xinetd[3390]: EXIT: shell status=0 pid=9193 duration=0(sec)
May 16 00:14:52 client-18 xinetd[3390]: START: shell pid=9228 from=192.168.4.5
May 16 00:14:52 client-18 rshd[9229]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 16 00:14:52 client-18 xinetd[3390]: EXIT: shell status=0 pid=9228 duration=0(sec)
May 16 00:14:52 client-18 xinetd[3390]: START: shell pid=9264 from=192.168.4.5
May 16 00:14:52 client-18 rshd[9265]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 16 00:14:52 client-18 xinetd[3390]: EXIT: shell status=0 pid=9264 duration=0(sec)
May 16 00:14:53 client-18 xinetd[3390]: START: shell pid=9301 from=192.168.4.5
May 16 00:14:53 client-18 rshd[9302]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 19a: touch ...\/f19 \; ls -l ... \; rm ...\/f19 =========================================== 00:14:52 \(1305616492\)");echo XXRETCODE:$?'
May 16 00:14:53 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 19a: touch .../f19 ; ls -l ... ; rm .../f19 =========================================== 00:14:52 (1305616492)
May 16 00:14:53 client-18 xinetd[3390]: EXIT: shell status=0 pid=9301 duration=0(sec)
May 16 00:14:53 client-18 xinetd[3390]: START: shell pid=9336 from=192.168.4.5
May 16 00:14:53 client-18 rshd[9337]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 16 00:14:53 client-18 xinetd[3390]: EXIT: shell status=0 pid=9336 duration=0(sec)
May 16 00:14:53 client-18 xinetd[3390]: START: shell pid=9372 from=192.168.4.5
May 16 00:14:53 client-18 rshd[9373]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 16 00:14:53 client-18 xinetd[3390]: EXIT: shell status=0 pid=9372 duration=0(sec)
May 16 00:14:53 client-18 xinetd[3390]: START: shell pid=9409 from=192.168.4.5
May 16 00:14:53 client-18 rshd[9410]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 19b: ls -l ...\/f19 \(should return error\) ============================================== 00:14:53 \(1305616493\)");echo XXRETCODE:$?'
May 16 00:14:53 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 19b: ls -l .../f19 (should return error) ============================================== 00:14:53 (1305616493)
May 16 00:14:53 client-18 xinetd[3390]: EXIT: shell status=0 pid=9409 duration=0(sec)
May 16 00:14:53 client-18 xinetd[3390]: START: shell pid=9444 from=192.168.4.5
May 16 00:14:53 client-18 rshd[9445]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 16 00:14:53 client-18 xinetd[3390]: EXIT: shell status=0 pid=9444 duration=0(sec)
May 16 00:14:53 client-18 xinetd[3390]: START: shell pid=9480 from=192.168.4.5
May 16 00:14:53 client-18 rshd[9481]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 16 00:14:53 client-18 xinetd[3390]: EXIT: shell status=0 pid=9480 duration=0(sec)
May 16 00:14:54 client-18 xinetd[3390]: START: shell pid=9517 from=192.168.4.5
May 16 00:14:54 client-18 rshd[9518]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 19c: runas -u 500 touch ...\/f19 \(should return error\) ================================= 00:14:53 \(1305616493\)");echo XXRETCODE:$?'
May 16 00:14:54 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 19c: runas -u 500 touch .../f19 (should return error) ================================= 00:14:53 (1305616493)
May 16 00:14:54 client-18 xinetd[3390]: EXIT: shell status=0 pid=9517 duration=0(sec)
May 16 00:14:54 client-18 xinetd[3390]: START: shell pid=9552 from=192.168.4.5
May 16 00:14:54 client-18 rshd[9553]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 16 00:14:54 client-18 xinetd[3390]: EXIT: shell status=0 pid=9552 duration=0(sec)
May 16 00:14:54 client-18 xinetd[3390]: START: shell pid=9588 from=192.168.4.5
May 16 00:14:54 client-18 rshd[9589]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 16 00:14:54 client-18 xinetd[3390]: EXIT: shell status=0 pid=9588 duration=0(sec)
May 16 00:14:54 client-18 xinetd[3390]: START: shell pid=9625 from=192.168.4.5
May 16 00:14:54 client-18 rshd[9626]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 19d: cat ...\/f19 \(should return error\) ================================================ 00:14:54 \(1305616494\)");echo XXRETCODE:$?'
May 16 00:14:54 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 19d: cat .../f19 (should return error) ================================================ 00:14:54 (1305616494)
May 16 00:14:54 client-18 xinetd[3390]: EXIT: shell status=0 pid=9625 duration=0(sec)
May 16 00:14:54 client-18 xinetd[3390]: START: shell pid=9660 from=192.168.4.5
May 16 00:14:54 client-18 rshd[9661]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 16 00:14:54 client-18 xinetd[3390]: EXIT: shell status=0 pid=9660 duration=0(sec)
May 16 00:14:54 client-18 xinetd[3390]: START: shell pid=9696 from=192.168.4.5
May 16 00:14:54 client-18 rshd[9697]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 16 00:14:54 client-18 xinetd[3390]: EXIT: shell status=0 pid=9696 duration=0(sec)
May 16 00:14:55 client-18 xinetd[3390]: START: shell pid=9733 from=192.168.4.5
May 16 00:14:55 client-18 rshd[9734]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 20: touch ...\/f \; ls -l ... =========================================================== 00:14:54 \(1305616494\)");echo XXRETCODE:$?'
May 16 00:14:55 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 20: touch .../f ; ls -l ... =========================================================== 00:14:54 (1305616494)
May 16 00:14:55 client-18 xinetd[3390]: EXIT: shell status=0 pid=9733 duration=0(sec)
May 16 00:14:55 client-18 xinetd[3390]: START: shell pid=9768 from=192.168.4.5
May 16 00:14:55 client-18 rshd[9769]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark 1 done");echo XXRETCODE:$?'
May 16 00:14:55 client-18 kernel: Lustre: DEBUG MARKER: 1 done
May 16 00:14:55 client-18 xinetd[3390]: EXIT: shell status=0 pid=9768 duration=0(sec)
May 16 00:14:55 client-18 xinetd[3390]: START: shell pid=9803 from=192.168.4.5
May 16 00:14:55 client-18 rshd[9804]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark 2 done");echo XXRETCODE:$?'
May 16 00:14:55 client-18 kernel: Lustre: DEBUG MARKER: 2 done
May 16 00:14:55 client-18 xinetd[3390]: EXIT: shell status=0 pid=9803 duration=0(sec)
May 16 00:14:55 client-18 xinetd[3390]: START: shell pid=9838 from=192.168.4.5
May 16 00:14:55 client-18 rshd[9839]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark 3 done");echo XXRETCODE:$?'
May 16 00:14:55 client-18 kernel: Lustre: DEBUG MARKER: 3 done
May 16 00:14:55 client-18 xinetd[3390]: EXIT: shell status=0 pid=9838 duration=0(sec)
May 16 00:14:55 client-18 xinetd[3390]: START: shell pid=9873 from=192.168.4.5
May 16 00:14:55 client-18 rshd[9874]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 16 00:14:55 client-18 xinetd[3390]: EXIT: shell status=0 pid=9873 duration=0(sec)
May 16 00:14:55 client-18 xinetd[3390]: START: shell pid=9909 from=192.168.4.5
May 16 00:14:55 client-18 rshd[9910]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 16 00:14:55 client-18 xinetd[3390]: EXIT: shell status=0 pid=9909 duration=0(sec)
May 16 00:14:56 client-18 xinetd[3390]: START: shell pid=9946 from=192.168.4.5
May 16 00:14:56 client-18 rshd[9947]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 21: write to dangling link ============================================================ 00:14:55 \(1305616495\)");echo XXRETCODE:$?'
May 16 00:14:56 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 21: write to dangling link ============================================================ 00:14:55 (1305616495)
May 16 00:14:56 client-18 xinetd[3390]: EXIT: shell status=0 pid=9946 duration=0(sec)
May 16 00:14:56 client-18 xinetd[3390]: START: shell pid=9981 from=192.168.4.5
May 16 00:14:56 client-18 rshd[9982]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 16 00:14:56 client-18 xinetd[3390]: EXIT: shell status=0 pid=9981 duration=0(sec)
May 16 00:14:56 client-18 xinetd[3390]: START: shell pid=10017 from=192.168.4.5
May 16 00:14:56 client-18 rshd[10018]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 16 00:14:56 client-18 xinetd[3390]: EXIT: shell status=0 pid=10017 duration=0(sec)
May 16 00:14:56 client-18 xinetd[3390]: START: shell pid=10054 from=192.168.4.5
May 16 00:14:56 client-18 rshd[10055]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 22: unpack tar archive as non-root user =============================================== 00:14:56 \(1305616496\)");echo XXRETCODE:$?'
May 16 00:14:56 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 22: unpack tar archive as non-root user =============================================== 00:14:56 (1305616496)
May 16 00:14:56 client-18 xinetd[3390]: EXIT: shell status=0 pid=10054 duration=0(sec)
May 16 00:14:56 client-18 xinetd[3390]: START: shell pid=10089 from=192.168.4.5
May 16 00:14:56 client-18 rshd[10090]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 16 00:14:56 client-18 xinetd[3390]: EXIT: shell status=0 pid=10089 duration=0(sec)
May 16 00:14:56 client-18 xinetd[3390]: START: shell pid=10125 from=192.168.4.5
May 16 00:14:56 client-18 rshd[10126]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 16 00:14:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=10125 duration=1(sec)
May 16 00:14:57 client-18 xinetd[3390]: START: shell pid=10162 from=192.168.4.5
May 16 00:14:57 client-18 rshd[10163]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 23a: O_CREAT\|O_EXCL in subdir ========================================================= 00:14:57 \(1305616497\)");echo XXRETCODE:$?'
May 16 00:14:57 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 23a: O_CREAT|O_EXCL in subdir ========================================================= 00:14:57 (1305616497)
May 16 00:14:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=10162 duration=0(sec)
May 16 00:14:57 client-18 xinetd[3390]: START: shell pid=10197 from=192.168.4.5
May 16 00:14:57 client-18 rshd[10198]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 16 00:14:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=10197 duration=0(sec)
May 16 00:14:57 client-18 xinetd[3390]: START: shell pid=10233 from=192.168.4.5
May 16 00:14:57 client-18 rshd[10234]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 16 00:14:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=10233 duration=0(sec)
May 16 00:14:57 client-18 xinetd[3390]: START: shell pid=10270 from=192.168.4.5
May 16 00:14:57 client-18 rshd[10271]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 23b: O_APPEND check =================================================================== 00:14:57 \(1305616497\)");echo XXRETCODE:$?'
May 16 00:14:57 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 23b: O_APPEND check =================================================================== 00:14:57 (1305616497)
May 16 00:14:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=10270 duration=0(sec)
May 16 00:14:57 client-18 xinetd[3390]: START: shell pid=10305 from=192.168.4.5
May 16 00:14:57 client-18 rshd[10306]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 16 00:14:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=10305 duration=0(sec)
May 16 00:14:57 client-18 xinetd[3390]: START: shell pid=10341 from=192.168.4.5
May 16 00:14:58 client-18 rshd[10342]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 16 00:14:58 client-18 xinetd[3390]: EXIT: shell status=0 pid=10341 duration=1(sec)
May 16 00:14:58 client-18 xinetd[3390]: START: shell pid=10378 from=192.168.4.5
May 16 00:14:58 client-18 rshd[10379]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 24a: touch ...\/R1\/f\; rename ...\/R1\/f ...\/R1\/g ========================================= 00:14:58 \(1305616498\)");echo XXRETCODE:$?'
May 16 00:14:58 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 24a: touch .../R1/f; rename .../R1/f .../R1/g ========================================= 00:14:58 (1305616498)
May 16 00:14:58 client-18 xinetd[3390]: EXIT: shell status=0 pid=10378 duration=0(sec)
May 16 00:14:58 client-18 xinetd[3390]: START: shell pid=10413 from=192.168.4.5
May 16 00:14:58 client-18 rshd[10414]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 16 00:14:58 client-18 xinetd[3390]: EXIT: shell status=0 pid=10413 duration=0(sec)
May 16 00:14:58 client-18 xinetd[3390]: START: shell pid=10449 from=192.168.4.5
May 16 00:14:58 client-18 rshd[10450]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 16 00:14:58 client-18 xinetd[3390]: EXIT: shell status=0 pid=10449 duration=0(sec)
May 16 00:14:58 client-18 xinetd[3390]: START: shell pid=10486 from=192.168.4.5
May 16 00:14:58 client-18 rshd[10487]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 24b: touch ...\/R2\/{f,g}\; rename ...\/R2\/f ...\/R2\/g ===================================== 00:14:58 \(1305616498\)");echo XXRETCODE:$?'
May 16 00:14:58 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 24b: touch .../R2/f; .../R2/g; rename .../R2/f .../R2/g ===================================== 00:14:58 (1305616498)
May 16 00:14:58 client-18 xinetd[3390]: EXIT: shell status=0 pid=10486 duration=0(sec)
May 16 00:14:58 client-18 xinetd[3390]: START: shell pid=10521 from=192.168.4.5
May 16 00:14:58 client-18 rshd[10522]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 16 00:14:58 client-18 xinetd[3390]: EXIT: shell status=0 pid=10521 duration=0(sec)
May 16 00:14:59 client-18 xinetd[3390]: START: shell pid=10557 from=192.168.4.5
May 16 00:14:59 client-18 rshd[10558]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 16 00:14:59 client-18 xinetd[3390]: EXIT: shell status=0 pid=10557 duration=0(sec)
May 16 00:14:59 client-18 xinetd[3390]: START: shell pid=10594 from=192.168.4.5
May 16 00:14:59 client-18 rshd[10595]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 24c: mkdir ...\/R3\/f\; rename ...\/R3\/f ...\/R3\/g ========================================= 00:14:59 \(1305616499\)");echo XXRETCODE:$?'
May 16 00:14:59 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 24c: mkdir .../R3/f; rename .../R3/f .../R3/g ========================================= 00:14:59 (1305616499)
May 16 00:14:59 client-18 xinetd[3390]: EXIT: shell status=0 pid=10594 duration=0(sec)
May 16 00:14:59 client-18 xinetd[3390]: START: shell pid=10629 from=192.168.4.5
May 16 00:14:59 client-18 rshd[10630]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 16 00:14:59 client-18 xinetd[3390]: EXIT: shell status=0 pid=10629 duration=0(sec)
May 16 00:14:59 client-18 xinetd[3390]: START: shell pid=10665 from=192.168.4.5
May 16 00:14:59 client-18 rshd[10666]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 16 00:14:59 client-18 xinetd[3390]: EXIT: shell status=0 pid=10665 duration=0(sec)
May 16 00:14:59 client-18 xinetd[3390]: START: shell pid=10702 from=192.168.4.5
May 16 00:14:59 client-18 rshd[10703]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 24d: mkdir ...\/R4\/{f,g}\; rename ...\/R4\/f ...\/R4\/g ===================================== 00:14:59 \(1305616499\)");echo XXRETCODE:$?'
May 16 00:14:59 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 24d: mkdir .../R4/f; .../R4/g; rename .../R4/f .../R4/g ===================================== 00:14:59 (1305616499)
May 16 00:14:59 client-18 xinetd[3390]: EXIT: shell status=0 pid=10702 duration=0(sec)
May 16 00:14:59 client-18 xinetd[3390]: START: shell pid=10737 from=192.168.4.5
May 16 00:14:59 client-18 rshd[10738]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 16 00:15:00 client-18 xinetd[3390]: EXIT: shell status=0 pid=10737 duration=1(sec)
May 16 00:15:00 client-18 xinetd[3390]: START: shell pid=10773 from=192.168.4.5
May 16 00:15:00 client-18 rshd[10774]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 16 00:15:00 client-18 xinetd[3390]: EXIT: shell status=0 pid=10773 duration=0(sec)
May 16 00:15:00 client-18 xinetd[3390]: START: shell pid=10810 from=192.168.4.5
May 16 00:15:00 client-18 rshd[10811]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 24e: touch ...\/R5a\/f\; rename ...\/R5a\/f ...\/R5b\/g ====================================== 00:15:00 \(1305616500\)");echo XXRETCODE:$?'
May 16 00:15:00 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 24e: touch .../R5a/f; rename .../R5a/f .../R5b/g ====================================== 00:15:00 (1305616500)
May 16 00:15:00 client-18 xinetd[3390]: EXIT: shell status=0 pid=10810 duration=0(sec)
May 16 00:15:00 client-18 xinetd[3390]: START: shell pid=10845 from=192.168.4.5
May 16 00:15:00 client-18 rshd[10846]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 16 00:15:00 client-18 xinetd[3390]: EXIT: shell status=0 pid=10845 duration=0(sec)
May 16 00:15:00 client-18 xinetd[3390]: START: shell pid=10881 from=192.168.4.5
May 16 00:15:00 client-18 rshd[10882]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 16 00:15:00 client-18 xinetd[3390]: EXIT: shell status=0 pid=10881 duration=0(sec)
May 16 00:15:00 client-18 xinetd[3390]: START: shell pid=10918 from=192.168.4.5
May 16 00:15:00 client-18 rshd[10919]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 24f: touch ...\/R6a\/f R6b\/g\; mv ...\/R6a\/f ...\/R6b\/g ==================================== 00:15:00 \(1305616500\)");echo XXRETCODE:$?'
May 16 00:15:00 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 24f: touch .../R6a/f R6b/g; mv .../R6a/f .../R6b/g ==================================== 00:15:00 (1305616500)
May 16 00:15:00 client-18 xinetd[3390]: EXIT: shell status=0 pid=10918 duration=0(sec)
May 16 00:15:00 client-18 xinetd[3390]: START: shell pid=10953 from=192.168.4.5
May 16 00:15:00 client-18 rshd[10954]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 16 00:15:01 client-18 xinetd[3390]: EXIT: shell status=0 pid=10953 duration=1(sec)
May 16 00:15:01 client-18 xinetd[3390]: START: shell pid=10989 from=192.168.4.5
May 16 00:15:01 client-18 rshd[10990]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 16 00:15:01 client-18 xinetd[3390]: EXIT: shell status=0 pid=10989 duration=0(sec)
May 16 00:15:01 client-18 xinetd[3390]: START: shell pid=11026 from=192.168.4.5
May 16 00:15:01 client-18 rshd[11027]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 24g: mkdir ...\/R7{a,b}\/d\; mv ...\/R7a\/d ...\/R7b\/e ====================================== 00:15:01 \(1305616501\)");echo XXRETCODE:$?'
May 16 00:15:01 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 24g: mkdir .../R7a/d; .../R7b/d; mv .../R7a/d .../R7b/e ====================================== 00:15:01 (1305616501)
May 16 00:15:01 client-18 xinetd[3390]: EXIT: shell status=0 pid=11026 duration=0(sec)
May 16 00:15:01 client-18 xinetd[3390]: START: shell pid=11061 from=192.168.4.5
May 16 00:15:01 client-18 rshd[11062]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 16 00:15:01 client-18 xinetd[3390]: EXIT: shell status=0 pid=11061 duration=0(sec)
May 16 00:15:01 client-18 xinetd[3390]: START: shell pid=11097 from=192.168.4.5
May 16 00:15:01 client-18 rshd[11098]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 16 00:15:01 client-18 xinetd[3390]: EXIT: shell status=0 pid=11097 duration=0(sec)
May 16 00:15:01 client-18 xinetd[3390]: START: shell pid=11134 from=192.168.4.5
May 16 00:15:01 client-18 rshd[11135]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 24h: mkdir ...\/R8{a,b}\/{d,e}\; rename ...\/R8a\/d ...\/R8b\/e ============================== 00:15:01 \(1305616501\)");echo XXRETCODE:$?'
May 16 00:15:01 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 24h: mkdir .../R8a/d; .../R8a/e; .../R8b/d; .../R8b/e; rename .../R8a/d .../R8b/e ============================== 00:15:01 (1305616501)
May 16 00:15:01 client-18 xinetd[3390]: EXIT: shell status=0 pid=11134 duration=0(sec)
May 16 00:15:01 client-18 xinetd[3390]: START: shell pid=11169 from=192.168.4.5
May 16 00:15:02 client-18 rshd[11170]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 16 00:15:02 client-18 xinetd[3390]: EXIT: shell status=0 pid=11169 duration=1(sec)
May 16 00:15:02 client-18 xinetd[3390]: START: shell pid=11205 from=192.168.4.5
May 16 00:15:02 client-18 rshd[11206]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 16 00:15:02 client-18 xinetd[3390]: EXIT: shell status=0 pid=11205 duration=0(sec)
May 16 00:15:02 client-18 xinetd[3390]: START: shell pid=11242 from=192.168.4.5
May 16 00:15:02 client-18 rshd[11243]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 24i: rename file to dir error: touch f \; mkdir a \; rename f a ========================= 00:15:02 \(1305616502\)");echo XXRETCODE:$?'
May 16 00:15:02 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 24i: rename file to dir error: touch f ; mkdir a ; rename f a ========================= 00:15:02 (1305616502)
May 16 00:15:02 client-18 xinetd[3390]: EXIT: shell status=0 pid=11242 duration=0(sec)
May 16 00:15:02 client-18 xinetd[3390]: START: shell pid=11277 from=192.168.4.5
May 16 00:15:02 client-18 rshd[11278]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 16 00:15:02 client-18 xinetd[3390]: EXIT: shell status=0 pid=11277 duration=0(sec)
May 16 00:15:02 client-18 xinetd[3390]: START: shell pid=11313 from=192.168.4.5
May 16 00:15:02 client-18 rshd[11314]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 16 00:15:02 client-18 xinetd[3390]: EXIT: shell status=0 pid=11313 duration=0(sec)
May 16 00:15:02 client-18 xinetd[3390]: START: shell pid=11350 from=192.168.4.5
May 16 00:15:02 client-18 rshd[11351]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 24j: source does not exist ============================================================ 00:15:02 \(1305616502\)");echo XXRETCODE:$?'
May 16 00:15:02 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 24j: source does not exist ============================================================ 00:15:02 (1305616502)
May 16 00:15:02 client-18 xinetd[3390]: EXIT: shell status=0 pid=11350 duration=0(sec)
May 16 00:15:03 client-18 xinetd[3390]: START: shell pid=11385 from=192.168.4.5
May 16 00:15:03 client-18 rshd[11386]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 16 00:15:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=11385 duration=0(sec)
May 16 00:15:03 client-18 xinetd[3390]: START: shell pid=11421 from=192.168.4.5
May 16 00:15:03 client-18 rshd[11422]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 16 00:15:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=11421 duration=0(sec)
May 16 00:15:03 client-18 xinetd[3390]: START: shell pid=11458 from=192.168.4.5
May 16 00:15:03 client-18 rshd[11459]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 24k: touch ...\/R11a\/f\; mv ...\/R11a\/f ...\/R11a\/d ======================================= 00:15:03 \(1305616503\)");echo XXRETCODE:$?'
May 16 00:15:03 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 24k: touch .../R11a/f; mv .../R11a/f .../R11a/d ======================================= 00:15:03 (1305616503)
May 16 00:15:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=11458 duration=0(sec)
May 16 00:15:03 client-18 xinetd[3390]: START: shell pid=11493 from=192.168.4.5
May 16 00:15:03 client-18 rshd[11494]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 16 00:15:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=11493 duration=0(sec)
May 16 00:15:03 client-18 xinetd[3390]: START: shell pid=11529 from=192.168.4.5
May 16 00:15:03 client-18 rshd[11530]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 16 00:15:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=11529 duration=0(sec)
May 16 00:15:03 client-18 xinetd[3390]: START: shell pid=11566 from=192.168.4.5
May 16 00:15:03 client-18 rshd[11567]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 24l: Renaming a file to itself ======================================================== 00:15:03 \(1305616503\)");echo XXRETCODE:$?'
May 16 00:15:03 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 24l: Renaming a file to itself ======================================================== 00:15:03 (1305616503)
May 16 00:15:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=11566 duration=1(sec)
May 16 00:15:04 client-18 xinetd[3390]: START: shell pid=11601 from=192.168.4.5
May 16 00:15:04 client-18 rshd[11602]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 16 00:15:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=11601 duration=0(sec)
May 16 00:15:04 client-18 xinetd[3390]: START: shell pid=11637 from=192.168.4.5
May 16 00:15:04 client-18 rshd[11638]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 16 00:15:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=11637 duration=0(sec)
May 16 00:15:04 client-18 xinetd[3390]: START: shell pid=11674 from=192.168.4.5
May 16 00:15:04 client-18 rshd[11675]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 24m: Renaming a file to a hard link to itself ========================================= 00:15:04 \(1305616504\)");echo XXRETCODE:$?'
May 16 00:15:04 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 24m: Renaming a file to a hard link to itself ========================================= 00:15:04 (1305616504)
May 16 00:15:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=11674 duration=0(sec)
May 16 00:15:04 client-18 xinetd[3390]: START: shell pid=11709 from=192.168.4.5
May 16 00:15:04 client-18 rshd[11710]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 16 00:15:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=11709 duration=0(sec)
May 16 00:15:04 client-18 xinetd[3390]: START: shell pid=11745 from=192.168.4.5
May 16 00:15:04 client-18 rshd[11746]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 16 00:15:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=11745 duration=0(sec)
May 16 00:15:04 client-18 xinetd[3390]: START: shell pid=11782 from=192.168.4.5
May 16 00:15:04 client-18 rshd[11783]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 24n: Statting the old file after renaming \(Posix rename 2\) ============================ 00:15:04 \(1305616504\)");echo XXRETCODE:$?'
May 16 00:15:05 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 24n: Statting the old file after renaming (Posix rename 2) ============================ 00:15:04 (1305616504)
May 16 00:15:05 client-18 xinetd[3390]: EXIT: shell status=0 pid=11782 duration=1(sec)
May 16 00:15:05 client-18 xinetd[3390]: START: shell pid=11817 from=192.168.4.5
May 16 00:15:05 client-18 rshd[11818]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 16 00:15:05 client-18 xinetd[3390]: EXIT: shell status=0 pid=11817 duration=0(sec)
May 16 00:15:05 client-18 xinetd[3390]: START: shell pid=11853 from=192.168.4.5
May 16 00:15:05 client-18 rshd[11854]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 16 00:15:05 client-18 xinetd[3390]: EXIT: shell status=0 pid=11853 duration=0(sec)
May 16 00:15:05 client-18 xinetd[3390]: START: shell pid=11890 from=192.168.4.5
May 16 00:15:05 client-18 rshd[11891]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 24o: rename of files during htree split =============================================== 00:15:05 \(1305616505\)");echo XXRETCODE:$?'
May 16 00:15:05 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 24o: rename of files during htree split =============================================== 00:15:05 (1305616505)
May 16 00:15:05 client-18 xinetd[3390]: EXIT: shell status=0 pid=11890 duration=0(sec)
May 16 00:15:44 client-18 ntpd[3401]: synchronized to 10.10.0.1, stratum 2
May 17 01:15:44 client-18 ntpd[3401]: time reset +90000.003029 s
May 17 01:16:05 client-18 xinetd[3390]: START: shell pid=11931 from=192.168.4.5
May 17 01:16:05 client-18 rshd[11932]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:16:05 client-18 xinetd[3390]: EXIT: shell status=0 pid=11931 duration=0(sec)
May 17 01:16:05 client-18 xinetd[3390]: START: shell pid=11967 from=192.168.4.5
May 17 01:16:05 client-18 rshd[11968]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:16:05 client-18 xinetd[3390]: EXIT: shell status=0 pid=11967 duration=0(sec)
May 17 01:16:05 client-18 xinetd[3390]: START: shell pid=12004 from=192.168.4.5
May 17 01:16:05 client-18 rshd[12005]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 24p: mkdir ...\/R12{a,b}\; rename ...\/R12a ...\/R12b ===================================== 01:16:05 \(1305620165\)");echo XXRETCODE:$?'
May 17 01:16:05 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 24p: mkdir .../R12a; .../R12b; rename .../R12a .../R12b ===================================== 01:16:05 (1305620165)
May 17 01:16:05 client-18 xinetd[3390]: EXIT: shell status=0 pid=12004 duration=0(sec)
May 17 01:16:05 client-18 xinetd[3390]: START: shell pid=12039 from=192.168.4.5
May 17 01:16:05 client-18 rshd[12040]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:16:06 client-18 xinetd[3390]: EXIT: shell status=0 pid=12039 duration=1(sec)
May 17 01:16:06 client-18 xinetd[3390]: START: shell pid=12075 from=192.168.4.5
May 17 01:16:06 client-18 rshd[12076]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:16:06 client-18 xinetd[3390]: EXIT: shell status=0 pid=12075 duration=0(sec)
May 17 01:16:06 client-18 xinetd[3390]: START: shell pid=12112 from=192.168.4.5
May 17 01:16:06 client-18 rshd[12113]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 24q: mkdir ...\/R13{a,b}\; open R13b rename R13a R13b =================================== 01:16:06 \(1305620166\)");echo XXRETCODE:$?'
May 17 01:16:06 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 24q: mkdir .../R13a; .../R13b; open R13b rename R13a R13b =================================== 01:16:06 (1305620166)
May 17 01:16:06 client-18 xinetd[3390]: EXIT: shell status=0 pid=12112 duration=0(sec)
May 17 01:16:06 client-18 xinetd[3390]: START: shell pid=12147 from=192.168.4.5
May 17 01:16:06 client-18 rshd[12148]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:16:06 client-18 xinetd[3390]: EXIT: shell status=0 pid=12147 duration=0(sec)
May 17 01:16:06 client-18 xinetd[3390]: START: shell pid=12183 from=192.168.4.5
May 17 01:16:06 client-18 rshd[12184]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:16:06 client-18 xinetd[3390]: EXIT: shell status=0 pid=12183 duration=0(sec)
May 17 01:16:06 client-18 xinetd[3390]: START: shell pid=12220 from=192.168.4.5
May 17 01:16:06 client-18 rshd[12221]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 24r: mkdir ...\/R14a\/b\; rename ...\/R14a ...\/R14a\/b ===================================== 01:16:06 \(1305620166\)");echo XXRETCODE:$?'
May 17 01:16:06 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 24r: mkdir .../R14a/b; rename .../R14a .../R14a/b ===================================== 01:16:06 (1305620166)
May 17 01:16:06 client-18 xinetd[3390]: EXIT: shell status=0 pid=12220 duration=0(sec)
May 17 01:16:07 client-18 xinetd[3390]: START: shell pid=12255 from=192.168.4.5
May 17 01:16:07 client-18 rshd[12256]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:16:07 client-18 xinetd[3390]: EXIT: shell status=0 pid=12255 duration=0(sec)
May 17 01:16:07 client-18 xinetd[3390]: START: shell pid=12291 from=192.168.4.5
May 17 01:16:07 client-18 rshd[12292]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:16:07 client-18 xinetd[3390]: EXIT: shell status=0 pid=12291 duration=0(sec)
May 17 01:16:07 client-18 xinetd[3390]: START: shell pid=12328 from=192.168.4.5
May 17 01:16:07 client-18 rshd[12329]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 24s: mkdir ...\/R15a\/b\/c\; rename ...\/R15a ...\/R15a\/b\/c ================================= 01:16:07 \(1305620167\)");echo XXRETCODE:$?'
May 17 01:16:07 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 24s: mkdir .../R15a/b/c; rename .../R15a .../R15a/b/c ================================= 01:16:07 (1305620167)
May 17 01:16:07 client-18 xinetd[3390]: EXIT: shell status=0 pid=12328 duration=0(sec)
May 17 01:16:07 client-18 xinetd[3390]: START: shell pid=12363 from=192.168.4.5
May 17 01:16:07 client-18 rshd[12364]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:16:07 client-18 xinetd[3390]: EXIT: shell status=0 pid=12363 duration=0(sec)
May 17 01:16:07 client-18 xinetd[3390]: START: shell pid=12399 from=192.168.4.5
May 17 01:16:07 client-18 rshd[12400]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:16:07 client-18 xinetd[3390]: EXIT: shell status=0 pid=12399 duration=0(sec)
May 17 01:16:07 client-18 xinetd[3390]: START: shell pid=12436 from=192.168.4.5
May 17 01:16:07 client-18 rshd[12437]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 24t: mkdir ...\/R16a\/b\/c\; rename ...\/R16a\/b\/c ...\/R16a ================================= 01:16:07 \(1305620167\)");echo XXRETCODE:$?'
May 17 01:16:07 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 24t: mkdir .../R16a/b/c; rename .../R16a/b/c .../R16a ================================= 01:16:07 (1305620167)
May 17 01:16:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=12436 duration=1(sec)
May 17 01:16:08 client-18 xinetd[3390]: START: shell pid=12471 from=192.168.4.5
May 17 01:16:08 client-18 rshd[12472]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:16:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=12471 duration=0(sec)
May 17 01:16:08 client-18 xinetd[3390]: START: shell pid=12507 from=192.168.4.5
May 17 01:16:08 client-18 rshd[12508]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:16:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=12507 duration=0(sec)
May 17 01:16:08 client-18 xinetd[3390]: START: shell pid=12544 from=192.168.4.5
May 17 01:16:08 client-18 rshd[12545]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 24u: create stripe file =============================================================== 01:16:08 \(1305620168\)");echo XXRETCODE:$?'
May 17 01:16:08 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 24u: create stripe file =============================================================== 01:16:08 (1305620168)
May 17 01:16:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=12544 duration=0(sec)
May 17 01:16:08 client-18 xinetd[3390]: START: shell pid=12579 from=192.168.4.5
May 17 01:16:08 client-18 rshd[12580]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:16:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=12579 duration=0(sec)
May 17 01:16:08 client-18 xinetd[3390]: START: shell pid=12615 from=192.168.4.5
May 17 01:16:08 client-18 rshd[12616]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:16:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=12615 duration=0(sec)
May 17 01:16:08 client-18 xinetd[3390]: START: shell pid=12652 from=192.168.4.5
May 17 01:16:09 client-18 rshd[12653]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 24v: list directory with large files \(handle hash collision, bug: 17560\) ============== 01:16:08 \(1305620168\)");echo XXRETCODE:$?'
May 17 01:16:09 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 24v: list directory with large files (handle hash collision, bug: 17560) ============== 01:16:08 (1305620168)
May 17 01:16:09 client-18 xinetd[3390]: EXIT: shell status=0 pid=12652 duration=1(sec)
May 17 01:20:43 client-18 ntpd[3401]: synchronized to 10.10.0.1, stratum 2
May 17 01:21:23 client-18 xinetd[3390]: START: shell pid=12703 from=192.168.4.5
May 17 01:21:23 client-18 rshd[12704]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:21:23 client-18 xinetd[3390]: EXIT: shell status=0 pid=12703 duration=0(sec)
May 17 01:21:23 client-18 xinetd[3390]: START: shell pid=12739 from=192.168.4.5
May 17 01:21:23 client-18 rshd[12740]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:21:23 client-18 xinetd[3390]: EXIT: shell status=0 pid=12739 duration=0(sec)
May 17 01:21:23 client-18 xinetd[3390]: START: shell pid=12776 from=192.168.4.5
May 17 01:21:23 client-18 rshd[12777]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 24w: Reading a file larger than 4Gb =================================================== 01:21:23 \(1305620483\)");echo XXRETCODE:$?'
May 17 01:21:24 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 24w: Reading a file larger than 4Gb =================================================== 01:21:23 (1305620483)
May 17 01:21:24 client-18 xinetd[3390]: EXIT: shell status=0 pid=12776 duration=1(sec)
May 17 01:21:24 client-18 xinetd[3390]: START: shell pid=12811 from=192.168.4.5
May 17 01:21:24 client-18 rshd[12812]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:21:24 client-18 xinetd[3390]: EXIT: shell status=0 pid=12811 duration=0(sec)
May 17 01:21:24 client-18 xinetd[3390]: START: shell pid=12847 from=192.168.4.5
May 17 01:21:24 client-18 rshd[12848]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:21:24 client-18 xinetd[3390]: EXIT: shell status=0 pid=12847 duration=0(sec)
May 17 01:21:24 client-18 xinetd[3390]: START: shell pid=12884 from=192.168.4.5
May 17 01:21:24 client-18 rshd[12885]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 25a: create file in symlinked directory =============================================== 01:21:24 \(1305620484\)");echo XXRETCODE:$?'
May 17 01:21:24 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 25a: create file in symlinked directory =============================================== 01:21:24 (1305620484)
May 17 01:21:24 client-18 xinetd[3390]: EXIT: shell status=0 pid=12884 duration=0(sec)
May 17 01:21:24 client-18 xinetd[3390]: START: shell pid=12919 from=192.168.4.5
May 17 01:21:24 client-18 rshd[12920]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:21:24 client-18 xinetd[3390]: EXIT: shell status=0 pid=12919 duration=0(sec)
May 17 01:21:24 client-18 xinetd[3390]: START: shell pid=12955 from=192.168.4.5
May 17 01:21:24 client-18 rshd[12956]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:21:24 client-18 xinetd[3390]: EXIT: shell status=0 pid=12955 duration=0(sec)
May 17 01:21:25 client-18 xinetd[3390]: START: shell pid=12992 from=192.168.4.5
May 17 01:21:25 client-18 rshd[12993]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 25b: lookup file in symlinked directory =============================================== 01:21:24 \(1305620484\)");echo XXRETCODE:$?'
May 17 01:21:25 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 25b: lookup file in symlinked directory =============================================== 01:21:24 (1305620484)
May 17 01:21:25 client-18 xinetd[3390]: EXIT: shell status=0 pid=12992 duration=0(sec)
May 17 01:21:25 client-18 xinetd[3390]: START: shell pid=13027 from=192.168.4.5
May 17 01:21:25 client-18 rshd[13028]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:21:25 client-18 xinetd[3390]: EXIT: shell status=0 pid=13027 duration=0(sec)
May 17 01:21:25 client-18 xinetd[3390]: START: shell pid=13063 from=192.168.4.5
May 17 01:21:25 client-18 rshd[13064]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:21:25 client-18 xinetd[3390]: EXIT: shell status=0 pid=13063 duration=0(sec)
May 17 01:21:25 client-18 xinetd[3390]: START: shell pid=13100 from=192.168.4.5
May 17 01:21:25 client-18 rshd[13101]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 26a: multiple component symlink ======================================================= 01:21:25 \(1305620485\)");echo XXRETCODE:$?'
May 17 01:21:25 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 26a: multiple component symlink ======================================================= 01:21:25 (1305620485)
May 17 01:21:25 client-18 xinetd[3390]: EXIT: shell status=0 pid=13100 duration=0(sec)
May 17 01:21:25 client-18 xinetd[3390]: START: shell pid=13135 from=192.168.4.5
May 17 01:21:25 client-18 rshd[13136]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:21:25 client-18 xinetd[3390]: EXIT: shell status=0 pid=13135 duration=0(sec)
May 17 01:21:25 client-18 xinetd[3390]: START: shell pid=13171 from=192.168.4.5
May 17 01:21:25 client-18 rshd[13172]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:21:25 client-18 xinetd[3390]: EXIT: shell status=0 pid=13171 duration=0(sec)
May 17 01:21:26 client-18 xinetd[3390]: START: shell pid=13208 from=192.168.4.5
May 17 01:21:26 client-18 rshd[13209]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 26b: multiple component symlink at end of lookup ====================================== 01:21:25 \(1305620485\)");echo XXRETCODE:$?'
May 17 01:21:26 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 26b: multiple component symlink at end of lookup ====================================== 01:21:25 (1305620485)
May 17 01:21:26 client-18 xinetd[3390]: EXIT: shell status=0 pid=13208 duration=0(sec)
May 17 01:21:26 client-18 xinetd[3390]: START: shell pid=13243 from=192.168.4.5
May 17 01:21:26 client-18 rshd[13244]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:21:26 client-18 xinetd[3390]: EXIT: shell status=0 pid=13243 duration=0(sec)
May 17 01:21:26 client-18 xinetd[3390]: START: shell pid=13279 from=192.168.4.5
May 17 01:21:26 client-18 rshd[13280]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:21:26 client-18 xinetd[3390]: EXIT: shell status=0 pid=13279 duration=0(sec)
May 17 01:21:26 client-18 xinetd[3390]: START: shell pid=13316 from=192.168.4.5
May 17 01:21:26 client-18 rshd[13317]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 26c: chain of symlinks ================================================================ 01:21:26 \(1305620486\)");echo XXRETCODE:$?'
May 17 01:21:26 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 26c: chain of symlinks ================================================================ 01:21:26 (1305620486)
May 17 01:21:26 client-18 xinetd[3390]: EXIT: shell status=0 pid=13316 duration=0(sec)
May 17 01:21:26 client-18 xinetd[3390]: START: shell pid=13351 from=192.168.4.5
May 17 01:21:26 client-18 rshd[13352]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:21:26 client-18 xinetd[3390]: EXIT: shell status=0 pid=13351 duration=0(sec)
May 17 01:21:26 client-18 xinetd[3390]: START: shell pid=13387 from=192.168.4.5
May 17 01:21:27 client-18 rshd[13388]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:21:27 client-18 xinetd[3390]: EXIT: shell status=0 pid=13387 duration=1(sec)
May 17 01:21:27 client-18 xinetd[3390]: START: shell pid=13424 from=192.168.4.5
May 17 01:21:27 client-18 rshd[13425]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 26d: create multiple component recursive symlink ====================================== 01:21:27 \(1305620487\)");echo XXRETCODE:$?'
May 17 01:21:27 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 26d: create multiple component recursive symlink ====================================== 01:21:27 (1305620487)
May 17 01:21:27 client-18 xinetd[3390]: EXIT: shell status=0 pid=13424 duration=0(sec)
May 17 01:21:27 client-18 xinetd[3390]: START: shell pid=13459 from=192.168.4.5
May 17 01:21:27 client-18 rshd[13460]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:21:27 client-18 xinetd[3390]: EXIT: shell status=0 pid=13459 duration=0(sec)
May 17 01:21:27 client-18 xinetd[3390]: START: shell pid=13495 from=192.168.4.5
May 17 01:21:27 client-18 rshd[13496]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:21:27 client-18 xinetd[3390]: EXIT: shell status=0 pid=13495 duration=0(sec)
May 17 01:21:27 client-18 xinetd[3390]: START: shell pid=13532 from=192.168.4.5
May 17 01:21:27 client-18 rshd[13533]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 26e: unlink multiple component recursive symlink ====================================== 01:21:27 \(1305620487\)");echo XXRETCODE:$?'
May 17 01:21:27 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 26e: unlink multiple component recursive symlink ====================================== 01:21:27 (1305620487)
May 17 01:21:27 client-18 xinetd[3390]: EXIT: shell status=0 pid=13532 duration=0(sec)
May 17 01:21:27 client-18 xinetd[3390]: START: shell pid=13567 from=192.168.4.5
May 17 01:21:27 client-18 rshd[13568]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:21:27 client-18 xinetd[3390]: EXIT: shell status=0 pid=13567 duration=0(sec)
May 17 01:21:27 client-18 xinetd[3390]: START: shell pid=13603 from=192.168.4.5
May 17 01:21:27 client-18 rshd[13604]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:21:28 client-18 xinetd[3390]: EXIT: shell status=0 pid=13603 duration=1(sec)
May 17 01:21:28 client-18 xinetd[3390]: START: shell pid=13640 from=192.168.4.5
May 17 01:21:28 client-18 rshd[13641]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 26f: rm -r of a directory which has recursive symlink ================================= 01:21:28 \(1305620488\)");echo XXRETCODE:$?'
May 17 01:21:28 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 26f: rm -r of a directory which has recursive symlink ================================= 01:21:28 (1305620488)
May 17 01:21:28 client-18 xinetd[3390]: EXIT: shell status=0 pid=13640 duration=0(sec)
May 17 01:21:28 client-18 xinetd[3390]: START: shell pid=13675 from=192.168.4.5
May 17 01:21:28 client-18 rshd[13676]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:21:28 client-18 xinetd[3390]: EXIT: shell status=0 pid=13675 duration=0(sec)
May 17 01:21:28 client-18 xinetd[3390]: START: shell pid=13711 from=192.168.4.5
May 17 01:21:28 client-18 rshd[13712]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:21:28 client-18 xinetd[3390]: EXIT: shell status=0 pid=13711 duration=0(sec)
May 17 01:21:28 client-18 xinetd[3390]: START: shell pid=13748 from=192.168.4.5
May 17 01:21:28 client-18 rshd[13749]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 27a: one stripe file ================================================================== 01:21:28 \(1305620488\)");echo XXRETCODE:$?'
May 17 01:21:28 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 27a: one stripe file ================================================================== 01:21:28 (1305620488)
May 17 01:21:28 client-18 xinetd[3390]: EXIT: shell status=0 pid=13748 duration=0(sec)
May 17 01:21:28 client-18 xinetd[3390]: START: shell pid=13783 from=192.168.4.5
May 17 01:21:28 client-18 rshd[13784]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == test_27a: write to one stripe file =========================");echo XXRETCODE:$?'
May 17 01:21:29 client-18 kernel: Lustre: DEBUG MARKER: == test_27a: write to one stripe file =========================
May 17 01:21:29 client-18 xinetd[3390]: EXIT: shell status=0 pid=13783 duration=1(sec)
May 17 01:21:29 client-18 xinetd[3390]: START: shell pid=13818 from=192.168.4.5
May 17 01:21:29 client-18 rshd[13819]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:21:29 client-18 xinetd[3390]: EXIT: shell status=0 pid=13818 duration=0(sec)
May 17 01:21:29 client-18 xinetd[3390]: START: shell pid=13854 from=192.168.4.5
May 17 01:21:29 client-18 rshd[13855]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:21:29 client-18 xinetd[3390]: EXIT: shell status=0 pid=13854 duration=0(sec)
May 17 01:21:29 client-18 xinetd[3390]: START: shell pid=13891 from=192.168.4.5
May 17 01:21:29 client-18 rshd[13892]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 27c: create two stripe file f01 ======================================================= 01:21:29 \(1305620489\)");echo XXRETCODE:$?'
May 17 01:21:29 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 27c: create two stripe file f01 ======================================================= 01:21:29 (1305620489)
May 17 01:21:29 client-18 xinetd[3390]: EXIT: shell status=0 pid=13891 duration=0(sec)
May 17 01:21:29 client-18 xinetd[3390]: START: shell pid=13926 from=192.168.4.5
May 17 01:21:29 client-18 rshd[13927]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == test_27c: write to two stripe file file f01 ================");echo XXRETCODE:$?'
May 17 01:21:29 client-18 kernel: Lustre: DEBUG MARKER: == test_27c: write to two stripe file file f01 ================
May 17 01:21:29 client-18 xinetd[3390]: EXIT: shell status=0 pid=13926 duration=0(sec)
May 17 01:21:29 client-18 xinetd[3390]: START: shell pid=13961 from=192.168.4.5
May 17 01:21:29 client-18 rshd[13962]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:21:29 client-18 xinetd[3390]: EXIT: shell status=0 pid=13961 duration=0(sec)
May 17 01:21:29 client-18 xinetd[3390]: START: shell pid=13997 from=192.168.4.5
May 17 01:21:29 client-18 rshd[13998]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:21:30 client-18 xinetd[3390]: EXIT: shell status=0 pid=13997 duration=1(sec)
May 17 01:21:30 client-18 xinetd[3390]: START: shell pid=14034 from=192.168.4.5
May 17 01:21:30 client-18 rshd[14035]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 27d: create file with default settings ================================================ 01:21:30 \(1305620490\)");echo XXRETCODE:$?'
May 17 01:21:30 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 27d: create file with default settings ================================================ 01:21:30 (1305620490)
May 17 01:21:30 client-18 xinetd[3390]: EXIT: shell status=0 pid=14034 duration=0(sec)
May 17 01:21:30 client-18 xinetd[3390]: START: shell pid=14069 from=192.168.4.5
May 17 01:21:30 client-18 rshd[14070]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:21:30 client-18 xinetd[3390]: EXIT: shell status=0 pid=14069 duration=0(sec)
May 17 01:21:30 client-18 xinetd[3390]: START: shell pid=14105 from=192.168.4.5
May 17 01:21:30 client-18 rshd[14106]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:21:30 client-18 xinetd[3390]: EXIT: shell status=0 pid=14105 duration=0(sec)
May 17 01:21:30 client-18 xinetd[3390]: START: shell pid=14142 from=192.168.4.5
May 17 01:21:30 client-18 rshd[14143]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 27e: setstripe existing file \(should return error\) ==================================== 01:21:30 \(1305620490\)");echo XXRETCODE:$?'
May 17 01:21:30 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 27e: setstripe existing file (should return error) ==================================== 01:21:30 (1305620490)
May 17 01:21:30 client-18 xinetd[3390]: EXIT: shell status=0 pid=14142 duration=0(sec)
May 17 01:21:30 client-18 xinetd[3390]: START: shell pid=14177 from=192.168.4.5
May 17 01:21:30 client-18 rshd[14178]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:21:30 client-18 xinetd[3390]: EXIT: shell status=0 pid=14177 duration=0(sec)
May 17 01:21:30 client-18 xinetd[3390]: START: shell pid=14213 from=192.168.4.5
May 17 01:21:30 client-18 rshd[14214]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:21:31 client-18 xinetd[3390]: EXIT: shell status=0 pid=14213 duration=1(sec)
May 17 01:21:31 client-18 xinetd[3390]: START: shell pid=14250 from=192.168.4.5
May 17 01:21:31 client-18 rshd[14251]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 27f: setstripe with bad stripe size \(should return error\) ============================= 01:21:31 \(1305620491\)");echo XXRETCODE:$?'
May 17 01:21:31 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 27f: setstripe with bad stripe size (should return error) ============================= 01:21:31 (1305620491)
May 17 01:21:31 client-18 xinetd[3390]: EXIT: shell status=0 pid=14250 duration=0(sec)
May 17 01:21:31 client-18 xinetd[3390]: START: shell pid=14285 from=192.168.4.5
May 17 01:21:31 client-18 rshd[14286]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:21:31 client-18 xinetd[3390]: EXIT: shell status=0 pid=14285 duration=0(sec)
May 17 01:21:31 client-18 xinetd[3390]: START: shell pid=14321 from=192.168.4.5
May 17 01:21:31 client-18 rshd[14322]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:21:31 client-18 xinetd[3390]: EXIT: shell status=0 pid=14321 duration=0(sec)
May 17 01:21:31 client-18 xinetd[3390]: START: shell pid=14358 from=192.168.4.5
May 17 01:21:31 client-18 rshd[14359]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 27g: test lfs getstripe =============================================================== 01:21:31 \(1305620491\)");echo XXRETCODE:$?'
May 17 01:21:31 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 27g: test lfs getstripe =============================================================== 01:21:31 (1305620491)
May 17 01:21:31 client-18 xinetd[3390]: EXIT: shell status=0 pid=14358 duration=0(sec)
May 17 01:21:31 client-18 xinetd[3390]: START: shell pid=14393 from=192.168.4.5
May 17 01:21:31 client-18 rshd[14394]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == test 27h: lfs getstripe with no objects ====================");echo XXRETCODE:$?'
May 17 01:21:31 client-18 kernel: Lustre: DEBUG MARKER: == test 27h: lfs getstripe with no objects ====================
May 17 01:21:31 client-18 xinetd[3390]: EXIT: shell status=0 pid=14393 duration=0(sec)
May 17 01:21:32 client-18 xinetd[3390]: START: shell pid=14428 from=192.168.4.5
May 17 01:21:32 client-18 rshd[14429]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == test 27i: lfs getstripe with some objects ==================");echo XXRETCODE:$?'
May 17 01:21:32 client-18 kernel: Lustre: DEBUG MARKER: == test 27i: lfs getstripe with some objects ==================
May 17 01:21:32 client-18 xinetd[3390]: EXIT: shell status=0 pid=14428 duration=0(sec)
May 17 01:21:32 client-18 xinetd[3390]: START: shell pid=14463 from=192.168.4.5
May 17 01:21:32 client-18 rshd[14464]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:21:32 client-18 xinetd[3390]: EXIT: shell status=0 pid=14463 duration=0(sec)
May 17 01:21:32 client-18 xinetd[3390]: START: shell pid=14499 from=192.168.4.5
May 17 01:21:32 client-18 rshd[14500]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:21:32 client-18 xinetd[3390]: EXIT: shell status=0 pid=14499 duration=0(sec)
May 17 01:21:32 client-18 xinetd[3390]: START: shell pid=14536 from=192.168.4.5
May 17 01:21:32 client-18 rshd[14537]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 27j: setstripe with bad stripe offset \(should return error\) =========================== 01:21:32 \(1305620492\)");echo XXRETCODE:$?'
May 17 01:21:32 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 27j: setstripe with bad stripe offset (should return error) =========================== 01:21:32 (1305620492)
May 17 01:21:32 client-18 xinetd[3390]: EXIT: shell status=0 pid=14536 duration=0(sec)
May 17 01:21:32 client-18 xinetd[3390]: START: shell pid=14571 from=192.168.4.5
May 17 01:21:32 client-18 rshd[14572]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:21:32 client-18 xinetd[3390]: EXIT: shell status=0 pid=14571 duration=0(sec)
May 17 01:21:32 client-18 xinetd[3390]: START: shell pid=14607 from=192.168.4.5
May 17 01:21:32 client-18 rshd[14608]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:21:32 client-18 xinetd[3390]: EXIT: shell status=0 pid=14607 duration=0(sec)
May 17 01:21:33 client-18 xinetd[3390]: START: shell pid=14644 from=192.168.4.5
May 17 01:21:33 client-18 rshd[14645]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 27k: limit i_blksize for broken user apps ============================================= 01:21:32 \(1305620492\)");echo XXRETCODE:$?'
May 17 01:21:33 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 27k: limit i_blksize for broken user apps ============================================= 01:21:32 (1305620492)
May 17 01:21:33 client-18 xinetd[3390]: EXIT: shell status=0 pid=14644 duration=0(sec)
May 17 01:21:33 client-18 xinetd[3390]: START: shell pid=14679 from=192.168.4.5
May 17 01:21:33 client-18 rshd[14680]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:21:33 client-18 xinetd[3390]: EXIT: shell status=0 pid=14679 duration=0(sec)
May 17 01:21:33 client-18 xinetd[3390]: START: shell pid=14715 from=192.168.4.5
May 17 01:21:33 client-18 rshd[14716]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:21:33 client-18 xinetd[3390]: EXIT: shell status=0 pid=14715 duration=0(sec)
May 17 01:21:33 client-18 xinetd[3390]: START: shell pid=14752 from=192.168.4.5
May 17 01:21:33 client-18 rshd[14753]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 27l: check setstripe permissions \(should return error\) ================================ 01:21:33 \(1305620493\)");echo XXRETCODE:$?'
May 17 01:21:33 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 27l: check setstripe permissions (should return error) ================================ 01:21:33 (1305620493)
May 17 01:21:33 client-18 xinetd[3390]: EXIT: shell status=0 pid=14752 duration=0(sec)
May 17 01:21:33 client-18 xinetd[3390]: START: shell pid=14787 from=192.168.4.5
May 17 01:21:33 client-18 rshd[14788]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:21:33 client-18 xinetd[3390]: EXIT: shell status=0 pid=14787 duration=0(sec)
May 17 01:21:33 client-18 xinetd[3390]: START: shell pid=14823 from=192.168.4.5
May 17 01:21:33 client-18 rshd[14824]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:21:33 client-18 xinetd[3390]: EXIT: shell status=0 pid=14823 duration=0(sec)
May 17 01:21:34 client-18 xinetd[3390]: START: shell pid=14860 from=192.168.4.5
May 17 01:21:34 client-18 rshd[14861]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 27m: create file while OST0 was full ================================================== 01:21:33 \(1305620493\)");echo XXRETCODE:$?'
May 17 01:21:34 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 27m: create file while OST0 was full ================================================== 01:21:33 (1305620493)
May 17 01:21:34 client-18 xinetd[3390]: EXIT: shell status=0 pid=14860 duration=0(sec)
May 17 01:21:34 client-18 xinetd[3390]: START: shell pid=14895 from=192.168.4.5
May 17 01:21:34 client-18 rshd[14896]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark  SKIP: sanity test_27m 53539848 \> 1200000 skipping out-of-space test on OST0");echo XXRETCODE:$?'
May 17 01:21:34 client-18 kernel: Lustre: DEBUG MARKER: SKIP: sanity test_27m 53539848 > 1200000 skipping out-of-space test on OST0
May 17 01:21:34 client-18 xinetd[3390]: EXIT: shell status=0 pid=14895 duration=0(sec)
May 17 01:21:34 client-18 xinetd[3390]: START: shell pid=14930 from=192.168.4.5
May 17 01:21:34 client-18 rshd[14931]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:21:34 client-18 xinetd[3390]: EXIT: shell status=0 pid=14930 duration=0(sec)
May 17 01:21:34 client-18 xinetd[3390]: START: shell pid=14966 from=192.168.4.5
May 17 01:21:34 client-18 rshd[14967]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:21:34 client-18 xinetd[3390]: EXIT: shell status=0 pid=14966 duration=0(sec)
May 17 01:21:34 client-18 xinetd[3390]: START: shell pid=15003 from=192.168.4.5
May 17 01:21:34 client-18 rshd[15004]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 27n: create file with some full OSTs ================================================== 01:21:34 \(1305620494\)");echo XXRETCODE:$?'
May 17 01:21:34 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 27n: create file with some full OSTs ================================================== 01:21:34 (1305620494)
May 17 01:21:34 client-18 xinetd[3390]: EXIT: shell status=0 pid=15003 duration=0(sec)
May 17 01:22:09 client-18 xinetd[3390]: START: shell pid=15038 from=192.168.4.5
May 17 01:22:09 client-18 rshd[15039]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:22:09 client-18 xinetd[3390]: EXIT: shell status=0 pid=15038 duration=0(sec)
May 17 01:22:09 client-18 xinetd[3390]: START: shell pid=15074 from=192.168.4.5
May 17 01:22:09 client-18 rshd[15075]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:22:09 client-18 xinetd[3390]: EXIT: shell status=0 pid=15074 duration=0(sec)
May 17 01:22:09 client-18 xinetd[3390]: START: shell pid=15111 from=192.168.4.5
May 17 01:22:09 client-18 rshd[15112]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 27o: create file with all full OSTs \(should error\) ==================================== 01:22:09 \(1305620529\)");echo XXRETCODE:$?'
May 17 01:22:09 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 27o: create file with all full OSTs (should error) ==================================== 01:22:09 (1305620529)
May 17 01:22:09 client-18 xinetd[3390]: EXIT: shell status=0 pid=15111 duration=0(sec)
May 17 01:23:54 client-18 xinetd[3390]: START: shell pid=15154 from=192.168.4.5
May 17 01:23:54 client-18 rshd[15155]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:23:54 client-18 xinetd[3390]: EXIT: shell status=0 pid=15154 duration=0(sec)
May 17 01:23:54 client-18 xinetd[3390]: START: shell pid=15190 from=192.168.4.5
May 17 01:23:54 client-18 rshd[15191]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:23:54 client-18 xinetd[3390]: EXIT: shell status=0 pid=15190 duration=0(sec)
May 17 01:23:54 client-18 xinetd[3390]: START: shell pid=15227 from=192.168.4.5
May 17 01:23:54 client-18 rshd[15228]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 27p: append to a truncated file with some full OSTs =================================== 01:23:54 \(1305620634\)");echo XXRETCODE:$?'
May 17 01:23:55 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 27p: append to a truncated file with some full OSTs =================================== 01:23:54 (1305620634)
May 17 01:23:55 client-18 xinetd[3390]: EXIT: shell status=0 pid=15227 duration=1(sec)
May 17 01:24:29 client-18 xinetd[3390]: START: shell pid=15262 from=192.168.4.5
May 17 01:24:29 client-18 rshd[15263]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:24:29 client-18 xinetd[3390]: EXIT: shell status=0 pid=15262 duration=0(sec)
May 17 01:24:29 client-18 xinetd[3390]: START: shell pid=15298 from=192.168.4.5
May 17 01:24:29 client-18 rshd[15299]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:24:29 client-18 xinetd[3390]: EXIT: shell status=0 pid=15298 duration=0(sec)
May 17 01:24:29 client-18 xinetd[3390]: START: shell pid=15335 from=192.168.4.5
May 17 01:24:29 client-18 rshd[15336]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 27q: append to truncated file with all OSTs full \(should error\) ======================= 01:24:29 \(1305620669\)");echo XXRETCODE:$?'
May 17 01:24:30 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 27q: append to truncated file with all OSTs full (should error) ======================= 01:24:29 (1305620669)
May 17 01:24:30 client-18 xinetd[3390]: EXIT: shell status=0 pid=15335 duration=1(sec)
May 17 01:26:14 client-18 xinetd[3390]: START: shell pid=15378 from=192.168.4.5
May 17 01:26:14 client-18 rshd[15379]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:26:14 client-18 xinetd[3390]: EXIT: shell status=0 pid=15378 duration=0(sec)
May 17 01:26:14 client-18 xinetd[3390]: START: shell pid=15414 from=192.168.4.5
May 17 01:26:14 client-18 rshd[15415]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:26:14 client-18 xinetd[3390]: EXIT: shell status=0 pid=15414 duration=0(sec)
May 17 01:26:15 client-18 xinetd[3390]: START: shell pid=15451 from=192.168.4.5
May 17 01:26:15 client-18 rshd[15452]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 27r: stripe file with some full OSTs \(shouldn\'t LBUG\) ================================= 01:26:14 \(1305620774\)");echo XXRETCODE:$?'
May 17 01:26:15 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 27r: stripe file with some full OSTs (shouldn't LBUG) ================================= 01:26:14 (1305620774)
May 17 01:26:15 client-18 xinetd[3390]: EXIT: shell status=0 pid=15451 duration=0(sec)
May 17 01:26:49 client-18 xinetd[3390]: START: shell pid=15486 from=192.168.4.5
May 17 01:26:49 client-18 rshd[15487]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:26:49 client-18 xinetd[3390]: EXIT: shell status=0 pid=15486 duration=0(sec)
May 17 01:26:49 client-18 xinetd[3390]: START: shell pid=15522 from=192.168.4.5
May 17 01:26:49 client-18 rshd[15523]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:26:49 client-18 xinetd[3390]: EXIT: shell status=0 pid=15522 duration=0(sec)
May 17 01:26:49 client-18 xinetd[3390]: START: shell pid=15559 from=192.168.4.5
May 17 01:26:50 client-18 rshd[15560]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 27s: lsm_xfersize overflow \(should error\) \(bug 10725\) ================================= 01:26:49 \(1305620809\)");echo XXRETCODE:$?'
May 17 01:26:50 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 27s: lsm_xfersize overflow (should error) (bug 10725) ================================= 01:26:49 (1305620809)
May 17 01:26:50 client-18 xinetd[3390]: EXIT: shell status=0 pid=15559 duration=1(sec)
May 17 01:26:50 client-18 xinetd[3390]: START: shell pid=15594 from=192.168.4.5
May 17 01:26:50 client-18 rshd[15595]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:26:50 client-18 xinetd[3390]: EXIT: shell status=0 pid=15594 duration=0(sec)
May 17 01:26:50 client-18 xinetd[3390]: START: shell pid=15630 from=192.168.4.5
May 17 01:26:50 client-18 rshd[15631]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:26:50 client-18 xinetd[3390]: EXIT: shell status=0 pid=15630 duration=0(sec)
May 17 01:26:50 client-18 xinetd[3390]: START: shell pid=15667 from=192.168.4.5
May 17 01:26:50 client-18 rshd[15668]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 27t: check that utils parse path correctly ============================================ 01:26:50 \(1305620810\)");echo XXRETCODE:$?'
May 17 01:26:50 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 27t: check that utils parse path correctly ============================================ 01:26:50 (1305620810)
May 17 01:26:50 client-18 xinetd[3390]: EXIT: shell status=0 pid=15667 duration=0(sec)
May 17 01:26:50 client-18 xinetd[3390]: START: shell pid=15702 from=192.168.4.5
May 17 01:26:50 client-18 rshd[15703]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:26:50 client-18 xinetd[3390]: EXIT: shell status=0 pid=15702 duration=0(sec)
May 17 01:26:50 client-18 xinetd[3390]: START: shell pid=15738 from=192.168.4.5
May 17 01:26:50 client-18 rshd[15739]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:26:50 client-18 xinetd[3390]: EXIT: shell status=0 pid=15738 duration=0(sec)
May 17 01:26:50 client-18 xinetd[3390]: START: shell pid=15775 from=192.168.4.5
May 17 01:26:51 client-18 rshd[15776]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark  SKIP: sanity test_27u skipping ALWAYS excluded test 27u");echo XXRETCODE:$?'
May 17 01:26:51 client-18 kernel: Lustre: DEBUG MARKER: SKIP: sanity test_27u skipping ALWAYS excluded test 27u
May 17 01:26:51 client-18 xinetd[3390]: EXIT: shell status=0 pid=15775 duration=1(sec)
May 17 01:26:51 client-18 xinetd[3390]: START: shell pid=15810 from=192.168.4.5
May 17 01:26:51 client-18 rshd[15811]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 27v: skip object creation on slow OST ================================================= 01:26:51 \(1305620811\)");echo XXRETCODE:$?'
May 17 01:26:51 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 27v: skip object creation on slow OST ================================================= 01:26:51 (1305620811)
May 17 01:26:51 client-18 xinetd[3390]: EXIT: shell status=0 pid=15810 duration=0(sec)
May 17 01:28:36 client-18 xinetd[3390]: START: shell pid=15853 from=192.168.4.5
May 17 01:28:36 client-18 rshd[15854]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:28:36 client-18 xinetd[3390]: EXIT: shell status=0 pid=15853 duration=0(sec)
May 17 01:28:36 client-18 xinetd[3390]: START: shell pid=15889 from=192.168.4.5
May 17 01:28:36 client-18 rshd[15890]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:28:36 client-18 xinetd[3390]: EXIT: shell status=0 pid=15889 duration=0(sec)
May 17 01:28:36 client-18 xinetd[3390]: START: shell pid=15926 from=192.168.4.5
May 17 01:28:36 client-18 rshd[15927]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 27w: check lfs setstripe -c -s -i options ============================================= 01:28:36 \(1305620916\)");echo XXRETCODE:$?'
May 17 01:28:36 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 27w: check lfs setstripe -c -s -i options ============================================= 01:28:36 (1305620916)
May 17 01:28:36 client-18 xinetd[3390]: EXIT: shell status=0 pid=15926 duration=0(sec)
May 17 01:28:37 client-18 xinetd[3390]: START: shell pid=15961 from=192.168.4.5
May 17 01:28:37 client-18 rshd[15962]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:28:37 client-18 xinetd[3390]: EXIT: shell status=0 pid=15961 duration=0(sec)
May 17 01:28:37 client-18 xinetd[3390]: START: shell pid=15997 from=192.168.4.5
May 17 01:28:37 client-18 rshd[15998]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:28:37 client-18 xinetd[3390]: EXIT: shell status=0 pid=15997 duration=0(sec)
May 17 01:28:37 client-18 xinetd[3390]: START: shell pid=16034 from=192.168.4.5
May 17 01:28:37 client-18 rshd[16035]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 27x: create files while OST0 is degraded ============================================== 01:28:37 \(1305620917\)");echo XXRETCODE:$?'
May 17 01:28:37 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 27x: create files while OST0 is degraded ============================================== 01:28:37 (1305620917)
May 17 01:28:37 client-18 xinetd[3390]: EXIT: shell status=0 pid=16034 duration=0(sec)
May 17 01:28:47 client-18 xinetd[3390]: START: shell pid=16069 from=192.168.4.5
May 17 01:28:47 client-18 rshd[16070]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:28:48 client-18 xinetd[3390]: EXIT: shell status=0 pid=16069 duration=1(sec)
May 17 01:28:48 client-18 xinetd[3390]: START: shell pid=16105 from=192.168.4.5
May 17 01:28:48 client-18 rshd[16106]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:28:48 client-18 xinetd[3390]: EXIT: shell status=0 pid=16105 duration=0(sec)
May 17 01:28:48 client-18 xinetd[3390]: START: shell pid=16142 from=192.168.4.5
May 17 01:28:48 client-18 rshd[16143]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 27y: create files while OST0 is degraded and the rest inactive ======================== 01:28:48 \(1305620928\)");echo XXRETCODE:$?'
May 17 01:28:48 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 27y: create files while OST0 is degraded and the rest inactive ======================== 01:28:48 (1305620928)
May 17 01:28:48 client-18 xinetd[3390]: EXIT: shell status=0 pid=16142 duration=0(sec)
May 17 01:30:00 client-18 xinetd[3390]: START: shell pid=16181 from=192.168.4.5
May 17 01:30:00 client-18 rshd[16182]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:30:00 client-18 xinetd[3390]: EXIT: shell status=0 pid=16181 duration=0(sec)
May 17 01:30:00 client-18 xinetd[3390]: START: shell pid=16217 from=192.168.4.5
May 17 01:30:00 client-18 rshd[16218]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:30:00 client-18 xinetd[3390]: EXIT: shell status=0 pid=16217 duration=0(sec)
May 17 01:30:00 client-18 xinetd[3390]: START: shell pid=16254 from=192.168.4.5
May 17 01:30:00 client-18 rshd[16255]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 27z: check SEQ\/OID on the MDT and OST filesystems ===================================== 01:30:00 \(1305621000\)");echo XXRETCODE:$?'
May 17 01:30:00 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 27z: check SEQ/OID on the MDT and OST filesystems ===================================== 01:30:00 (1305621000)
May 17 01:30:00 client-18 xinetd[3390]: EXIT: shell status=0 pid=16254 duration=0(sec)
May 17 01:30:13 client-18 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.129@o2ib. The obd_ping operation failed with -107
May 17 01:30:13 client-18 kernel: Lustre: lustre-OST0001-osc-ffff81031ffac400: Connection to service lustre-OST0001 via nid 192.168.4.129@o2ib was lost; in progress operations using this service will wait for recovery to complete.
May 17 01:30:13 client-18 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.129@o2ib. The ost_connect operation failed with -19
May 17 01:30:18 client-18 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.129@o2ib. The obd_ping operation failed with -107
May 17 01:30:18 client-18 kernel: Lustre: lustre-OST0000-osc-ffff81031ffac400: Connection to service lustre-OST0000 via nid 192.168.4.129@o2ib was lost; in progress operations using this service will wait for recovery to complete.
May 17 01:30:19 client-18 kernel: Lustre: lustre-OST0005-osc-ffff81031ffac400: Connection restored to service lustre-OST0005 using nid 192.168.4.129@o2ib.
May 17 01:30:23 client-18 kernel: Lustre: 3861:0:(import.c:529:import_select_connection()) lustre-OST0001-osc-ffff81031ffac400: tried all connections, increasing latency to 6s
May 17 01:30:23 client-18 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.129@o2ib. The obd_ping operation failed with -107
May 17 01:30:23 client-18 kernel: LustreError: Skipped 2 previous similar messages
May 17 01:30:23 client-18 kernel: Lustre: lustre-OST0002-osc-ffff81031ffac400: Connection to service lustre-OST0002 via nid 192.168.4.129@o2ib was lost; in progress operations using this service will wait for recovery to complete.
May 17 01:30:23 client-18 kernel: Lustre: Skipped 1 previous similar message
May 17 01:30:28 client-18 kernel: Lustre: 3861:0:(import.c:529:import_select_connection()) lustre-OST0000-osc-ffff81031ffac400: tried all connections, increasing latency to 6s
May 17 01:30:28 client-18 kernel: Lustre: lustre-OST0000-osc-ffff81031ffac400: Connection restored to service lustre-OST0000 using nid 192.168.4.129@o2ib.
May 17 01:30:33 client-18 kernel: Lustre: 3861:0:(import.c:529:import_select_connection()) lustre-OST0002-osc-ffff81031ffac400: tried all connections, increasing latency to 6s
May 17 01:30:33 client-18 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.129@o2ib. The obd_ping operation failed with -107
May 17 01:30:33 client-18 kernel: LustreError: Skipped 1 previous similar message
May 17 01:30:33 client-18 kernel: Lustre: lustre-OST0003-osc-ffff81031ffac400: Connection to service lustre-OST0003 via nid 192.168.4.129@o2ib was lost; in progress operations using this service will wait for recovery to complete.
May 17 01:30:34 client-18 kernel: Lustre: lustre-OST0002-osc-ffff81031ffac400: Connection restored to service lustre-OST0002 using nid 192.168.4.129@o2ib.
May 17 01:30:34 client-18 xinetd[3390]: START: shell pid=16289 from=192.168.4.5
May 17 01:30:34 client-18 rshd[16290]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:30:34 client-18 xinetd[3390]: EXIT: shell status=0 pid=16289 duration=0(sec)
May 17 01:30:34 client-18 xinetd[3390]: START: shell pid=16325 from=192.168.4.5
May 17 01:30:34 client-18 rshd[16326]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:30:34 client-18 xinetd[3390]: EXIT: shell status=0 pid=16325 duration=0(sec)
May 17 01:30:34 client-18 xinetd[3390]: START: shell pid=16362 from=192.168.4.5
May 17 01:30:34 client-18 rshd[16363]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 27A: check filesystem-wide default LOV EA values ====================================== 01:30:34 \(1305621034\)");echo XXRETCODE:$?'
May 17 01:30:34 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 27A: check filesystem-wide default LOV EA values ====================================== 01:30:34 (1305621034)
May 17 01:30:34 client-18 xinetd[3390]: EXIT: shell status=0 pid=16362 duration=0(sec)
May 17 01:30:34 client-18 xinetd[3390]: START: shell pid=16399 from=192.168.4.5
May 17 01:30:34 client-18 rshd[16400]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:30:34 client-18 xinetd[3390]: EXIT: shell status=0 pid=16399 duration=0(sec)
May 17 01:30:35 client-18 xinetd[3390]: START: shell pid=16435 from=192.168.4.5
May 17 01:30:35 client-18 rshd[16436]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:30:35 client-18 xinetd[3390]: EXIT: shell status=0 pid=16435 duration=0(sec)
May 17 01:30:35 client-18 xinetd[3390]: START: shell pid=16472 from=192.168.4.5
May 17 01:30:35 client-18 rshd[16473]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 28: create\/mknod\/mkdir with bad file types ============================================ 01:30:35 \(1305621035\)");echo XXRETCODE:$?'
May 17 01:30:35 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 28: create/mknod/mkdir with bad file types ============================================ 01:30:35 (1305621035)
May 17 01:30:35 client-18 xinetd[3390]: EXIT: shell status=0 pid=16472 duration=0(sec)
May 17 01:30:35 client-18 xinetd[3390]: START: shell pid=16507 from=192.168.4.5
May 17 01:30:35 client-18 rshd[16508]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:30:35 client-18 xinetd[3390]: EXIT: shell status=0 pid=16507 duration=0(sec)
May 17 01:30:35 client-18 xinetd[3390]: START: shell pid=16543 from=192.168.4.5
May 17 01:30:35 client-18 rshd[16544]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:30:35 client-18 xinetd[3390]: EXIT: shell status=0 pid=16543 duration=0(sec)
May 17 01:30:35 client-18 xinetd[3390]: START: shell pid=16580 from=192.168.4.5
May 17 01:30:35 client-18 rshd[16581]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 29: IT_GETATTR regression  ============================================================ 01:30:35 \(1305621035\)");echo XXRETCODE:$?'
May 17 01:30:35 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 29: IT_GETATTR regression ============================================================ 01:30:35 (1305621035)
May 17 01:30:35 client-18 xinetd[3390]: EXIT: shell status=0 pid=16580 duration=0(sec)
May 17 01:30:36 client-18 xinetd[3390]: START: shell pid=16615 from=192.168.4.5
May 17 01:30:36 client-18 rshd[16616]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark first d29");echo XXRETCODE:$?'
May 17 01:30:36 client-18 kernel: Lustre: DEBUG MARKER: first d29
May 17 01:30:36 client-18 xinetd[3390]: EXIT: shell status=0 pid=16615 duration=0(sec)
May 17 01:30:36 client-18 xinetd[3390]: START: shell pid=16650 from=192.168.4.5
May 17 01:30:36 client-18 rshd[16651]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark second d29");echo XXRETCODE:$?'
May 17 01:30:36 client-18 kernel: Lustre: DEBUG MARKER: second d29
May 17 01:30:36 client-18 xinetd[3390]: EXIT: shell status=0 pid=16650 duration=0(sec)
May 17 01:30:36 client-18 xinetd[3390]: START: shell pid=16685 from=192.168.4.5
May 17 01:30:36 client-18 rshd[16686]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark done");echo XXRETCODE:$?'
May 17 01:30:36 client-18 kernel: Lustre: DEBUG MARKER: done
May 17 01:30:36 client-18 xinetd[3390]: EXIT: shell status=0 pid=16685 duration=0(sec)
May 17 01:30:36 client-18 xinetd[3390]: START: shell pid=16720 from=192.168.4.5
May 17 01:30:36 client-18 rshd[16721]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:30:36 client-18 xinetd[3390]: EXIT: shell status=0 pid=16720 duration=0(sec)
May 17 01:30:36 client-18 xinetd[3390]: START: shell pid=16756 from=192.168.4.5
May 17 01:30:36 client-18 rshd[16757]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:30:36 client-18 xinetd[3390]: EXIT: shell status=0 pid=16756 duration=0(sec)
May 17 01:30:36 client-18 xinetd[3390]: START: shell pid=16793 from=192.168.4.5
May 17 01:30:36 client-18 rshd[16794]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 30a: execute binary from Lustre \(execve\) ============================================== 01:30:36 \(1305621036\)");echo XXRETCODE:$?'
May 17 01:30:37 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 30a: execute binary from Lustre (execve) ============================================== 01:30:36 (1305621036)
May 17 01:30:37 client-18 xinetd[3390]: EXIT: shell status=0 pid=16793 duration=1(sec)
May 17 01:30:37 client-18 xinetd[3390]: START: shell pid=16828 from=192.168.4.5
May 17 01:30:37 client-18 rshd[16829]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:30:37 client-18 xinetd[3390]: EXIT: shell status=0 pid=16828 duration=0(sec)
May 17 01:30:37 client-18 xinetd[3390]: START: shell pid=16864 from=192.168.4.5
May 17 01:30:37 client-18 rshd[16865]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:30:37 client-18 xinetd[3390]: EXIT: shell status=0 pid=16864 duration=0(sec)
May 17 01:30:37 client-18 xinetd[3390]: START: shell pid=16901 from=192.168.4.5
May 17 01:30:37 client-18 rshd[16902]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 30b: execute binary from Lustre as non-root =========================================== 01:30:37 \(1305621037\)");echo XXRETCODE:$?'
May 17 01:30:37 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 30b: execute binary from Lustre as non-root =========================================== 01:30:37 (1305621037)
May 17 01:30:37 client-18 xinetd[3390]: EXIT: shell status=0 pid=16901 duration=0(sec)
May 17 01:30:37 client-18 xinetd[3390]: START: shell pid=16936 from=192.168.4.5
May 17 01:30:37 client-18 rshd[16937]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:30:37 client-18 xinetd[3390]: EXIT: shell status=0 pid=16936 duration=0(sec)
May 17 01:30:37 client-18 xinetd[3390]: START: shell pid=16972 from=192.168.4.5
May 17 01:30:37 client-18 rshd[16973]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:30:37 client-18 xinetd[3390]: EXIT: shell status=0 pid=16972 duration=0(sec)
May 17 01:30:38 client-18 xinetd[3390]: START: shell pid=17009 from=192.168.4.5
May 17 01:30:38 client-18 rshd[17010]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 30c: execute binary from Lustre without read perms ==================================== 01:30:37 \(1305621037\)");echo XXRETCODE:$?'
May 17 01:30:38 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 30c: execute binary from Lustre without read perms ==================================== 01:30:37 (1305621037)
May 17 01:30:38 client-18 xinetd[3390]: EXIT: shell status=0 pid=17009 duration=0(sec)
May 17 01:30:38 client-18 xinetd[3390]: START: shell pid=17044 from=192.168.4.5
May 17 01:30:38 client-18 rshd[17045]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:30:38 client-18 xinetd[3390]: EXIT: shell status=0 pid=17044 duration=0(sec)
May 17 01:30:38 client-18 xinetd[3390]: START: shell pid=17080 from=192.168.4.5
May 17 01:30:38 client-18 rshd[17081]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:30:39 client-18 xinetd[3390]: EXIT: shell status=0 pid=17080 duration=1(sec)
May 17 01:30:39 client-18 xinetd[3390]: START: shell pid=17117 from=192.168.4.5
May 17 01:30:39 client-18 rshd[17118]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 31a: open-unlink file ================================================================= 01:30:39 \(1305621039\)");echo XXRETCODE:$?'
May 17 01:30:39 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 31a: open-unlink file ================================================================= 01:30:39 (1305621039)
May 17 01:30:39 client-18 xinetd[3390]: EXIT: shell status=0 pid=17117 duration=0(sec)
May 17 01:30:39 client-18 xinetd[3390]: START: shell pid=17152 from=192.168.4.5
May 17 01:30:39 client-18 kernel: Lustre: lustre-OST0003-osc-ffff81031ffac400: Connection restored to service lustre-OST0003 using nid 192.168.4.129@o2ib.
May 17 01:30:39 client-18 rshd[17153]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:30:39 client-18 xinetd[3390]: EXIT: shell status=0 pid=17152 duration=0(sec)
May 17 01:30:39 client-18 xinetd[3390]: START: shell pid=17188 from=192.168.4.5
May 17 01:30:39 client-18 rshd[17189]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:30:39 client-18 xinetd[3390]: EXIT: shell status=0 pid=17188 duration=0(sec)
May 17 01:30:39 client-18 xinetd[3390]: START: shell pid=17225 from=192.168.4.5
May 17 01:30:39 client-18 rshd[17226]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 31b: unlink file with multiple links while open ======================================= 01:30:39 \(1305621039\)");echo XXRETCODE:$?'
May 17 01:30:39 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 31b: unlink file with multiple links while open ======================================= 01:30:39 (1305621039)
May 17 01:30:39 client-18 xinetd[3390]: EXIT: shell status=0 pid=17225 duration=0(sec)
May 17 01:30:39 client-18 xinetd[3390]: START: shell pid=17260 from=192.168.4.5
May 17 01:30:39 client-18 rshd[17261]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:30:40 client-18 xinetd[3390]: EXIT: shell status=0 pid=17260 duration=1(sec)
May 17 01:30:40 client-18 xinetd[3390]: START: shell pid=17296 from=192.168.4.5
May 17 01:30:40 client-18 rshd[17297]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:30:40 client-18 xinetd[3390]: EXIT: shell status=0 pid=17296 duration=0(sec)
May 17 01:30:40 client-18 xinetd[3390]: START: shell pid=17333 from=192.168.4.5
May 17 01:30:40 client-18 rshd[17334]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 31c: open-unlink file with multiple links ============================================= 01:30:40 \(1305621040\)");echo XXRETCODE:$?'
May 17 01:30:40 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 31c: open-unlink file with multiple links ============================================= 01:30:40 (1305621040)
May 17 01:30:40 client-18 xinetd[3390]: EXIT: shell status=0 pid=17333 duration=0(sec)
May 17 01:30:40 client-18 xinetd[3390]: START: shell pid=17370 from=192.168.4.5
May 17 01:30:40 client-18 rshd[17371]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:30:40 client-18 xinetd[3390]: EXIT: shell status=0 pid=17370 duration=0(sec)
May 17 01:30:40 client-18 xinetd[3390]: START: shell pid=17406 from=192.168.4.5
May 17 01:30:40 client-18 rshd[17407]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:30:40 client-18 xinetd[3390]: EXIT: shell status=0 pid=17406 duration=0(sec)
May 17 01:30:40 client-18 xinetd[3390]: START: shell pid=17443 from=192.168.4.5
May 17 01:30:40 client-18 rshd[17444]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 31d: remove of open directory ========================================================= 01:30:40 \(1305621040\)");echo XXRETCODE:$?'
May 17 01:30:40 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 31d: remove of open directory ========================================================= 01:30:40 (1305621040)
May 17 01:30:40 client-18 xinetd[3390]: EXIT: shell status=0 pid=17443 duration=0(sec)
May 17 01:30:40 client-18 xinetd[3390]: START: shell pid=17478 from=192.168.4.5
May 17 01:30:40 client-18 rshd[17479]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:30:41 client-18 xinetd[3390]: EXIT: shell status=0 pid=17478 duration=1(sec)
May 17 01:30:41 client-18 xinetd[3390]: START: shell pid=17514 from=192.168.4.5
May 17 01:30:41 client-18 rshd[17515]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:30:41 client-18 xinetd[3390]: EXIT: shell status=0 pid=17514 duration=0(sec)
May 17 01:30:41 client-18 xinetd[3390]: START: shell pid=17551 from=192.168.4.5
May 17 01:30:41 client-18 rshd[17552]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 31e: remove of open non-empty directory =============================================== 01:30:41 \(1305621041\)");echo XXRETCODE:$?'
May 17 01:30:41 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 31e: remove of open non-empty directory =============================================== 01:30:41 (1305621041)
May 17 01:30:41 client-18 xinetd[3390]: EXIT: shell status=0 pid=17551 duration=0(sec)
May 17 01:30:41 client-18 xinetd[3390]: START: shell pid=17586 from=192.168.4.5
May 17 01:30:41 client-18 rshd[17587]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:30:41 client-18 xinetd[3390]: EXIT: shell status=0 pid=17586 duration=0(sec)
May 17 01:30:41 client-18 xinetd[3390]: START: shell pid=17622 from=192.168.4.5
May 17 01:30:41 client-18 rshd[17623]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:30:41 client-18 xinetd[3390]: EXIT: shell status=0 pid=17622 duration=0(sec)
May 17 01:30:41 client-18 xinetd[3390]: START: shell pid=17659 from=192.168.4.5
May 17 01:30:41 client-18 rshd[17660]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 31f: remove of open directory with open-unlink file =================================== 01:30:41 \(1305621041\)");echo XXRETCODE:$?'
May 17 01:30:41 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 31f: remove of open directory with open-unlink file =================================== 01:30:41 (1305621041)
May 17 01:30:41 client-18 xinetd[3390]: EXIT: shell status=0 pid=17659 duration=0(sec)
May 17 01:30:43 client-18 kernel: Lustre: 3861:0:(import.c:529:import_select_connection()) lustre-OST0004-osc-ffff81031ffac400: tried all connections, increasing latency to 6s
May 17 01:30:43 client-18 kernel: Lustre: lustre-OST0004-osc-ffff81031ffac400: Connection restored to service lustre-OST0004 using nid 192.168.4.129@o2ib.
May 17 01:30:48 client-18 xinetd[3390]: START: shell pid=17694 from=192.168.4.5
May 17 01:30:48 client-18 rshd[17695]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:30:48 client-18 xinetd[3390]: EXIT: shell status=0 pid=17694 duration=0(sec)
May 17 01:30:48 client-18 xinetd[3390]: START: shell pid=17730 from=192.168.4.5
May 17 01:30:48 client-18 rshd[17731]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:30:48 client-18 xinetd[3390]: EXIT: shell status=0 pid=17730 duration=0(sec)
May 17 01:30:48 client-18 xinetd[3390]: START: shell pid=17767 from=192.168.4.5
May 17 01:30:48 client-18 rshd[17768]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 31g: cross directory link============================================================== 01:30:48 \(1305621048\)");echo XXRETCODE:$?'
May 17 01:30:48 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 31g: cross directory link============================================================== 01:30:48 (1305621048)
May 17 01:30:48 client-18 xinetd[3390]: EXIT: shell status=0 pid=17767 duration=0(sec)
May 17 01:30:48 client-18 xinetd[3390]: START: shell pid=17802 from=192.168.4.5
May 17 01:30:48 client-18 rshd[17803]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:30:48 client-18 xinetd[3390]: EXIT: shell status=0 pid=17802 duration=0(sec)
May 17 01:30:48 client-18 xinetd[3390]: START: shell pid=17838 from=192.168.4.5
May 17 01:30:48 client-18 rshd[17839]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:30:48 client-18 xinetd[3390]: EXIT: shell status=0 pid=17838 duration=0(sec)
May 17 01:30:48 client-18 xinetd[3390]: START: shell pid=17875 from=192.168.4.5
May 17 01:30:48 client-18 rshd[17876]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 31h: cross directory link under child================================================== 01:30:48 \(1305621048\)");echo XXRETCODE:$?'
May 17 01:30:49 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 31h: cross directory link under child================================================== 01:30:48 (1305621048)
May 17 01:30:49 client-18 xinetd[3390]: EXIT: shell status=0 pid=17875 duration=1(sec)
May 17 01:30:49 client-18 xinetd[3390]: START: shell pid=17910 from=192.168.4.5
May 17 01:30:49 client-18 rshd[17911]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:30:49 client-18 xinetd[3390]: EXIT: shell status=0 pid=17910 duration=0(sec)
May 17 01:30:49 client-18 xinetd[3390]: START: shell pid=17946 from=192.168.4.5
May 17 01:30:49 client-18 rshd[17947]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:30:49 client-18 xinetd[3390]: EXIT: shell status=0 pid=17946 duration=0(sec)
May 17 01:30:49 client-18 xinetd[3390]: START: shell pid=17983 from=192.168.4.5
May 17 01:30:49 client-18 rshd[17984]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 31i: cross directory link under parent================================================= 01:30:49 \(1305621049\)");echo XXRETCODE:$?'
May 17 01:30:49 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 31i: cross directory link under parent================================================= 01:30:49 (1305621049)
May 17 01:30:49 client-18 xinetd[3390]: EXIT: shell status=0 pid=17983 duration=0(sec)
May 17 01:30:49 client-18 xinetd[3390]: START: shell pid=18018 from=192.168.4.5
May 17 01:30:49 client-18 rshd[18019]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:30:49 client-18 xinetd[3390]: EXIT: shell status=0 pid=18018 duration=0(sec)
May 17 01:30:49 client-18 xinetd[3390]: START: shell pid=18054 from=192.168.4.5
May 17 01:30:49 client-18 rshd[18055]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:30:49 client-18 xinetd[3390]: EXIT: shell status=0 pid=18054 duration=0(sec)
May 17 01:30:50 client-18 xinetd[3390]: START: shell pid=18091 from=192.168.4.5
May 17 01:30:50 client-18 rshd[18092]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 31j: link for directory================================================================ 01:30:49 \(1305621049\)");echo XXRETCODE:$?'
May 17 01:30:50 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 31j: link for directory================================================================ 01:30:49 (1305621049)
May 17 01:30:50 client-18 xinetd[3390]: EXIT: shell status=0 pid=18091 duration=0(sec)
May 17 01:30:50 client-18 xinetd[3390]: START: shell pid=18126 from=192.168.4.5
May 17 01:30:50 client-18 rshd[18127]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:30:50 client-18 xinetd[3390]: EXIT: shell status=0 pid=18126 duration=0(sec)
May 17 01:30:50 client-18 xinetd[3390]: START: shell pid=18162 from=192.168.4.5
May 17 01:30:50 client-18 rshd[18163]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:30:50 client-18 xinetd[3390]: EXIT: shell status=0 pid=18162 duration=0(sec)
May 17 01:30:50 client-18 xinetd[3390]: START: shell pid=18199 from=192.168.4.5
May 17 01:30:50 client-18 rshd[18200]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 31k: link to file: the same, non-existing, dir========================================= 01:30:50 \(1305621050\)");echo XXRETCODE:$?'
May 17 01:30:50 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 31k: link to file: the same, non-existing, dir========================================= 01:30:50 (1305621050)
May 17 01:30:50 client-18 xinetd[3390]: EXIT: shell status=0 pid=18199 duration=0(sec)
May 17 01:30:50 client-18 xinetd[3390]: START: shell pid=18234 from=192.168.4.5
May 17 01:30:50 client-18 rshd[18235]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:30:50 client-18 xinetd[3390]: EXIT: shell status=0 pid=18234 duration=0(sec)
May 17 01:30:50 client-18 xinetd[3390]: START: shell pid=18270 from=192.168.4.5
May 17 01:30:50 client-18 rshd[18271]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:30:51 client-18 xinetd[3390]: EXIT: shell status=0 pid=18270 duration=1(sec)
May 17 01:30:51 client-18 xinetd[3390]: START: shell pid=18307 from=192.168.4.5
May 17 01:30:51 client-18 rshd[18308]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 31m: link to file: the same, non-existing, dir========================================= 01:30:51 \(1305621051\)");echo XXRETCODE:$?'
May 17 01:30:51 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 31m: link to file: the same, non-existing, dir========================================= 01:30:51 (1305621051)
May 17 01:30:51 client-18 xinetd[3390]: EXIT: shell status=0 pid=18307 duration=0(sec)
May 17 01:30:51 client-18 xinetd[3390]: START: shell pid=18342 from=192.168.4.5
May 17 01:30:51 client-18 rshd[18343]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:30:51 client-18 xinetd[3390]: EXIT: shell status=0 pid=18342 duration=0(sec)
May 17 01:30:51 client-18 xinetd[3390]: START: shell pid=18378 from=192.168.4.5
May 17 01:30:51 client-18 rshd[18379]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:30:51 client-18 xinetd[3390]: EXIT: shell status=0 pid=18378 duration=0(sec)
May 17 01:30:51 client-18 xinetd[3390]: START: shell pid=18415 from=192.168.4.5
May 17 01:30:51 client-18 rshd[18416]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 32a: stat d32a\/ext2-mountpoint\/.. ===================================================== 01:30:51 \(1305621051\)");echo XXRETCODE:$?'
May 17 01:30:51 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 32a: stat d32a/ext2-mountpoint/.. ===================================================== 01:30:51 (1305621051)
May 17 01:30:51 client-18 xinetd[3390]: EXIT: shell status=0 pid=18415 duration=0(sec)
May 17 01:30:51 client-18 xinetd[3390]: START: shell pid=18450 from=192.168.4.5
May 17 01:30:51 client-18 rshd[18451]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:30:51 client-18 xinetd[3390]: EXIT: shell status=0 pid=18450 duration=0(sec)
May 17 01:30:52 client-18 xinetd[3390]: START: shell pid=18486 from=192.168.4.5
May 17 01:30:52 client-18 rshd[18487]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:30:52 client-18 xinetd[3390]: EXIT: shell status=0 pid=18486 duration=0(sec)
May 17 01:30:52 client-18 xinetd[3390]: START: shell pid=18523 from=192.168.4.5
May 17 01:30:52 client-18 rshd[18524]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 32b: open d32b\/ext2-mountpoint\/.. ===================================================== 01:30:52 \(1305621052\)");echo XXRETCODE:$?'
May 17 01:30:52 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 32b: open d32b/ext2-mountpoint/.. ===================================================== 01:30:52 (1305621052)
May 17 01:30:52 client-18 xinetd[3390]: EXIT: shell status=0 pid=18523 duration=0(sec)
May 17 01:30:52 client-18 xinetd[3390]: START: shell pid=18558 from=192.168.4.5
May 17 01:30:52 client-18 rshd[18559]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:30:52 client-18 xinetd[3390]: EXIT: shell status=0 pid=18558 duration=0(sec)
May 17 01:30:52 client-18 xinetd[3390]: START: shell pid=18594 from=192.168.4.5
May 17 01:30:52 client-18 rshd[18595]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:30:52 client-18 xinetd[3390]: EXIT: shell status=0 pid=18594 duration=0(sec)
May 17 01:30:52 client-18 xinetd[3390]: START: shell pid=18631 from=192.168.4.5
May 17 01:30:52 client-18 rshd[18632]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 32c: stat d32c\/ext2-mountpoint\/..\/d2\/test_dir ========================================= 01:30:52 \(1305621052\)");echo XXRETCODE:$?'
May 17 01:30:52 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 32c: stat d32c/ext2-mountpoint/../d2/test_dir ========================================= 01:30:52 (1305621052)
May 17 01:30:52 client-18 xinetd[3390]: EXIT: shell status=0 pid=18631 duration=0(sec)
May 17 01:30:52 client-18 xinetd[3390]: START: shell pid=18666 from=192.168.4.5
May 17 01:30:53 client-18 rshd[18667]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:30:53 client-18 xinetd[3390]: EXIT: shell status=0 pid=18666 duration=1(sec)
May 17 01:30:53 client-18 xinetd[3390]: START: shell pid=18702 from=192.168.4.5
May 17 01:30:53 client-18 rshd[18703]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:30:53 client-18 xinetd[3390]: EXIT: shell status=0 pid=18702 duration=0(sec)
May 17 01:30:53 client-18 xinetd[3390]: START: shell pid=18739 from=192.168.4.5
May 17 01:30:53 client-18 rshd[18740]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 32d: open d32d\/ext2-mountpoint\/..\/d2\/test_dir ========================================= 01:30:53 \(1305621053\)");echo XXRETCODE:$?'
May 17 01:30:53 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 32d: open d32d/ext2-mountpoint/../d2/test_dir ========================================= 01:30:53 (1305621053)
May 17 01:30:53 client-18 xinetd[3390]: EXIT: shell status=0 pid=18739 duration=0(sec)
May 17 01:30:53 client-18 xinetd[3390]: START: shell pid=18774 from=192.168.4.5
May 17 01:30:53 client-18 rshd[18775]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:30:53 client-18 xinetd[3390]: EXIT: shell status=0 pid=18774 duration=0(sec)
May 17 01:30:53 client-18 xinetd[3390]: START: shell pid=18810 from=192.168.4.5
May 17 01:30:53 client-18 rshd[18811]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:30:53 client-18 xinetd[3390]: EXIT: shell status=0 pid=18810 duration=0(sec)
May 17 01:30:53 client-18 xinetd[3390]: START: shell pid=18847 from=192.168.4.5
May 17 01:30:53 client-18 rshd[18848]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 32e: stat d32e\/symlink-\>tmp\/symlink-\>lustre-subdir ==================================== 01:30:53 \(1305621053\)");echo XXRETCODE:$?'
May 17 01:30:53 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 32e: stat d32e/symlink->tmp/symlink->lustre-subdir ==================================== 01:30:53 (1305621053)
May 17 01:30:53 client-18 xinetd[3390]: EXIT: shell status=0 pid=18847 duration=0(sec)
May 17 01:30:54 client-18 xinetd[3390]: START: shell pid=18882 from=192.168.4.5
May 17 01:30:54 client-18 rshd[18883]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:30:54 client-18 xinetd[3390]: EXIT: shell status=0 pid=18882 duration=0(sec)
May 17 01:30:54 client-18 xinetd[3390]: START: shell pid=18918 from=192.168.4.5
May 17 01:30:54 client-18 rshd[18919]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:30:54 client-18 xinetd[3390]: EXIT: shell status=0 pid=18918 duration=0(sec)
May 17 01:30:54 client-18 xinetd[3390]: START: shell pid=18955 from=192.168.4.5
May 17 01:30:54 client-18 rshd[18956]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 32f: open d32f\/symlink-\>tmp\/symlink-\>lustre-subdir ==================================== 01:30:54 \(1305621054\)");echo XXRETCODE:$?'
May 17 01:30:54 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 32f: open d32f/symlink->tmp/symlink->lustre-subdir ==================================== 01:30:54 (1305621054)
May 17 01:30:54 client-18 xinetd[3390]: EXIT: shell status=0 pid=18955 duration=0(sec)
May 17 01:30:54 client-18 xinetd[3390]: START: shell pid=18990 from=192.168.4.5
May 17 01:30:54 client-18 rshd[18991]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:30:54 client-18 xinetd[3390]: EXIT: shell status=0 pid=18990 duration=0(sec)
May 17 01:30:54 client-18 xinetd[3390]: START: shell pid=19026 from=192.168.4.5
May 17 01:30:54 client-18 rshd[19027]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:30:54 client-18 xinetd[3390]: EXIT: shell status=0 pid=19026 duration=0(sec)
May 17 01:30:54 client-18 xinetd[3390]: START: shell pid=19063 from=192.168.4.5
May 17 01:30:54 client-18 rshd[19064]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 32g: stat d32g\/symlink-\>tmp\/symlink-\>lustre-subdir\/2 ================================== 01:30:54 \(1305621054\)");echo XXRETCODE:$?'
May 17 01:30:55 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 32g: stat d32g/symlink->tmp/symlink->lustre-subdir/2 ================================== 01:30:54 (1305621054)
May 17 01:30:55 client-18 xinetd[3390]: EXIT: shell status=0 pid=19063 duration=1(sec)
May 17 01:30:55 client-18 xinetd[3390]: START: shell pid=19098 from=192.168.4.5
May 17 01:30:55 client-18 rshd[19099]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:30:55 client-18 xinetd[3390]: EXIT: shell status=0 pid=19098 duration=0(sec)
May 17 01:30:55 client-18 xinetd[3390]: START: shell pid=19134 from=192.168.4.5
May 17 01:30:55 client-18 rshd[19135]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:30:55 client-18 xinetd[3390]: EXIT: shell status=0 pid=19134 duration=0(sec)
May 17 01:30:55 client-18 xinetd[3390]: START: shell pid=19171 from=192.168.4.5
May 17 01:30:55 client-18 rshd[19172]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 32h: open d32h\/symlink-\>tmp\/symlink-\>lustre-subdir\/2 ================================== 01:30:55 \(1305621055\)");echo XXRETCODE:$?'
May 17 01:30:55 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 32h: open d32h/symlink->tmp/symlink->lustre-subdir/2 ================================== 01:30:55 (1305621055)
May 17 01:30:55 client-18 xinetd[3390]: EXIT: shell status=0 pid=19171 duration=0(sec)
May 17 01:30:55 client-18 xinetd[3390]: START: shell pid=19206 from=192.168.4.5
May 17 01:30:55 client-18 rshd[19207]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:30:55 client-18 xinetd[3390]: EXIT: shell status=0 pid=19206 duration=0(sec)
May 17 01:30:55 client-18 xinetd[3390]: START: shell pid=19242 from=192.168.4.5
May 17 01:30:55 client-18 rshd[19243]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:30:55 client-18 xinetd[3390]: EXIT: shell status=0 pid=19242 duration=0(sec)
May 17 01:30:56 client-18 xinetd[3390]: START: shell pid=19279 from=192.168.4.5
May 17 01:30:56 client-18 rshd[19280]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 32i: stat d32i\/ext2-mountpoint\/..\/test_file =========================================== 01:30:55 \(1305621055\)");echo XXRETCODE:$?'
May 17 01:30:56 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 32i: stat d32i/ext2-mountpoint/../test_file =========================================== 01:30:55 (1305621055)
May 17 01:30:56 client-18 xinetd[3390]: EXIT: shell status=0 pid=19279 duration=0(sec)
May 17 01:30:56 client-18 xinetd[3390]: START: shell pid=19314 from=192.168.4.5
May 17 01:30:56 client-18 rshd[19315]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:30:56 client-18 xinetd[3390]: EXIT: shell status=0 pid=19314 duration=0(sec)
May 17 01:30:56 client-18 xinetd[3390]: START: shell pid=19350 from=192.168.4.5
May 17 01:30:56 client-18 rshd[19351]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:30:56 client-18 xinetd[3390]: EXIT: shell status=0 pid=19350 duration=0(sec)
May 17 01:30:56 client-18 xinetd[3390]: START: shell pid=19387 from=192.168.4.5
May 17 01:30:56 client-18 rshd[19388]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 32j: open d32j\/ext2-mountpoint\/..\/test_file =========================================== 01:30:56 \(1305621056\)");echo XXRETCODE:$?'
May 17 01:30:56 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 32j: open d32j/ext2-mountpoint/../test_file =========================================== 01:30:56 (1305621056)
May 17 01:30:56 client-18 xinetd[3390]: EXIT: shell status=0 pid=19387 duration=0(sec)
May 17 01:30:56 client-18 xinetd[3390]: START: shell pid=19422 from=192.168.4.5
May 17 01:30:56 client-18 rshd[19423]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:30:56 client-18 xinetd[3390]: EXIT: shell status=0 pid=19422 duration=0(sec)
May 17 01:30:56 client-18 xinetd[3390]: START: shell pid=19458 from=192.168.4.5
May 17 01:30:57 client-18 rshd[19459]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:30:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=19458 duration=1(sec)
May 17 01:30:57 client-18 xinetd[3390]: START: shell pid=19495 from=192.168.4.5
May 17 01:30:57 client-18 rshd[19496]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 32k: stat d32k\/ext2-mountpoint\/..\/d2\/test_file ======================================== 01:30:57 \(1305621057\)");echo XXRETCODE:$?'
May 17 01:30:57 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 32k: stat d32k/ext2-mountpoint/../d2/test_file ======================================== 01:30:57 (1305621057)
May 17 01:30:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=19495 duration=0(sec)
May 17 01:30:57 client-18 xinetd[3390]: START: shell pid=19530 from=192.168.4.5
May 17 01:30:57 client-18 rshd[19531]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:30:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=19530 duration=0(sec)
May 17 01:30:57 client-18 xinetd[3390]: START: shell pid=19566 from=192.168.4.5
May 17 01:30:57 client-18 rshd[19567]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:30:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=19566 duration=0(sec)
May 17 01:30:57 client-18 xinetd[3390]: START: shell pid=19603 from=192.168.4.5
May 17 01:30:57 client-18 rshd[19604]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 32l: open d32l\/ext2-mountpoint\/..\/d2\/test_file ======================================== 01:30:57 \(1305621057\)");echo XXRETCODE:$?'
May 17 01:30:57 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 32l: open d32l/ext2-mountpoint/../d2/test_file ======================================== 01:30:57 (1305621057)
May 17 01:30:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=19603 duration=0(sec)
May 17 01:30:57 client-18 xinetd[3390]: START: shell pid=19638 from=192.168.4.5
May 17 01:30:57 client-18 rshd[19639]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:30:58 client-18 xinetd[3390]: EXIT: shell status=0 pid=19638 duration=1(sec)
May 17 01:30:58 client-18 xinetd[3390]: START: shell pid=19674 from=192.168.4.5
May 17 01:30:58 client-18 rshd[19675]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:30:58 client-18 xinetd[3390]: EXIT: shell status=0 pid=19674 duration=0(sec)
May 17 01:30:58 client-18 xinetd[3390]: START: shell pid=19711 from=192.168.4.5
May 17 01:30:58 client-18 rshd[19712]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 32m: stat d32m\/symlink-\>tmp\/symlink-\>lustre-root ====================================== 01:30:58 \(1305621058\)");echo XXRETCODE:$?'
May 17 01:30:58 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 32m: stat d32m/symlink->tmp/symlink->lustre-root ====================================== 01:30:58 (1305621058)
May 17 01:30:58 client-18 xinetd[3390]: EXIT: shell status=0 pid=19711 duration=0(sec)
May 17 01:30:58 client-18 xinetd[3390]: START: shell pid=19746 from=192.168.4.5
May 17 01:30:58 client-18 rshd[19747]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:30:58 client-18 xinetd[3390]: EXIT: shell status=0 pid=19746 duration=0(sec)
May 17 01:30:58 client-18 xinetd[3390]: START: shell pid=19782 from=192.168.4.5
May 17 01:30:58 client-18 rshd[19783]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:30:58 client-18 xinetd[3390]: EXIT: shell status=0 pid=19782 duration=0(sec)
May 17 01:30:58 client-18 xinetd[3390]: START: shell pid=19819 from=192.168.4.5
May 17 01:30:58 client-18 rshd[19820]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 32n: open d32n\/symlink-\>tmp\/symlink-\>lustre-root ====================================== 01:30:58 \(1305621058\)");echo XXRETCODE:$?'
May 17 01:30:58 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 32n: open d32n/symlink->tmp/symlink->lustre-root ====================================== 01:30:58 (1305621058)
May 17 01:30:58 client-18 xinetd[3390]: EXIT: shell status=0 pid=19819 duration=0(sec)
May 17 01:30:59 client-18 xinetd[3390]: START: shell pid=19854 from=192.168.4.5
May 17 01:30:59 client-18 rshd[19855]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:30:59 client-18 xinetd[3390]: EXIT: shell status=0 pid=19854 duration=0(sec)
May 17 01:30:59 client-18 xinetd[3390]: START: shell pid=19890 from=192.168.4.5
May 17 01:30:59 client-18 rshd[19891]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:30:59 client-18 xinetd[3390]: EXIT: shell status=0 pid=19890 duration=0(sec)
May 17 01:30:59 client-18 xinetd[3390]: START: shell pid=19927 from=192.168.4.5
May 17 01:30:59 client-18 rshd[19928]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 32o: stat d32o\/symlink-\>tmp\/symlink-\>lustre-root\/ ===================================== 01:30:59 \(1305621059\)");echo XXRETCODE:$?'
May 17 01:30:59 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 32o: stat d32o/symlink->tmp/symlink->lustre-root/ ===================================== 01:30:59 (1305621059)
May 17 01:30:59 client-18 xinetd[3390]: EXIT: shell status=0 pid=19927 duration=0(sec)
May 17 01:30:59 client-18 xinetd[3390]: START: shell pid=19962 from=192.168.4.5
May 17 01:30:59 client-18 rshd[19963]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:30:59 client-18 xinetd[3390]: EXIT: shell status=0 pid=19962 duration=0(sec)
May 17 01:30:59 client-18 xinetd[3390]: START: shell pid=19998 from=192.168.4.5
May 17 01:30:59 client-18 rshd[19999]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:30:59 client-18 xinetd[3390]: EXIT: shell status=0 pid=19998 duration=0(sec)
May 17 01:30:59 client-18 xinetd[3390]: START: shell pid=20035 from=192.168.4.5
May 17 01:30:59 client-18 rshd[20036]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 32p: open d32p\/symlink-\>tmp\/symlink-\>lustre-root\/ ===================================== 01:30:59 \(1305621059\)");echo XXRETCODE:$?'
May 17 01:31:00 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 32p: open d32p/symlink->tmp/symlink->lustre-root/ ===================================== 01:30:59 (1305621059)
May 17 01:31:00 client-18 xinetd[3390]: EXIT: shell status=0 pid=20035 duration=1(sec)
May 17 01:31:00 client-18 xinetd[3390]: START: shell pid=20070 from=192.168.4.5
May 17 01:31:00 client-18 rshd[20071]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark 32p_1");echo XXRETCODE:$?'
May 17 01:31:00 client-18 kernel: Lustre: DEBUG MARKER: 32p_1
May 17 01:31:00 client-18 xinetd[3390]: EXIT: shell status=0 pid=20070 duration=0(sec)
May 17 01:31:00 client-18 xinetd[3390]: START: shell pid=20105 from=192.168.4.5
May 17 01:31:00 client-18 rshd[20106]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark 32p_2");echo XXRETCODE:$?'
May 17 01:31:00 client-18 kernel: Lustre: DEBUG MARKER: 32p_2
May 17 01:31:00 client-18 xinetd[3390]: EXIT: shell status=0 pid=20105 duration=0(sec)
May 17 01:31:00 client-18 xinetd[3390]: START: shell pid=20140 from=192.168.4.5
May 17 01:31:00 client-18 rshd[20141]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark 32p_3");echo XXRETCODE:$?'
May 17 01:31:00 client-18 kernel: Lustre: DEBUG MARKER: 32p_3
May 17 01:31:00 client-18 xinetd[3390]: EXIT: shell status=0 pid=20140 duration=0(sec)
May 17 01:31:00 client-18 xinetd[3390]: START: shell pid=20175 from=192.168.4.5
May 17 01:31:00 client-18 rshd[20176]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark 32p_4");echo XXRETCODE:$?'
May 17 01:31:00 client-18 kernel: Lustre: DEBUG MARKER: 32p_4
May 17 01:31:00 client-18 xinetd[3390]: EXIT: shell status=0 pid=20175 duration=0(sec)
May 17 01:31:00 client-18 xinetd[3390]: START: shell pid=20210 from=192.168.4.5
May 17 01:31:00 client-18 rshd[20211]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark 32p_5");echo XXRETCODE:$?'
May 17 01:31:00 client-18 kernel: Lustre: DEBUG MARKER: 32p_5
May 17 01:31:00 client-18 xinetd[3390]: EXIT: shell status=0 pid=20210 duration=0(sec)
May 17 01:31:00 client-18 xinetd[3390]: START: shell pid=20245 from=192.168.4.5
May 17 01:31:00 client-18 rshd[20246]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark 32p_6");echo XXRETCODE:$?'
May 17 01:31:00 client-18 kernel: Lustre: DEBUG MARKER: 32p_6
May 17 01:31:00 client-18 xinetd[3390]: EXIT: shell status=0 pid=20245 duration=0(sec)
May 17 01:31:01 client-18 xinetd[3390]: START: shell pid=20280 from=192.168.4.5
May 17 01:31:01 client-18 rshd[20281]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark 32p_7");echo XXRETCODE:$?'
May 17 01:31:01 client-18 kernel: Lustre: DEBUG MARKER: 32p_7
May 17 01:31:01 client-18 xinetd[3390]: EXIT: shell status=0 pid=20280 duration=0(sec)
May 17 01:31:01 client-18 xinetd[3390]: START: shell pid=20315 from=192.168.4.5
May 17 01:31:01 client-18 rshd[20316]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark 32p_8");echo XXRETCODE:$?'
May 17 01:31:01 client-18 kernel: Lustre: DEBUG MARKER: 32p_8
May 17 01:31:01 client-18 xinetd[3390]: EXIT: shell status=0 pid=20315 duration=0(sec)
May 17 01:31:01 client-18 xinetd[3390]: START: shell pid=20350 from=192.168.4.5
May 17 01:31:01 client-18 rshd[20351]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark 32p_9");echo XXRETCODE:$?'
May 17 01:31:01 client-18 kernel: Lustre: DEBUG MARKER: 32p_9
May 17 01:31:01 client-18 xinetd[3390]: EXIT: shell status=0 pid=20350 duration=0(sec)
May 17 01:31:01 client-18 xinetd[3390]: START: shell pid=20385 from=192.168.4.5
May 17 01:31:01 client-18 rshd[20386]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark 32p_10");echo XXRETCODE:$?'
May 17 01:31:01 client-18 kernel: Lustre: DEBUG MARKER: 32p_10
May 17 01:31:01 client-18 xinetd[3390]: EXIT: shell status=0 pid=20385 duration=0(sec)
May 17 01:31:01 client-18 xinetd[3390]: START: shell pid=20420 from=192.168.4.5
May 17 01:31:01 client-18 rshd[20421]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:31:01 client-18 xinetd[3390]: EXIT: shell status=0 pid=20420 duration=0(sec)
May 17 01:31:01 client-18 xinetd[3390]: START: shell pid=20456 from=192.168.4.5
May 17 01:31:01 client-18 rshd[20457]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:31:01 client-18 xinetd[3390]: EXIT: shell status=0 pid=20456 duration=0(sec)
May 17 01:31:02 client-18 xinetd[3390]: START: shell pid=20493 from=192.168.4.5
May 17 01:31:02 client-18 rshd[20494]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 32q: stat follows mountpoints in Lustre \(should return error\) ========================= 01:31:01 \(1305621061\)");echo XXRETCODE:$?'
May 17 01:31:02 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 32q: stat follows mountpoints in Lustre (should return error) ========================= 01:31:01 (1305621061)
May 17 01:31:02 client-18 xinetd[3390]: EXIT: shell status=0 pid=20493 duration=0(sec)
May 17 01:31:02 client-18 xinetd[3390]: START: shell pid=20528 from=192.168.4.5
May 17 01:31:02 client-18 rshd[20529]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:31:02 client-18 xinetd[3390]: EXIT: shell status=0 pid=20528 duration=0(sec)
May 17 01:31:02 client-18 xinetd[3390]: START: shell pid=20564 from=192.168.4.5
May 17 01:31:02 client-18 rshd[20565]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:31:02 client-18 xinetd[3390]: EXIT: shell status=0 pid=20564 duration=0(sec)
May 17 01:31:02 client-18 xinetd[3390]: START: shell pid=20601 from=192.168.4.5
May 17 01:31:02 client-18 rshd[20602]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 32r: opendir follows mountpoints in Lustre \(should return error\) ====================== 01:31:02 \(1305621062\)");echo XXRETCODE:$?'
May 17 01:31:02 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 32r: opendir follows mountpoints in Lustre (should return error) ====================== 01:31:02 (1305621062)
May 17 01:31:02 client-18 xinetd[3390]: EXIT: shell status=0 pid=20601 duration=0(sec)
May 17 01:31:02 client-18 xinetd[3390]: START: shell pid=20636 from=192.168.4.5
May 17 01:31:02 client-18 rshd[20637]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:31:02 client-18 xinetd[3390]: EXIT: shell status=0 pid=20636 duration=0(sec)
May 17 01:31:02 client-18 xinetd[3390]: START: shell pid=20676 from=192.168.4.5
May 17 01:31:02 client-18 rshd[20677]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:31:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=20676 duration=1(sec)
May 17 01:31:03 client-18 xinetd[3390]: START: shell pid=20713 from=192.168.4.5
May 17 01:31:03 client-18 rshd[20714]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 33: write file with mode 444 \(should return error\) ==================================== 01:31:03 \(1305621063\)");echo XXRETCODE:$?'
May 17 01:31:03 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 33: write file with mode 444 (should return error) ==================================== 01:31:03 (1305621063)
May 17 01:31:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=20713 duration=0(sec)
May 17 01:31:03 client-18 xinetd[3390]: START: shell pid=20748 from=192.168.4.5
May 17 01:31:03 client-18 rshd[20749]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark 33_1");echo XXRETCODE:$?'
May 17 01:31:03 client-18 kernel: Lustre: DEBUG MARKER: 33_1
May 17 01:31:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=20748 duration=0(sec)
May 17 01:31:03 client-18 xinetd[3390]: START: shell pid=20783 from=192.168.4.5
May 17 01:31:03 client-18 rshd[20784]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark 33_2");echo XXRETCODE:$?'
May 17 01:31:03 client-18 kernel: Lustre: DEBUG MARKER: 33_2
May 17 01:31:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=20783 duration=0(sec)
May 17 01:31:03 client-18 xinetd[3390]: START: shell pid=20818 from=192.168.4.5
May 17 01:31:03 client-18 rshd[20819]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:31:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=20818 duration=0(sec)
May 17 01:31:03 client-18 xinetd[3390]: START: shell pid=20854 from=192.168.4.5
May 17 01:31:03 client-18 rshd[20855]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:31:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=20854 duration=0(sec)
May 17 01:31:03 client-18 xinetd[3390]: START: shell pid=20891 from=192.168.4.5
May 17 01:31:04 client-18 rshd[20892]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 33a: test open file\(mode=0444\) with O_RDWR \(should return error\) ====================== 01:31:03 \(1305621063\)");echo XXRETCODE:$?'
May 17 01:31:04 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 33a: test open file(mode=0444) with O_RDWR (should return error) ====================== 01:31:03 (1305621063)
May 17 01:31:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=20891 duration=1(sec)
May 17 01:31:04 client-18 xinetd[3390]: START: shell pid=20926 from=192.168.4.5
May 17 01:31:04 client-18 rshd[20927]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:31:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=20926 duration=0(sec)
May 17 01:31:04 client-18 xinetd[3390]: START: shell pid=20962 from=192.168.4.5
May 17 01:31:04 client-18 rshd[20963]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:31:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=20962 duration=0(sec)
May 17 01:31:04 client-18 xinetd[3390]: START: shell pid=20999 from=192.168.4.5
May 17 01:31:04 client-18 rshd[21000]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 33b: test open file with malformed flags \(No panic and return error\) ================== 01:31:04 \(1305621064\)");echo XXRETCODE:$?'
May 17 01:31:04 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 33b: test open file with malformed flags (No panic and return error) ================== 01:31:04 (1305621064)
May 17 01:31:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=20999 duration=0(sec)
May 17 01:31:04 client-18 xinetd[3390]: START: shell pid=21034 from=192.168.4.5
May 17 01:31:04 client-18 rshd[21035]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:31:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=21034 duration=0(sec)
May 17 01:31:04 client-18 xinetd[3390]: START: shell pid=21070 from=192.168.4.5
May 17 01:31:04 client-18 rshd[21071]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:31:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=21070 duration=0(sec)
May 17 01:31:05 client-18 xinetd[3390]: START: shell pid=21107 from=192.168.4.5
May 17 01:31:05 client-18 rshd[21108]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 33c: test llobdstat and write_bytes =================================================== 01:31:04 \(1305621064\)");echo XXRETCODE:$?'
May 17 01:31:05 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 33c: test llobdstat and write_bytes =================================================== 01:31:04 (1305621064)
May 17 01:31:05 client-18 xinetd[3390]: EXIT: shell status=0 pid=21107 duration=0(sec)
May 17 01:31:05 client-18 xinetd[3390]: START: shell pid=21142 from=192.168.4.5
May 17 01:31:05 client-18 rshd[21143]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:31:05 client-18 xinetd[3390]: EXIT: shell status=0 pid=21142 duration=0(sec)
May 17 01:31:05 client-18 xinetd[3390]: START: shell pid=21178 from=192.168.4.5
May 17 01:31:05 client-18 rshd[21179]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:31:05 client-18 xinetd[3390]: EXIT: shell status=0 pid=21178 duration=0(sec)
May 17 01:31:05 client-18 xinetd[3390]: START: shell pid=21215 from=192.168.4.5
May 17 01:31:05 client-18 rshd[21216]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 34a: truncate file that has not been opened =========================================== 01:31:05 \(1305621065\)");echo XXRETCODE:$?'
May 17 01:31:06 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 34a: truncate file that has not been opened =========================================== 01:31:05 (1305621065)
May 17 01:31:06 client-18 xinetd[3390]: EXIT: shell status=0 pid=21215 duration=1(sec)
May 17 01:31:06 client-18 xinetd[3390]: START: shell pid=21250 from=192.168.4.5
May 17 01:31:06 client-18 rshd[21251]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:31:06 client-18 xinetd[3390]: EXIT: shell status=0 pid=21250 duration=0(sec)
May 17 01:31:06 client-18 xinetd[3390]: START: shell pid=21286 from=192.168.4.5
May 17 01:31:06 client-18 rshd[21287]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:31:06 client-18 xinetd[3390]: EXIT: shell status=0 pid=21286 duration=0(sec)
May 17 01:31:06 client-18 xinetd[3390]: START: shell pid=21323 from=192.168.4.5
May 17 01:31:06 client-18 rshd[21324]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 34b: O_RDONLY opening file doesn\'t create objects ===================================== 01:31:06 \(1305621066\)");echo XXRETCODE:$?'
May 17 01:31:06 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 34b: O_RDONLY opening file doesn't create objects ===================================== 01:31:06 (1305621066)
May 17 01:31:06 client-18 xinetd[3390]: EXIT: shell status=0 pid=21323 duration=0(sec)
May 17 01:31:06 client-18 xinetd[3390]: START: shell pid=21358 from=192.168.4.5
May 17 01:31:06 client-18 rshd[21359]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:31:06 client-18 xinetd[3390]: EXIT: shell status=0 pid=21358 duration=0(sec)
May 17 01:31:06 client-18 xinetd[3390]: START: shell pid=21394 from=192.168.4.5
May 17 01:31:06 client-18 rshd[21395]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:31:06 client-18 xinetd[3390]: EXIT: shell status=0 pid=21394 duration=0(sec)
May 17 01:31:06 client-18 xinetd[3390]: START: shell pid=21431 from=192.168.4.5
May 17 01:31:06 client-18 rshd[21432]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 34c: O_RDWR opening file-with-size works ============================================== 01:31:06 \(1305621066\)");echo XXRETCODE:$?'
May 17 01:31:07 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 34c: O_RDWR opening file-with-size works ============================================== 01:31:06 (1305621066)
May 17 01:31:07 client-18 xinetd[3390]: EXIT: shell status=0 pid=21431 duration=1(sec)
May 17 01:31:07 client-18 xinetd[3390]: START: shell pid=21466 from=192.168.4.5
May 17 01:31:07 client-18 rshd[21467]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:31:07 client-18 xinetd[3390]: EXIT: shell status=0 pid=21466 duration=0(sec)
May 17 01:31:07 client-18 xinetd[3390]: START: shell pid=21502 from=192.168.4.5
May 17 01:31:07 client-18 rshd[21503]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:31:07 client-18 xinetd[3390]: EXIT: shell status=0 pid=21502 duration=0(sec)
May 17 01:31:07 client-18 xinetd[3390]: START: shell pid=21539 from=192.168.4.5
May 17 01:31:07 client-18 rshd[21540]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 34d: write to sparse file ============================================================= 01:31:07 \(1305621067\)");echo XXRETCODE:$?'
May 17 01:31:07 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 34d: write to sparse file ============================================================= 01:31:07 (1305621067)
May 17 01:31:07 client-18 xinetd[3390]: EXIT: shell status=0 pid=21539 duration=0(sec)
May 17 01:31:07 client-18 xinetd[3390]: START: shell pid=21574 from=192.168.4.5
May 17 01:31:07 client-18 rshd[21575]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:31:07 client-18 xinetd[3390]: EXIT: shell status=0 pid=21574 duration=0(sec)
May 17 01:31:07 client-18 xinetd[3390]: START: shell pid=21610 from=192.168.4.5
May 17 01:31:07 client-18 rshd[21611]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:31:07 client-18 xinetd[3390]: EXIT: shell status=0 pid=21610 duration=0(sec)
May 17 01:31:08 client-18 xinetd[3390]: START: shell pid=21647 from=192.168.4.5
May 17 01:31:08 client-18 rshd[21648]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 34e: create objects, some with size and some without ================================== 01:31:07 \(1305621067\)");echo XXRETCODE:$?'
May 17 01:31:08 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 34e: create objects, some with size and some without ================================== 01:31:07 (1305621067)
May 17 01:31:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=21647 duration=0(sec)
May 17 01:31:08 client-18 xinetd[3390]: START: shell pid=21682 from=192.168.4.5
May 17 01:31:08 client-18 rshd[21683]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:31:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=21682 duration=0(sec)
May 17 01:31:08 client-18 xinetd[3390]: START: shell pid=21718 from=192.168.4.5
May 17 01:31:08 client-18 rshd[21719]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:31:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=21718 duration=0(sec)
May 17 01:31:08 client-18 xinetd[3390]: START: shell pid=21755 from=192.168.4.5
May 17 01:31:08 client-18 rshd[21756]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 34f: read from a file with no objects until EOF ======================================= 01:31:08 \(1305621068\)");echo XXRETCODE:$?'
May 17 01:31:08 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 34f: read from a file with no objects until EOF ======================================= 01:31:08 (1305621068)
May 17 01:31:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=21755 duration=0(sec)
May 17 01:31:08 client-18 xinetd[3390]: START: shell pid=21790 from=192.168.4.5
May 17 01:31:08 client-18 rshd[21791]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:31:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=21790 duration=0(sec)
May 17 01:31:08 client-18 xinetd[3390]: START: shell pid=21826 from=192.168.4.5
May 17 01:31:08 client-18 rshd[21827]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:31:09 client-18 xinetd[3390]: EXIT: shell status=0 pid=21826 duration=1(sec)
May 17 01:31:09 client-18 xinetd[3390]: START: shell pid=21863 from=192.168.4.5
May 17 01:31:09 client-18 rshd[21864]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 34g: truncate long file =============================================================== 01:31:09 \(1305621069\)");echo XXRETCODE:$?'
May 17 01:31:09 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 34g: truncate long file =============================================================== 01:31:09 (1305621069)
May 17 01:31:09 client-18 xinetd[3390]: EXIT: shell status=0 pid=21863 duration=0(sec)
May 17 01:31:09 client-18 xinetd[3390]: START: shell pid=21898 from=192.168.4.5
May 17 01:31:09 client-18 rshd[21899]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:31:09 client-18 xinetd[3390]: EXIT: shell status=0 pid=21898 duration=0(sec)
May 17 01:31:09 client-18 xinetd[3390]: START: shell pid=21934 from=192.168.4.5
May 17 01:31:09 client-18 rshd[21935]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:31:09 client-18 xinetd[3390]: EXIT: shell status=0 pid=21934 duration=0(sec)
May 17 01:31:09 client-18 xinetd[3390]: START: shell pid=21971 from=192.168.4.5
May 17 01:31:09 client-18 rshd[21972]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 35a: exec file with mode 444 \(should return and not leak\) ============================= 01:31:09 \(1305621069\)");echo XXRETCODE:$?'
May 17 01:31:09 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 35a: exec file with mode 444 (should return and not leak) ============================= 01:31:09 (1305621069)
May 17 01:31:09 client-18 xinetd[3390]: EXIT: shell status=0 pid=21971 duration=0(sec)
May 17 01:31:10 client-18 xinetd[3390]: START: shell pid=22006 from=192.168.4.5
May 17 01:31:10 client-18 rshd[22007]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:31:10 client-18 xinetd[3390]: EXIT: shell status=0 pid=22006 duration=0(sec)
May 17 01:31:10 client-18 xinetd[3390]: START: shell pid=22042 from=192.168.4.5
May 17 01:31:10 client-18 rshd[22043]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:31:10 client-18 xinetd[3390]: EXIT: shell status=0 pid=22042 duration=0(sec)
May 17 01:31:10 client-18 xinetd[3390]: START: shell pid=22079 from=192.168.4.5
May 17 01:31:10 client-18 rshd[22080]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 36a: MDS utime check \(mknod, utime\) =================================================== 01:31:10 \(1305621070\)");echo XXRETCODE:$?'
May 17 01:31:10 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 36a: MDS utime check (mknod, utime) =================================================== 01:31:10 (1305621070)
May 17 01:31:10 client-18 xinetd[3390]: EXIT: shell status=0 pid=22079 duration=0(sec)
May 17 01:31:10 client-18 xinetd[3390]: START: shell pid=22114 from=192.168.4.5
May 17 01:31:10 client-18 rshd[22115]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:31:10 client-18 xinetd[3390]: EXIT: shell status=0 pid=22114 duration=0(sec)
May 17 01:31:10 client-18 xinetd[3390]: START: shell pid=22150 from=192.168.4.5
May 17 01:31:10 client-18 rshd[22151]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:31:10 client-18 xinetd[3390]: EXIT: shell status=0 pid=22150 duration=0(sec)
May 17 01:31:10 client-18 xinetd[3390]: START: shell pid=22187 from=192.168.4.5
May 17 01:31:11 client-18 rshd[22188]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 36b: OST utime check \(open, utime\) ==================================================== 01:31:10 \(1305621070\)");echo XXRETCODE:$?'
May 17 01:31:11 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 36b: OST utime check (open, utime) ==================================================== 01:31:10 (1305621070)
May 17 01:31:11 client-18 xinetd[3390]: EXIT: shell status=0 pid=22187 duration=1(sec)
May 17 01:31:11 client-18 xinetd[3390]: START: shell pid=22222 from=192.168.4.5
May 17 01:31:11 client-18 rshd[22223]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:31:11 client-18 xinetd[3390]: EXIT: shell status=0 pid=22222 duration=0(sec)
May 17 01:31:11 client-18 xinetd[3390]: START: shell pid=22258 from=192.168.4.5
May 17 01:31:11 client-18 rshd[22259]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:31:11 client-18 xinetd[3390]: EXIT: shell status=0 pid=22258 duration=0(sec)
May 17 01:31:11 client-18 xinetd[3390]: START: shell pid=22295 from=192.168.4.5
May 17 01:31:11 client-18 rshd[22296]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 36c: non-root MDS utime check \(mknod, utime\) ========================================== 01:31:11 \(1305621071\)");echo XXRETCODE:$?'
May 17 01:31:11 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 36c: non-root MDS utime check (mknod, utime) ========================================== 01:31:11 (1305621071)
May 17 01:31:11 client-18 xinetd[3390]: EXIT: shell status=0 pid=22295 duration=0(sec)
May 17 01:31:11 client-18 xinetd[3390]: START: shell pid=22330 from=192.168.4.5
May 17 01:31:11 client-18 rshd[22331]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:31:11 client-18 xinetd[3390]: EXIT: shell status=0 pid=22330 duration=0(sec)
May 17 01:31:11 client-18 xinetd[3390]: START: shell pid=22366 from=192.168.4.5
May 17 01:31:11 client-18 rshd[22367]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:31:11 client-18 xinetd[3390]: EXIT: shell status=0 pid=22366 duration=0(sec)
May 17 01:31:12 client-18 xinetd[3390]: START: shell pid=22403 from=192.168.4.5
May 17 01:31:12 client-18 rshd[22404]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 36d: non-root OST utime check \(open, utime\) =========================================== 01:31:11 \(1305621071\)");echo XXRETCODE:$?'
May 17 01:31:12 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 36d: non-root OST utime check (open, utime) =========================================== 01:31:11 (1305621071)
May 17 01:31:12 client-18 xinetd[3390]: EXIT: shell status=0 pid=22403 duration=0(sec)
May 17 01:31:12 client-18 xinetd[3390]: START: shell pid=22438 from=192.168.4.5
May 17 01:31:12 client-18 rshd[22439]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:31:12 client-18 xinetd[3390]: EXIT: shell status=0 pid=22438 duration=0(sec)
May 17 01:31:12 client-18 xinetd[3390]: START: shell pid=22474 from=192.168.4.5
May 17 01:31:12 client-18 rshd[22475]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:31:12 client-18 xinetd[3390]: EXIT: shell status=0 pid=22474 duration=0(sec)
May 17 01:31:12 client-18 xinetd[3390]: START: shell pid=22511 from=192.168.4.5
May 17 01:31:12 client-18 rshd[22512]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 36e: utime on non-owned file \(should return error\) ==================================== 01:31:12 \(1305621072\)");echo XXRETCODE:$?'
May 17 01:31:12 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 36e: utime on non-owned file (should return error) ==================================== 01:31:12 (1305621072)
May 17 01:31:12 client-18 xinetd[3390]: EXIT: shell status=0 pid=22511 duration=0(sec)
May 17 01:31:12 client-18 xinetd[3390]: START: shell pid=22546 from=192.168.4.5
May 17 01:31:12 client-18 rshd[22547]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:31:12 client-18 xinetd[3390]: EXIT: shell status=0 pid=22546 duration=0(sec)
May 17 01:31:12 client-18 xinetd[3390]: START: shell pid=22582 from=192.168.4.5
May 17 01:31:12 client-18 rshd[22583]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:31:12 client-18 xinetd[3390]: EXIT: shell status=0 pid=22582 duration=0(sec)
May 17 01:31:13 client-18 xinetd[3390]: START: shell pid=22619 from=192.168.4.5
May 17 01:31:13 client-18 rshd[22620]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 36f: utime on file racing with OST BRW write ========================================== 01:31:12 \(1305621072\)");echo XXRETCODE:$?'
May 17 01:31:13 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 36f: utime on file racing with OST BRW write ========================================== 01:31:12 (1305621072)
May 17 01:31:13 client-18 xinetd[3390]: EXIT: shell status=0 pid=22619 duration=0(sec)
May 17 01:31:14 client-18 xinetd[3390]: START: shell pid=22654 from=192.168.4.5
May 17 01:31:14 client-18 rshd[22655]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:31:14 client-18 xinetd[3390]: EXIT: shell status=0 pid=22654 duration=0(sec)
May 17 01:31:14 client-18 xinetd[3390]: START: shell pid=22690 from=192.168.4.5
May 17 01:31:14 client-18 rshd[22691]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:31:14 client-18 xinetd[3390]: EXIT: shell status=0 pid=22690 duration=0(sec)
May 17 01:31:14 client-18 xinetd[3390]: START: shell pid=22727 from=192.168.4.5
May 17 01:31:14 client-18 rshd[22728]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 36g: filter mod data cache expiry ===================================================== 01:31:14 \(1305621074\)");echo XXRETCODE:$?'
May 17 01:31:14 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 36g: filter mod data cache expiry ===================================================== 01:31:14 (1305621074)
May 17 01:31:14 client-18 xinetd[3390]: EXIT: shell status=0 pid=22727 duration=0(sec)
May 17 01:31:57 client-18 xinetd[3390]: START: shell pid=22762 from=192.168.4.5
May 17 01:31:57 client-18 rshd[22763]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:31:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=22762 duration=0(sec)
May 17 01:31:57 client-18 xinetd[3390]: START: shell pid=22798 from=192.168.4.5
May 17 01:31:57 client-18 rshd[22799]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:31:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=22798 duration=0(sec)
May 17 01:31:57 client-18 xinetd[3390]: START: shell pid=22835 from=192.168.4.5
May 17 01:31:57 client-18 rshd[22836]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 36h: utime on file racing with OST BRW write ========================================== 01:31:57 \(1305621117\)");echo XXRETCODE:$?'
May 17 01:31:57 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 36h: utime on file racing with OST BRW write ========================================== 01:31:57 (1305621117)
May 17 01:31:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=22835 duration=0(sec)
May 17 01:31:58 client-18 xinetd[3390]: START: shell pid=22870 from=192.168.4.5
May 17 01:31:58 client-18 rshd[22871]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:31:58 client-18 xinetd[3390]: EXIT: shell status=0 pid=22870 duration=0(sec)
May 17 01:31:59 client-18 xinetd[3390]: START: shell pid=22906 from=192.168.4.5
May 17 01:31:59 client-18 rshd[22907]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:31:59 client-18 xinetd[3390]: EXIT: shell status=0 pid=22906 duration=0(sec)
May 17 01:31:59 client-18 xinetd[3390]: START: shell pid=22943 from=192.168.4.5
May 17 01:31:59 client-18 rshd[22944]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 37: ls a mounted file system to check old content ===================================== 01:31:59 \(1305621119\)");echo XXRETCODE:$?'
May 17 01:31:59 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 37: ls a mounted file system to check old content ===================================== 01:31:59 (1305621119)
May 17 01:31:59 client-18 xinetd[3390]: EXIT: shell status=0 pid=22943 duration=0(sec)
May 17 01:31:59 client-18 xinetd[3390]: START: shell pid=22978 from=192.168.4.5
May 17 01:31:59 client-18 rshd[22979]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:31:59 client-18 xinetd[3390]: EXIT: shell status=0 pid=22978 duration=0(sec)
May 17 01:31:59 client-18 xinetd[3390]: START: shell pid=23014 from=192.168.4.5
May 17 01:31:59 client-18 rshd[23015]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:31:59 client-18 xinetd[3390]: EXIT: shell status=0 pid=23014 duration=0(sec)
May 17 01:31:59 client-18 xinetd[3390]: START: shell pid=23051 from=192.168.4.5
May 17 01:31:59 client-18 rshd[23052]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 38: open a regular file with O_DIRECTORY should return -ENOTDIR ======================= 01:31:59 \(1305621119\)");echo XXRETCODE:$?'
May 17 01:31:59 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 38: open a regular file with O_DIRECTORY should return -ENOTDIR ======================= 01:31:59 (1305621119)
May 17 01:31:59 client-18 xinetd[3390]: EXIT: shell status=0 pid=23051 duration=0(sec)
May 17 01:31:59 client-18 xinetd[3390]: START: shell pid=23086 from=192.168.4.5
May 17 01:31:59 client-18 rshd[23087]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:32:00 client-18 xinetd[3390]: EXIT: shell status=0 pid=23086 duration=1(sec)
May 17 01:32:00 client-18 xinetd[3390]: START: shell pid=23122 from=192.168.4.5
May 17 01:32:00 client-18 rshd[23123]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:32:00 client-18 xinetd[3390]: EXIT: shell status=0 pid=23122 duration=0(sec)
May 17 01:32:00 client-18 xinetd[3390]: START: shell pid=23159 from=192.168.4.5
May 17 01:32:00 client-18 rshd[23160]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 39: mtime changed on create =========================================================== 01:32:00 \(1305621120\)");echo XXRETCODE:$?'
May 17 01:32:00 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 39: mtime changed on create =========================================================== 01:32:00 (1305621120)
May 17 01:32:00 client-18 xinetd[3390]: EXIT: shell status=0 pid=23159 duration=0(sec)
May 17 01:32:02 client-18 xinetd[3390]: START: shell pid=23194 from=192.168.4.5
May 17 01:32:02 client-18 rshd[23195]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:32:02 client-18 xinetd[3390]: EXIT: shell status=0 pid=23194 duration=0(sec)
May 17 01:32:02 client-18 xinetd[3390]: START: shell pid=23230 from=192.168.4.5
May 17 01:32:02 client-18 rshd[23231]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:32:02 client-18 xinetd[3390]: EXIT: shell status=0 pid=23230 duration=0(sec)
May 17 01:32:02 client-18 xinetd[3390]: START: shell pid=23267 from=192.168.4.5
May 17 01:32:02 client-18 rshd[23268]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 39b: mtime change on open, link, unlink, rename  ====================================== 01:32:02 \(1305621122\)");echo XXRETCODE:$?'
May 17 01:32:02 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 39b: mtime change on open, link, unlink, rename ====================================== 01:32:02 (1305621122)
May 17 01:32:02 client-18 xinetd[3390]: EXIT: shell status=0 pid=23267 duration=0(sec)
May 17 01:32:04 client-18 xinetd[3390]: START: shell pid=23302 from=192.168.4.5
May 17 01:32:04 client-18 rshd[23303]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:32:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=23302 duration=0(sec)
May 17 01:32:04 client-18 xinetd[3390]: START: shell pid=23338 from=192.168.4.5
May 17 01:32:04 client-18 rshd[23339]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:32:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=23338 duration=0(sec)
May 17 01:32:04 client-18 xinetd[3390]: START: shell pid=23375 from=192.168.4.5
May 17 01:32:04 client-18 rshd[23376]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 39c: mtime change on rename =========================================================== 01:32:04 \(1305621124\)");echo XXRETCODE:$?'
May 17 01:32:04 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 39c: mtime change on rename =========================================================== 01:32:04 (1305621124)
May 17 01:32:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=23375 duration=0(sec)
May 17 01:32:07 client-18 xinetd[3390]: START: shell pid=23410 from=192.168.4.5
May 17 01:32:07 client-18 rshd[23411]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:32:07 client-18 xinetd[3390]: EXIT: shell status=0 pid=23410 duration=0(sec)
May 17 01:32:07 client-18 xinetd[3390]: START: shell pid=23446 from=192.168.4.5
May 17 01:32:07 client-18 rshd[23447]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:32:07 client-18 xinetd[3390]: EXIT: shell status=0 pid=23446 duration=0(sec)
May 17 01:32:07 client-18 xinetd[3390]: START: shell pid=23483 from=192.168.4.5
May 17 01:32:07 client-18 rshd[23484]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 39d: create, utime, stat ============================================================== 01:32:07 \(1305621127\)");echo XXRETCODE:$?'
May 17 01:32:07 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 39d: create, utime, stat ============================================================== 01:32:07 (1305621127)
May 17 01:32:07 client-18 xinetd[3390]: EXIT: shell status=0 pid=23483 duration=0(sec)
May 17 01:32:07 client-18 xinetd[3390]: START: shell pid=23518 from=192.168.4.5
May 17 01:32:07 client-18 rshd[23519]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:32:07 client-18 xinetd[3390]: EXIT: shell status=0 pid=23518 duration=0(sec)
May 17 01:32:07 client-18 xinetd[3390]: START: shell pid=23554 from=192.168.4.5
May 17 01:32:07 client-18 rshd[23555]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:32:07 client-18 xinetd[3390]: EXIT: shell status=0 pid=23554 duration=0(sec)
May 17 01:32:07 client-18 xinetd[3390]: START: shell pid=23591 from=192.168.4.5
May 17 01:32:07 client-18 rshd[23592]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 39e: create, stat, utime, stat ======================================================== 01:32:07 \(1305621127\)");echo XXRETCODE:$?'
May 17 01:32:08 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 39e: create, stat, utime, stat ======================================================== 01:32:07 (1305621127)
May 17 01:32:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=23591 duration=1(sec)
May 17 01:32:08 client-18 xinetd[3390]: START: shell pid=23626 from=192.168.4.5
May 17 01:32:08 client-18 rshd[23627]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:32:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=23626 duration=0(sec)
May 17 01:32:08 client-18 xinetd[3390]: START: shell pid=23662 from=192.168.4.5
May 17 01:32:08 client-18 rshd[23663]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:32:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=23662 duration=0(sec)
May 17 01:32:08 client-18 xinetd[3390]: START: shell pid=23699 from=192.168.4.5
May 17 01:32:08 client-18 rshd[23700]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 39f: create, stat, sleep, utime, stat ================================================= 01:32:08 \(1305621128\)");echo XXRETCODE:$?'
May 17 01:32:08 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 39f: create, stat, sleep, utime, stat ================================================= 01:32:08 (1305621128)
May 17 01:32:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=23699 duration=0(sec)
May 17 01:32:10 client-18 xinetd[3390]: START: shell pid=23734 from=192.168.4.5
May 17 01:32:10 client-18 rshd[23735]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:32:10 client-18 xinetd[3390]: EXIT: shell status=0 pid=23734 duration=0(sec)
May 17 01:32:10 client-18 xinetd[3390]: START: shell pid=23770 from=192.168.4.5
May 17 01:32:10 client-18 rshd[23771]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:32:11 client-18 xinetd[3390]: EXIT: shell status=0 pid=23770 duration=1(sec)
May 17 01:32:11 client-18 xinetd[3390]: START: shell pid=23807 from=192.168.4.5
May 17 01:32:11 client-18 rshd[23808]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 39g: write, chmod, stat =============================================================== 01:32:11 \(1305621131\)");echo XXRETCODE:$?'
May 17 01:32:11 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 39g: write, chmod, stat =============================================================== 01:32:11 (1305621131)
May 17 01:32:11 client-18 xinetd[3390]: EXIT: shell status=0 pid=23807 duration=0(sec)
May 17 01:32:13 client-18 xinetd[3390]: START: shell pid=23842 from=192.168.4.5
May 17 01:32:13 client-18 rshd[23843]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:32:13 client-18 xinetd[3390]: EXIT: shell status=0 pid=23842 duration=0(sec)
May 17 01:32:13 client-18 xinetd[3390]: START: shell pid=23878 from=192.168.4.5
May 17 01:32:13 client-18 rshd[23879]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:32:13 client-18 xinetd[3390]: EXIT: shell status=0 pid=23878 duration=0(sec)
May 17 01:32:13 client-18 xinetd[3390]: START: shell pid=23915 from=192.168.4.5
May 17 01:32:13 client-18 rshd[23916]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 39h: write, utime within one second, stat ============================================= 01:32:13 \(1305621133\)");echo XXRETCODE:$?'
May 17 01:32:13 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 39h: write, utime within one second, stat ============================================= 01:32:13 (1305621133)
May 17 01:32:13 client-18 xinetd[3390]: EXIT: shell status=0 pid=23915 duration=0(sec)
May 17 01:32:15 client-18 xinetd[3390]: START: shell pid=23950 from=192.168.4.5
May 17 01:32:15 client-18 rshd[23951]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:32:15 client-18 xinetd[3390]: EXIT: shell status=0 pid=23950 duration=0(sec)
May 17 01:32:15 client-18 xinetd[3390]: START: shell pid=23986 from=192.168.4.5
May 17 01:32:15 client-18 rshd[23987]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:32:15 client-18 xinetd[3390]: EXIT: shell status=0 pid=23986 duration=0(sec)
May 17 01:32:15 client-18 xinetd[3390]: START: shell pid=24023 from=192.168.4.5
May 17 01:32:15 client-18 rshd[24024]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 39i: write, rename, stat ============================================================== 01:32:15 \(1305621135\)");echo XXRETCODE:$?'
May 17 01:32:15 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 39i: write, rename, stat ============================================================== 01:32:15 (1305621135)
May 17 01:32:15 client-18 xinetd[3390]: EXIT: shell status=0 pid=24023 duration=0(sec)
May 17 01:32:16 client-18 xinetd[3390]: START: shell pid=24058 from=192.168.4.5
May 17 01:32:16 client-18 rshd[24059]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:32:16 client-18 xinetd[3390]: EXIT: shell status=0 pid=24058 duration=0(sec)
May 17 01:32:16 client-18 xinetd[3390]: START: shell pid=24094 from=192.168.4.5
May 17 01:32:16 client-18 rshd[24095]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:32:16 client-18 xinetd[3390]: EXIT: shell status=0 pid=24094 duration=0(sec)
May 17 01:32:17 client-18 xinetd[3390]: START: shell pid=24131 from=192.168.4.5
May 17 01:32:17 client-18 rshd[24132]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 39j: write, rename, close, stat ======================================================= 01:32:16 \(1305621136\)");echo XXRETCODE:$?'
May 17 01:32:17 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 39j: write, rename, close, stat ======================================================= 01:32:16 (1305621136)
May 17 01:32:17 client-18 xinetd[3390]: EXIT: shell status=0 pid=24131 duration=0(sec)
May 17 01:32:18 client-18 xinetd[3390]: START: shell pid=24170 from=192.168.4.5
May 17 01:32:18 client-18 rshd[24171]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:32:18 client-18 xinetd[3390]: EXIT: shell status=0 pid=24170 duration=0(sec)
May 17 01:32:18 client-18 xinetd[3390]: START: shell pid=24206 from=192.168.4.5
May 17 01:32:18 client-18 rshd[24207]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:32:18 client-18 xinetd[3390]: EXIT: shell status=0 pid=24206 duration=0(sec)
May 17 01:32:18 client-18 xinetd[3390]: START: shell pid=24243 from=192.168.4.5
May 17 01:32:18 client-18 rshd[24244]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 39k: write, utime, close, stat ======================================================== 01:32:18 \(1305621138\)");echo XXRETCODE:$?'
May 17 01:32:19 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 39k: write, utime, close, stat ======================================================== 01:32:18 (1305621138)
May 17 01:32:19 client-18 xinetd[3390]: EXIT: shell status=0 pid=24243 duration=1(sec)
May 17 01:32:20 client-18 xinetd[3390]: START: shell pid=24278 from=192.168.4.5
May 17 01:32:20 client-18 rshd[24279]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:32:20 client-18 xinetd[3390]: EXIT: shell status=0 pid=24278 duration=0(sec)
May 17 01:32:20 client-18 xinetd[3390]: START: shell pid=24314 from=192.168.4.5
May 17 01:32:20 client-18 rshd[24315]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:32:20 client-18 xinetd[3390]: EXIT: shell status=0 pid=24314 duration=0(sec)
May 17 01:32:20 client-18 xinetd[3390]: START: shell pid=24351 from=192.168.4.5
May 17 01:32:20 client-18 rshd[24352]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 39l: directory atime update =========================================================== 01:32:20 \(1305621140\)");echo XXRETCODE:$?'
May 17 01:32:20 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 39l: directory atime update =========================================================== 01:32:20 (1305621140)
May 17 01:32:20 client-18 xinetd[3390]: EXIT: shell status=0 pid=24351 duration=0(sec)
May 17 01:32:27 client-18 xinetd[3390]: START: shell pid=24386 from=192.168.4.5
May 17 01:32:27 client-18 rshd[24387]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:32:27 client-18 xinetd[3390]: EXIT: shell status=0 pid=24386 duration=0(sec)
May 17 01:32:27 client-18 xinetd[3390]: START: shell pid=24422 from=192.168.4.5
May 17 01:32:27 client-18 rshd[24423]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:32:27 client-18 xinetd[3390]: EXIT: shell status=0 pid=24422 duration=0(sec)
May 17 01:32:27 client-18 xinetd[3390]: START: shell pid=24459 from=192.168.4.5
May 17 01:32:27 client-18 rshd[24460]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 39m: test atime and mtime before 1970 ================================================= 01:32:27 \(1305621147\)");echo XXRETCODE:$?'
May 17 01:32:27 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 39m: test atime and mtime before 1970 ================================================= 01:32:27 (1305621147)
May 17 01:32:27 client-18 xinetd[3390]: EXIT: shell status=0 pid=24459 duration=0(sec)
May 17 01:32:30 client-18 xinetd[3390]: START: shell pid=24494 from=192.168.4.5
May 17 01:32:30 client-18 rshd[24495]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:32:30 client-18 xinetd[3390]: EXIT: shell status=0 pid=24494 duration=0(sec)
May 17 01:32:30 client-18 xinetd[3390]: START: shell pid=24530 from=192.168.4.5
May 17 01:32:30 client-18 rshd[24531]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:32:30 client-18 xinetd[3390]: EXIT: shell status=0 pid=24530 duration=0(sec)
May 17 01:32:30 client-18 xinetd[3390]: START: shell pid=24567 from=192.168.4.5
May 17 01:32:30 client-18 rshd[24568]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 40: failed open\(O_TRUNC\) doesn\'t truncate ============================================= 01:32:30 \(1305621150\)");echo XXRETCODE:$?'
May 17 01:32:30 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 40: failed open(O_TRUNC) doesn't truncate ============================================= 01:32:30 (1305621150)
May 17 01:32:30 client-18 xinetd[3390]: EXIT: shell status=0 pid=24567 duration=0(sec)
May 17 01:32:30 client-18 xinetd[3390]: START: shell pid=24602 from=192.168.4.5
May 17 01:32:30 client-18 rshd[24603]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:32:30 client-18 xinetd[3390]: EXIT: shell status=0 pid=24602 duration=0(sec)
May 17 01:32:30 client-18 xinetd[3390]: START: shell pid=24638 from=192.168.4.5
May 17 01:32:30 client-18 rshd[24639]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:32:30 client-18 xinetd[3390]: EXIT: shell status=0 pid=24638 duration=0(sec)
May 17 01:32:31 client-18 xinetd[3390]: START: shell pid=24675 from=192.168.4.5
May 17 01:32:31 client-18 rshd[24676]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 41: test small file write + fstat ===================================================== 01:32:30 \(1305621150\)");echo XXRETCODE:$?'
May 17 01:32:31 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 41: test small file write + fstat ===================================================== 01:32:30 (1305621150)
May 17 01:32:31 client-18 xinetd[3390]: EXIT: shell status=0 pid=24675 duration=0(sec)
May 17 01:32:32 client-18 xinetd[3390]: START: shell pid=24710 from=192.168.4.5
May 17 01:32:32 client-18 rshd[24711]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:32:32 client-18 xinetd[3390]: EXIT: shell status=0 pid=24710 duration=0(sec)
May 17 01:32:32 client-18 xinetd[3390]: START: shell pid=24746 from=192.168.4.5
May 17 01:32:32 client-18 rshd[24747]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:32:32 client-18 xinetd[3390]: EXIT: shell status=0 pid=24746 duration=0(sec)
May 17 01:32:32 client-18 xinetd[3390]: START: shell pid=24783 from=192.168.4.5
May 17 01:32:32 client-18 rshd[24784]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark  SKIP: sanity test_42a skipping ALWAYS excluded test 42a");echo XXRETCODE:$?'
May 17 01:32:32 client-18 kernel: Lustre: DEBUG MARKER: SKIP: sanity test_42a skipping ALWAYS excluded test 42a
May 17 01:32:32 client-18 xinetd[3390]: EXIT: shell status=0 pid=24783 duration=0(sec)
May 17 01:32:32 client-18 xinetd[3390]: START: shell pid=24818 from=192.168.4.5
May 17 01:32:32 client-18 rshd[24819]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark  SKIP: sanity test_42b skipping ALWAYS excluded test 42b");echo XXRETCODE:$?'
May 17 01:32:32 client-18 kernel: Lustre: DEBUG MARKER: SKIP: sanity test_42b skipping ALWAYS excluded test 42b
May 17 01:32:32 client-18 xinetd[3390]: EXIT: shell status=0 pid=24818 duration=0(sec)
May 17 01:32:32 client-18 xinetd[3390]: START: shell pid=24853 from=192.168.4.5
May 17 01:32:32 client-18 rshd[24854]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark  SKIP: sanity test_42c skipping ALWAYS excluded test 42c");echo XXRETCODE:$?'
May 17 01:32:32 client-18 kernel: Lustre: DEBUG MARKER: SKIP: sanity test_42c skipping ALWAYS excluded test 42c
May 17 01:32:32 client-18 xinetd[3390]: EXIT: shell status=0 pid=24853 duration=0(sec)
May 17 01:32:33 client-18 xinetd[3390]: START: shell pid=24888 from=192.168.4.5
May 17 01:32:33 client-18 rshd[24889]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark  SKIP: sanity test_42d skipping ALWAYS excluded test 42d");echo XXRETCODE:$?'
May 17 01:32:33 client-18 kernel: Lustre: DEBUG MARKER: SKIP: sanity test_42d skipping ALWAYS excluded test 42d
May 17 01:32:33 client-18 xinetd[3390]: EXIT: shell status=0 pid=24888 duration=0(sec)
May 17 01:32:33 client-18 xinetd[3390]: START: shell pid=24923 from=192.168.4.5
May 17 01:32:33 client-18 rshd[24924]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 42e: verify sub-RPC writes are not done synchronously ================================= 01:32:33 \(1305621153\)");echo XXRETCODE:$?'
May 17 01:32:33 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 42e: verify sub-RPC writes are not done synchronously ================================= 01:32:33 (1305621153)
May 17 01:32:33 client-18 xinetd[3390]: EXIT: shell status=0 pid=24923 duration=0(sec)
May 17 01:33:20 client-18 xinetd[3390]: START: shell pid=24958 from=192.168.4.5
May 17 01:33:20 client-18 rshd[24959]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:33:20 client-18 xinetd[3390]: EXIT: shell status=0 pid=24958 duration=0(sec)
May 17 01:33:20 client-18 xinetd[3390]: START: shell pid=24994 from=192.168.4.5
May 17 01:33:20 client-18 rshd[24995]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:33:20 client-18 xinetd[3390]: EXIT: shell status=0 pid=24994 duration=0(sec)
May 17 01:33:20 client-18 xinetd[3390]: START: shell pid=25031 from=192.168.4.5
May 17 01:33:20 client-18 rshd[25032]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 43: execution of file opened for write should return -ETXTBSY ========================= 01:33:20 \(1305621200\)");echo XXRETCODE:$?'
May 17 01:33:20 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 43: execution of file opened for write should return -ETXTBSY ========================= 01:33:20 (1305621200)
May 17 01:33:20 client-18 xinetd[3390]: EXIT: shell status=0 pid=25031 duration=0(sec)
May 17 01:33:21 client-18 xinetd[3390]: START: shell pid=25066 from=192.168.4.5
May 17 01:33:21 client-18 rshd[25067]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:33:21 client-18 xinetd[3390]: EXIT: shell status=0 pid=25066 duration=0(sec)
May 17 01:33:21 client-18 xinetd[3390]: START: shell pid=25102 from=192.168.4.5
May 17 01:33:21 client-18 rshd[25103]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:33:21 client-18 xinetd[3390]: EXIT: shell status=0 pid=25102 duration=0(sec)
May 17 01:33:22 client-18 xinetd[3390]: START: shell pid=25139 from=192.168.4.5
May 17 01:33:22 client-18 rshd[25140]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 43a: open\(RDWR\) of file being executed should return -ETXTBSY ========================= 01:33:21 \(1305621201\)");echo XXRETCODE:$?'
May 17 01:33:22 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 43a: open(RDWR) of file being executed should return -ETXTBSY ========================= 01:33:21 (1305621201)
May 17 01:33:22 client-18 xinetd[3390]: EXIT: shell status=0 pid=25139 duration=0(sec)
May 17 01:33:22 client-18 xinetd[3390]: START: shell pid=25174 from=192.168.4.5
May 17 01:33:22 client-18 rshd[25175]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:33:22 client-18 xinetd[3390]: EXIT: shell status=0 pid=25174 duration=0(sec)
May 17 01:33:22 client-18 xinetd[3390]: START: shell pid=25210 from=192.168.4.5
May 17 01:33:22 client-18 rshd[25211]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:33:22 client-18 xinetd[3390]: EXIT: shell status=0 pid=25210 duration=0(sec)
May 17 01:33:22 client-18 xinetd[3390]: START: shell pid=25247 from=192.168.4.5
May 17 01:33:22 client-18 rshd[25248]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 43b: truncate of file being executed should return -ETXTBSY =========================== 01:33:22 \(1305621202\)");echo XXRETCODE:$?'
May 17 01:33:22 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 43b: truncate of file being executed should return -ETXTBSY =========================== 01:33:22 (1305621202)
May 17 01:33:22 client-18 xinetd[3390]: EXIT: shell status=0 pid=25247 duration=0(sec)
May 17 01:33:22 client-18 xinetd[3390]: START: shell pid=25282 from=192.168.4.5
May 17 01:33:22 client-18 rshd[25283]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:33:22 client-18 xinetd[3390]: EXIT: shell status=0 pid=25282 duration=0(sec)
May 17 01:33:23 client-18 xinetd[3390]: START: shell pid=25318 from=192.168.4.5
May 17 01:33:23 client-18 rshd[25319]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:33:23 client-18 xinetd[3390]: EXIT: shell status=0 pid=25318 duration=0(sec)
May 17 01:33:23 client-18 xinetd[3390]: START: shell pid=25355 from=192.168.4.5
May 17 01:33:23 client-18 rshd[25356]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 43c: md5sum of copy into lustre======================================================== 01:33:23 \(1305621203\)");echo XXRETCODE:$?'
May 17 01:33:23 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 43c: md5sum of copy into lustre======================================================== 01:33:23 (1305621203)
May 17 01:33:23 client-18 xinetd[3390]: EXIT: shell status=0 pid=25355 duration=0(sec)
May 17 01:33:23 client-18 xinetd[3390]: START: shell pid=25390 from=192.168.4.5
May 17 01:33:23 client-18 rshd[25391]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:33:23 client-18 xinetd[3390]: EXIT: shell status=0 pid=25390 duration=0(sec)
May 17 01:33:23 client-18 xinetd[3390]: START: shell pid=25426 from=192.168.4.5
May 17 01:33:23 client-18 rshd[25427]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:33:23 client-18 xinetd[3390]: EXIT: shell status=0 pid=25426 duration=0(sec)
May 17 01:33:23 client-18 xinetd[3390]: START: shell pid=25463 from=192.168.4.5
May 17 01:33:23 client-18 rshd[25464]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 44: zero length read from a sparse stripe ============================================= 01:33:23 \(1305621203\)");echo XXRETCODE:$?'
May 17 01:33:23 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 44: zero length read from a sparse stripe ============================================= 01:33:23 (1305621203)
May 17 01:33:23 client-18 xinetd[3390]: EXIT: shell status=0 pid=25463 duration=0(sec)
May 17 01:33:24 client-18 xinetd[3390]: START: shell pid=25498 from=192.168.4.5
May 17 01:33:24 client-18 rshd[25499]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:33:24 client-18 xinetd[3390]: EXIT: shell status=0 pid=25498 duration=0(sec)
May 17 01:33:24 client-18 xinetd[3390]: START: shell pid=25534 from=192.168.4.5
May 17 01:33:24 client-18 rshd[25535]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:33:24 client-18 xinetd[3390]: EXIT: shell status=0 pid=25534 duration=0(sec)
May 17 01:33:24 client-18 xinetd[3390]: START: shell pid=25571 from=192.168.4.5
May 17 01:33:24 client-18 rshd[25572]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 44a: test sparse pwrite =============================================================== 01:33:24 \(1305621204\)");echo XXRETCODE:$?'
May 17 01:33:24 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 44a: test sparse pwrite =============================================================== 01:33:24 (1305621204)
May 17 01:33:24 client-18 xinetd[3390]: EXIT: shell status=0 pid=25571 duration=0(sec)
May 17 01:33:25 client-18 xinetd[3390]: START: shell pid=25606 from=192.168.4.5
May 17 01:33:25 client-18 rshd[25607]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:33:25 client-18 xinetd[3390]: EXIT: shell status=0 pid=25606 duration=0(sec)
May 17 01:33:25 client-18 xinetd[3390]: START: shell pid=25642 from=192.168.4.5
May 17 01:33:25 client-18 rshd[25643]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:33:25 client-18 xinetd[3390]: EXIT: shell status=0 pid=25642 duration=0(sec)
May 17 01:33:25 client-18 xinetd[3390]: START: shell pid=25679 from=192.168.4.5
May 17 01:33:25 client-18 rshd[25680]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark  SKIP: sanity test_45 skipping ALWAYS excluded test 45");echo XXRETCODE:$?'
May 17 01:33:25 client-18 kernel: Lustre: DEBUG MARKER: SKIP: sanity test_45 skipping ALWAYS excluded test 45
May 17 01:33:25 client-18 xinetd[3390]: EXIT: shell status=0 pid=25679 duration=0(sec)
May 17 01:33:25 client-18 xinetd[3390]: START: shell pid=25714 from=192.168.4.5
May 17 01:33:25 client-18 rshd[25715]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 46: dirtying a previously written page ================================================ 01:33:25 \(1305621205\)");echo XXRETCODE:$?'
May 17 01:33:26 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 46: dirtying a previously written page ================================================ 01:33:25 (1305621205)
May 17 01:33:26 client-18 xinetd[3390]: EXIT: shell status=0 pid=25714 duration=1(sec)
May 17 01:33:26 client-18 xinetd[3390]: START: shell pid=25749 from=192.168.4.5
May 17 01:33:26 client-18 rshd[25750]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:33:26 client-18 xinetd[3390]: EXIT: shell status=0 pid=25749 duration=0(sec)
May 17 01:33:26 client-18 xinetd[3390]: START: shell pid=25785 from=192.168.4.5
May 17 01:33:26 client-18 rshd[25786]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:33:26 client-18 xinetd[3390]: EXIT: shell status=0 pid=25785 duration=0(sec)
May 17 01:33:26 client-18 xinetd[3390]: START: shell pid=25822 from=192.168.4.5
May 17 01:33:26 client-18 rshd[25823]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 48a: Access renamed working dir \(should return errors\)================================= 01:33:26 \(1305621206\)");echo XXRETCODE:$?'
May 17 01:33:26 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 48a: Access renamed working dir (should return errors)================================= 01:33:26 (1305621206)
May 17 01:33:26 client-18 xinetd[3390]: EXIT: shell status=0 pid=25822 duration=0(sec)
May 17 01:33:26 client-18 xinetd[3390]: START: shell pid=25857 from=192.168.4.5
May 17 01:33:26 client-18 rshd[25858]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:33:26 client-18 xinetd[3390]: EXIT: shell status=0 pid=25857 duration=0(sec)
May 17 01:33:26 client-18 xinetd[3390]: START: shell pid=25893 from=192.168.4.5
May 17 01:33:26 client-18 rshd[25894]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:33:27 client-18 xinetd[3390]: EXIT: shell status=0 pid=25893 duration=1(sec)
May 17 01:33:27 client-18 xinetd[3390]: START: shell pid=25930 from=192.168.4.5
May 17 01:33:27 client-18 rshd[25931]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 48b: Access removed working dir \(should return errors\)================================= 01:33:27 \(1305621207\)");echo XXRETCODE:$?'
May 17 01:33:27 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 48b: Access removed working dir (should return errors)================================= 01:33:27 (1305621207)
May 17 01:33:27 client-18 xinetd[3390]: EXIT: shell status=0 pid=25930 duration=0(sec)
May 17 01:33:27 client-18 xinetd[3390]: START: shell pid=25965 from=192.168.4.5
May 17 01:33:27 client-18 rshd[25966]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:33:27 client-18 xinetd[3390]: EXIT: shell status=0 pid=25965 duration=0(sec)
May 17 01:33:27 client-18 xinetd[3390]: START: shell pid=26001 from=192.168.4.5
May 17 01:33:27 client-18 rshd[26002]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:33:28 client-18 xinetd[3390]: EXIT: shell status=0 pid=26001 duration=1(sec)
May 17 01:33:28 client-18 xinetd[3390]: START: shell pid=26038 from=192.168.4.5
May 17 01:33:28 client-18 rshd[26039]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 48c: Access removed working subdir \(should return errors\) ============================= 01:33:28 \(1305621208\)");echo XXRETCODE:$?'
May 17 01:33:28 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 48c: Access removed working subdir (should return errors) ============================= 01:33:28 (1305621208)
May 17 01:33:28 client-18 xinetd[3390]: EXIT: shell status=0 pid=26038 duration=0(sec)
May 17 01:33:28 client-18 xinetd[3390]: START: shell pid=26073 from=192.168.4.5
May 17 01:33:28 client-18 rshd[26074]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:33:28 client-18 xinetd[3390]: EXIT: shell status=0 pid=26073 duration=0(sec)
May 17 01:33:29 client-18 xinetd[3390]: START: shell pid=26109 from=192.168.4.5
May 17 01:33:29 client-18 rshd[26110]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:33:29 client-18 xinetd[3390]: EXIT: shell status=0 pid=26109 duration=0(sec)
May 17 01:33:29 client-18 xinetd[3390]: START: shell pid=26146 from=192.168.4.5
May 17 01:33:29 client-18 rshd[26147]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 48d: Access removed parent subdir \(should return errors\) ============================== 01:33:29 \(1305621209\)");echo XXRETCODE:$?'
May 17 01:33:29 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 48d: Access removed parent subdir (should return errors) ============================== 01:33:29 (1305621209)
May 17 01:33:29 client-18 xinetd[3390]: EXIT: shell status=0 pid=26146 duration=0(sec)
May 17 01:33:30 client-18 xinetd[3390]: START: shell pid=26181 from=192.168.4.5
May 17 01:33:30 client-18 rshd[26182]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:33:30 client-18 xinetd[3390]: EXIT: shell status=0 pid=26181 duration=0(sec)
May 17 01:33:30 client-18 xinetd[3390]: START: shell pid=26217 from=192.168.4.5
May 17 01:33:30 client-18 rshd[26218]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:33:30 client-18 xinetd[3390]: EXIT: shell status=0 pid=26217 duration=0(sec)
May 17 01:33:30 client-18 xinetd[3390]: START: shell pid=26254 from=192.168.4.5
May 17 01:33:30 client-18 rshd[26255]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 48e: Access to recreated parent subdir \(should return errors\) ========================= 01:33:30 \(1305621210\)");echo XXRETCODE:$?'
May 17 01:33:30 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 48e: Access to recreated parent subdir (should return errors) ========================= 01:33:30 (1305621210)
May 17 01:33:30 client-18 xinetd[3390]: EXIT: shell status=0 pid=26254 duration=0(sec)
May 17 01:33:30 client-18 xinetd[3390]: START: shell pid=26289 from=192.168.4.5
May 17 01:33:30 client-18 rshd[26290]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:33:30 client-18 xinetd[3390]: EXIT: shell status=0 pid=26289 duration=0(sec)
May 17 01:33:30 client-18 xinetd[3390]: START: shell pid=26325 from=192.168.4.5
May 17 01:33:30 client-18 rshd[26326]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:33:30 client-18 xinetd[3390]: EXIT: shell status=0 pid=26325 duration=0(sec)
May 17 01:33:30 client-18 xinetd[3390]: START: shell pid=26362 from=192.168.4.5
May 17 01:33:30 client-18 rshd[26363]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 50: special situations: \/proc symlinks  =============================================== 01:33:30 \(1305621210\)");echo XXRETCODE:$?'
May 17 01:33:31 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 50: special situations: /proc symlinks =============================================== 01:33:30 (1305621210)
May 17 01:33:31 client-18 xinetd[3390]: EXIT: shell status=0 pid=26362 duration=1(sec)
May 17 01:33:31 client-18 xinetd[3390]: START: shell pid=26397 from=192.168.4.5
May 17 01:33:31 client-18 rshd[26398]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:33:31 client-18 xinetd[3390]: EXIT: shell status=0 pid=26397 duration=0(sec)
May 17 01:33:31 client-18 xinetd[3390]: START: shell pid=26433 from=192.168.4.5
May 17 01:33:31 client-18 rshd[26434]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:33:31 client-18 xinetd[3390]: EXIT: shell status=0 pid=26433 duration=0(sec)
May 17 01:33:31 client-18 xinetd[3390]: START: shell pid=26470 from=192.168.4.5
May 17 01:33:31 client-18 rshd[26471]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 51a: special situations: split htree with empty entry ================================= 01:33:31 \(1305621211\)");echo XXRETCODE:$?'
May 17 01:33:31 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 51a: special situations: split htree with empty entry ================================= 01:33:31 (1305621211)
May 17 01:33:31 client-18 xinetd[3390]: EXIT: shell status=0 pid=26470 duration=0(sec)
May 17 01:33:31 client-18 xinetd[3390]: START: shell pid=26505 from=192.168.4.5
May 17 01:33:32 client-18 rshd[26506]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:33:32 client-18 xinetd[3390]: EXIT: shell status=0 pid=26505 duration=1(sec)
May 17 01:33:32 client-18 xinetd[3390]: START: shell pid=26541 from=192.168.4.5
May 17 01:33:32 client-18 rshd[26542]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:33:32 client-18 xinetd[3390]: EXIT: shell status=0 pid=26541 duration=0(sec)
May 17 01:33:32 client-18 xinetd[3390]: START: shell pid=26578 from=192.168.4.5
May 17 01:33:32 client-18 rshd[26579]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 51b: mkdir ...\/t-0 --- ...\/t-70 ======================================================= 01:33:32 \(1305621212\)");echo XXRETCODE:$?'
May 17 01:33:32 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 51b: mkdir .../t-0 --- .../t-70 ======================================================= 01:33:32 (1305621212)
May 17 01:33:32 client-18 xinetd[3390]: EXIT: shell status=0 pid=26578 duration=0(sec)
May 17 01:33:32 client-18 xinetd[3390]: START: shell pid=26613 from=192.168.4.5
May 17 01:33:32 client-18 rshd[26614]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:33:32 client-18 xinetd[3390]: EXIT: shell status=0 pid=26613 duration=0(sec)
May 17 01:33:32 client-18 xinetd[3390]: START: shell pid=26649 from=192.168.4.5
May 17 01:33:32 client-18 rshd[26650]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:33:32 client-18 xinetd[3390]: EXIT: shell status=0 pid=26649 duration=0(sec)
May 17 01:33:32 client-18 xinetd[3390]: START: shell pid=26690 from=192.168.4.5
May 17 01:33:32 client-18 rshd[26691]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 51bb: mkdir createmany CMD 1  ========================================================= 01:33:32 \(1305621212\)");echo XXRETCODE:$?'
May 17 01:33:33 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 51bb: mkdir createmany CMD 1 ========================================================= 01:33:32 (1305621212)
May 17 01:33:33 client-18 xinetd[3390]: EXIT: shell status=0 pid=26690 duration=1(sec)
May 17 01:33:33 client-18 xinetd[3390]: START: shell pid=26725 from=192.168.4.5
May 17 01:33:33 client-18 rshd[26726]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark  SKIP: sanity test_51bb needs \>= 2 MDTs");echo XXRETCODE:$?'
May 17 01:33:33 client-18 kernel: Lustre: DEBUG MARKER: SKIP: sanity test_51bb needs >= 2 MDTs
May 17 01:33:33 client-18 xinetd[3390]: EXIT: shell status=0 pid=26725 duration=0(sec)
May 17 01:33:33 client-18 xinetd[3390]: START: shell pid=26760 from=192.168.4.5
May 17 01:33:33 client-18 rshd[26761]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:33:33 client-18 xinetd[3390]: EXIT: shell status=0 pid=26760 duration=0(sec)
May 17 01:33:33 client-18 xinetd[3390]: START: shell pid=26796 from=192.168.4.5
May 17 01:33:33 client-18 rshd[26797]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:33:33 client-18 xinetd[3390]: EXIT: shell status=0 pid=26796 duration=0(sec)
May 17 01:33:33 client-18 xinetd[3390]: START: shell pid=26833 from=192.168.4.5
May 17 01:33:33 client-18 rshd[26834]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 51c: rmdir ...\/t-0 --- ...\/t-70 ======================================================= 01:33:33 \(1305621213\)");echo XXRETCODE:$?'
May 17 01:33:33 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 51c: rmdir .../t-0 --- .../t-70 ======================================================= 01:33:33 (1305621213)
May 17 01:33:33 client-18 xinetd[3390]: EXIT: shell status=0 pid=26833 duration=0(sec)
May 17 01:33:33 client-18 xinetd[3390]: START: shell pid=26868 from=192.168.4.5
May 17 01:33:34 client-18 rshd[26869]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:33:34 client-18 xinetd[3390]: EXIT: shell status=0 pid=26868 duration=1(sec)
May 17 01:33:34 client-18 xinetd[3390]: START: shell pid=26904 from=192.168.4.5
May 17 01:33:34 client-18 rshd[26905]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:33:34 client-18 xinetd[3390]: EXIT: shell status=0 pid=26904 duration=0(sec)
May 17 01:33:34 client-18 xinetd[3390]: START: shell pid=26941 from=192.168.4.5
May 17 01:33:34 client-18 rshd[26942]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark  SKIP: sanity test_51d skipping ALWAYS excluded test 51d");echo XXRETCODE:$?'
May 17 01:33:34 client-18 kernel: Lustre: DEBUG MARKER: SKIP: sanity test_51d skipping ALWAYS excluded test 51d
May 17 01:33:34 client-18 xinetd[3390]: EXIT: shell status=0 pid=26941 duration=0(sec)
May 17 01:33:34 client-18 xinetd[3390]: START: shell pid=26976 from=192.168.4.5
May 17 01:33:34 client-18 rshd[26977]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 52a: append-only flag test \(should return errors\) ===================================== 01:33:34 \(1305621214\)");echo XXRETCODE:$?'
May 17 01:33:34 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 52a: append-only flag test (should return errors) ===================================== 01:33:34 (1305621214)
May 17 01:33:34 client-18 xinetd[3390]: EXIT: shell status=0 pid=26976 duration=0(sec)
May 17 01:33:34 client-18 xinetd[3390]: START: shell pid=27011 from=192.168.4.5
May 17 01:33:34 client-18 rshd[27012]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:33:34 client-18 xinetd[3390]: EXIT: shell status=0 pid=27011 duration=0(sec)
May 17 01:33:34 client-18 xinetd[3390]: START: shell pid=27047 from=192.168.4.5
May 17 01:33:34 client-18 rshd[27048]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:33:35 client-18 xinetd[3390]: EXIT: shell status=0 pid=27047 duration=1(sec)
May 17 01:33:35 client-18 xinetd[3390]: START: shell pid=27084 from=192.168.4.5
May 17 01:33:35 client-18 rshd[27085]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 52b: immutable flag test \(should return errors\) ======================================= 01:33:35 \(1305621215\)");echo XXRETCODE:$?'
May 17 01:33:35 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 52b: immutable flag test (should return errors) ======================================= 01:33:35 (1305621215)
May 17 01:33:35 client-18 xinetd[3390]: EXIT: shell status=0 pid=27084 duration=0(sec)
May 17 01:33:35 client-18 xinetd[3390]: START: shell pid=27119 from=192.168.4.5
May 17 01:33:35 client-18 rshd[27120]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:33:35 client-18 xinetd[3390]: EXIT: shell status=0 pid=27119 duration=0(sec)
May 17 01:33:35 client-18 xinetd[3390]: START: shell pid=27155 from=192.168.4.5
May 17 01:33:35 client-18 rshd[27156]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:33:35 client-18 xinetd[3390]: EXIT: shell status=0 pid=27155 duration=0(sec)
May 17 01:33:35 client-18 xinetd[3390]: START: shell pid=27192 from=192.168.4.5
May 17 01:33:35 client-18 rshd[27193]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 53: verify that MDS and OSTs agree on pre-creation ==================================== 01:33:35 \(1305621215\)");echo XXRETCODE:$?'
May 17 01:33:35 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 53: verify that MDS and OSTs agree on pre-creation ==================================== 01:33:35 (1305621215)
May 17 01:33:35 client-18 xinetd[3390]: EXIT: shell status=0 pid=27192 duration=0(sec)
May 17 01:33:36 client-18 ntpd[3401]: kernel time sync enabled 0001
May 17 01:33:37 client-18 xinetd[3390]: START: shell pid=27227 from=192.168.4.5
May 17 01:33:37 client-18 rshd[27228]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:33:38 client-18 xinetd[3390]: EXIT: shell status=0 pid=27227 duration=1(sec)
May 17 01:33:38 client-18 xinetd[3390]: START: shell pid=27263 from=192.168.4.5
May 17 01:33:38 client-18 rshd[27264]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:33:38 client-18 xinetd[3390]: EXIT: shell status=0 pid=27263 duration=0(sec)
May 17 01:33:38 client-18 xinetd[3390]: START: shell pid=27300 from=192.168.4.5
May 17 01:33:38 client-18 rshd[27301]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 54a: unix domain socket test ========================================================== 01:33:38 \(1305621218\)");echo XXRETCODE:$?'
May 17 01:33:38 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 54a: unix domain socket test ========================================================== 01:33:38 (1305621218)
May 17 01:33:38 client-18 xinetd[3390]: EXIT: shell status=0 pid=27300 duration=0(sec)
May 17 01:33:38 client-18 xinetd[3390]: START: shell pid=27335 from=192.168.4.5
May 17 01:33:38 client-18 rshd[27336]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:33:38 client-18 xinetd[3390]: EXIT: shell status=0 pid=27335 duration=0(sec)
May 17 01:33:38 client-18 xinetd[3390]: START: shell pid=27371 from=192.168.4.5
May 17 01:33:38 client-18 rshd[27372]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:33:38 client-18 xinetd[3390]: EXIT: shell status=0 pid=27371 duration=0(sec)
May 17 01:33:38 client-18 xinetd[3390]: START: shell pid=27408 from=192.168.4.5
May 17 01:33:38 client-18 rshd[27409]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 54b: char device works in lustre ====================================================== 01:33:38 \(1305621218\)");echo XXRETCODE:$?'
May 17 01:33:39 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 54b: char device works in lustre ====================================================== 01:33:38 (1305621218)
May 17 01:33:39 client-18 xinetd[3390]: EXIT: shell status=0 pid=27408 duration=1(sec)
May 17 01:33:39 client-18 xinetd[3390]: START: shell pid=27443 from=192.168.4.5
May 17 01:33:39 client-18 rshd[27444]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:33:39 client-18 xinetd[3390]: EXIT: shell status=0 pid=27443 duration=0(sec)
May 17 01:33:39 client-18 xinetd[3390]: START: shell pid=27479 from=192.168.4.5
May 17 01:33:39 client-18 rshd[27480]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:33:39 client-18 xinetd[3390]: EXIT: shell status=0 pid=27479 duration=0(sec)
May 17 01:33:39 client-18 xinetd[3390]: START: shell pid=27516 from=192.168.4.5
May 17 01:33:39 client-18 rshd[27517]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 54c: block device works in lustre ===================================================== 01:33:39 \(1305621219\)");echo XXRETCODE:$?'
May 17 01:33:39 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 54c: block device works in lustre ===================================================== 01:33:39 (1305621219)
May 17 01:33:39 client-18 xinetd[3390]: EXIT: shell status=0 pid=27516 duration=0(sec)
May 17 01:33:40 client-18 xinetd[3390]: START: shell pid=27551 from=192.168.4.5
May 17 01:33:40 client-18 rshd[27552]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:33:40 client-18 xinetd[3390]: EXIT: shell status=0 pid=27551 duration=0(sec)
May 17 01:33:40 client-18 xinetd[3390]: START: shell pid=27587 from=192.168.4.5
May 17 01:33:40 client-18 rshd[27588]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:33:40 client-18 xinetd[3390]: EXIT: shell status=0 pid=27587 duration=0(sec)
May 17 01:33:40 client-18 xinetd[3390]: START: shell pid=27624 from=192.168.4.5
May 17 01:33:40 client-18 rshd[27625]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 54d: fifo device works in lustre ====================================================== 01:33:40 \(1305621220\)");echo XXRETCODE:$?'
May 17 01:33:40 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 54d: fifo device works in lustre ====================================================== 01:33:40 (1305621220)
May 17 01:33:40 client-18 xinetd[3390]: EXIT: shell status=0 pid=27624 duration=0(sec)
May 17 01:33:40 client-18 xinetd[3390]: START: shell pid=27659 from=192.168.4.5
May 17 01:33:40 client-18 rshd[27660]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:33:40 client-18 xinetd[3390]: EXIT: shell status=0 pid=27659 duration=0(sec)
May 17 01:33:40 client-18 xinetd[3390]: START: shell pid=27695 from=192.168.4.5
May 17 01:33:40 client-18 rshd[27696]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:33:40 client-18 xinetd[3390]: EXIT: shell status=0 pid=27695 duration=0(sec)
May 17 01:33:40 client-18 xinetd[3390]: START: shell pid=27732 from=192.168.4.5
May 17 01:33:40 client-18 rshd[27733]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 54e: console\/tty device works in lustre =============================================== 01:33:40 \(1305621220\)");echo XXRETCODE:$?'
May 17 01:33:41 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 54e: console/tty device works in lustre =============================================== 01:33:40 (1305621220)
May 17 01:33:41 client-18 xinetd[3390]: EXIT: shell status=0 pid=27732 duration=1(sec)
May 17 01:33:41 client-18 xinetd[3390]: START: shell pid=27767 from=192.168.4.5
May 17 01:33:41 client-18 rshd[27768]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:33:41 client-18 xinetd[3390]: EXIT: shell status=0 pid=27767 duration=0(sec)
May 17 01:33:41 client-18 xinetd[3390]: START: shell pid=27803 from=192.168.4.5
May 17 01:33:41 client-18 rshd[27804]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:33:41 client-18 xinetd[3390]: EXIT: shell status=0 pid=27803 duration=0(sec)
May 17 01:33:41 client-18 xinetd[3390]: START: shell pid=27840 from=192.168.4.5
May 17 01:33:41 client-18 rshd[27841]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 56a: check lfs getstripe ============================================================== 01:33:41 \(1305621221\)");echo XXRETCODE:$?'
May 17 01:33:41 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 56a: check lfs getstripe ============================================================== 01:33:41 (1305621221)
May 17 01:33:41 client-18 xinetd[3390]: EXIT: shell status=0 pid=27840 duration=0(sec)
May 17 01:33:41 client-18 xinetd[3390]: START: shell pid=27876 from=192.168.4.5
May 17 01:33:41 client-18 rshd[27877]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:33:41 client-18 xinetd[3390]: EXIT: shell status=0 pid=27876 duration=0(sec)
May 17 01:33:41 client-18 xinetd[3390]: START: shell pid=27912 from=192.168.4.5
May 17 01:33:41 client-18 rshd[27913]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:33:42 client-18 xinetd[3390]: EXIT: shell status=0 pid=27912 duration=1(sec)
May 17 01:33:42 client-18 xinetd[3390]: START: shell pid=27949 from=192.168.4.5
May 17 01:33:42 client-18 rshd[27950]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 56g: check lfs find -name ============================================================= 01:33:42 \(1305621222\)");echo XXRETCODE:$?'
May 17 01:33:42 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 56g: check lfs find -name ============================================================= 01:33:42 (1305621222)
May 17 01:33:42 client-18 xinetd[3390]: EXIT: shell status=0 pid=27949 duration=0(sec)
May 17 01:33:42 client-18 xinetd[3390]: START: shell pid=27984 from=192.168.4.5
May 17 01:33:42 client-18 rshd[27985]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:33:42 client-18 xinetd[3390]: EXIT: shell status=0 pid=27984 duration=0(sec)
May 17 01:33:42 client-18 xinetd[3390]: START: shell pid=28020 from=192.168.4.5
May 17 01:33:42 client-18 rshd[28021]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:33:42 client-18 xinetd[3390]: EXIT: shell status=0 pid=28020 duration=0(sec)
May 17 01:33:42 client-18 xinetd[3390]: START: shell pid=28057 from=192.168.4.5
May 17 01:33:42 client-18 rshd[28058]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 56h: check lfs find ! -name =========================================================== 01:33:42 \(1305621222\)");echo XXRETCODE:$?'
May 17 01:33:42 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 56h: check lfs find ! -name =========================================================== 01:33:42 (1305621222)
May 17 01:33:42 client-18 xinetd[3390]: EXIT: shell status=0 pid=28057 duration=0(sec)
May 17 01:33:42 client-18 xinetd[3390]: START: shell pid=28092 from=192.168.4.5
May 17 01:33:43 client-18 rshd[28093]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:33:43 client-18 xinetd[3390]: EXIT: shell status=0 pid=28092 duration=1(sec)
May 17 01:33:43 client-18 xinetd[3390]: START: shell pid=28128 from=192.168.4.5
May 17 01:33:43 client-18 rshd[28129]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:33:43 client-18 xinetd[3390]: EXIT: shell status=0 pid=28128 duration=0(sec)
May 17 01:33:43 client-18 xinetd[3390]: START: shell pid=28165 from=192.168.4.5
May 17 01:33:43 client-18 rshd[28166]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 56i: check \'lfs find -ost UUID\' skips directories ===================================== 01:33:43 \(1305621223\)");echo XXRETCODE:$?'
May 17 01:33:43 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 56i: check 'lfs find -ost UUID' skips directories ===================================== 01:33:43 (1305621223)
May 17 01:33:43 client-18 xinetd[3390]: EXIT: shell status=0 pid=28165 duration=0(sec)
May 17 01:33:43 client-18 xinetd[3390]: START: shell pid=28200 from=192.168.4.5
May 17 01:33:43 client-18 rshd[28201]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:33:43 client-18 xinetd[3390]: EXIT: shell status=0 pid=28200 duration=0(sec)
May 17 01:33:43 client-18 xinetd[3390]: START: shell pid=28236 from=192.168.4.5
May 17 01:33:43 client-18 rshd[28237]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:33:43 client-18 xinetd[3390]: EXIT: shell status=0 pid=28236 duration=0(sec)
May 17 01:33:43 client-18 xinetd[3390]: START: shell pid=28273 from=192.168.4.5
May 17 01:33:43 client-18 rshd[28274]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 56j: check lfs find -type d =========================================================== 01:33:43 \(1305621223\)");echo XXRETCODE:$?'
May 17 01:33:43 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 56j: check lfs find -type d =========================================================== 01:33:43 (1305621223)
May 17 01:33:43 client-18 xinetd[3390]: EXIT: shell status=0 pid=28273 duration=0(sec)
May 17 01:33:44 client-18 xinetd[3390]: START: shell pid=28308 from=192.168.4.5
May 17 01:33:44 client-18 rshd[28309]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:33:44 client-18 xinetd[3390]: EXIT: shell status=0 pid=28308 duration=0(sec)
May 17 01:33:44 client-18 xinetd[3390]: START: shell pid=28344 from=192.168.4.5
May 17 01:33:44 client-18 rshd[28345]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:33:44 client-18 xinetd[3390]: EXIT: shell status=0 pid=28344 duration=0(sec)
May 17 01:33:44 client-18 xinetd[3390]: START: shell pid=28381 from=192.168.4.5
May 17 01:33:44 client-18 rshd[28382]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 56k: check lfs find -type f =========================================================== 01:33:44 \(1305621224\)");echo XXRETCODE:$?'
May 17 01:33:44 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 56k: check lfs find -type f =========================================================== 01:33:44 (1305621224)
May 17 01:33:44 client-18 xinetd[3390]: EXIT: shell status=0 pid=28381 duration=0(sec)
May 17 01:33:44 client-18 xinetd[3390]: START: shell pid=28416 from=192.168.4.5
May 17 01:33:44 client-18 rshd[28417]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:33:44 client-18 xinetd[3390]: EXIT: shell status=0 pid=28416 duration=0(sec)
May 17 01:33:44 client-18 xinetd[3390]: START: shell pid=28452 from=192.168.4.5
May 17 01:33:44 client-18 rshd[28453]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:33:44 client-18 xinetd[3390]: EXIT: shell status=0 pid=28452 duration=0(sec)
May 17 01:33:45 client-18 xinetd[3390]: START: shell pid=28489 from=192.168.4.5
May 17 01:33:45 client-18 rshd[28490]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 56l: check lfs find -type b =========================================================== 01:33:44 \(1305621224\)");echo XXRETCODE:$?'
May 17 01:33:45 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 56l: check lfs find -type b =========================================================== 01:33:44 (1305621224)
May 17 01:33:45 client-18 xinetd[3390]: EXIT: shell status=0 pid=28489 duration=0(sec)
May 17 01:33:45 client-18 xinetd[3390]: START: shell pid=28524 from=192.168.4.5
May 17 01:33:45 client-18 rshd[28525]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:33:45 client-18 xinetd[3390]: EXIT: shell status=0 pid=28524 duration=0(sec)
May 17 01:33:45 client-18 xinetd[3390]: START: shell pid=28560 from=192.168.4.5
May 17 01:33:45 client-18 rshd[28561]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:33:45 client-18 xinetd[3390]: EXIT: shell status=0 pid=28560 duration=0(sec)
May 17 01:33:45 client-18 xinetd[3390]: START: shell pid=28597 from=192.168.4.5
May 17 01:33:45 client-18 rshd[28598]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 56m: check lfs find -type c =========================================================== 01:33:45 \(1305621225\)");echo XXRETCODE:$?'
May 17 01:33:45 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 56m: check lfs find -type c =========================================================== 01:33:45 (1305621225)
May 17 01:33:45 client-18 xinetd[3390]: EXIT: shell status=0 pid=28597 duration=0(sec)
May 17 01:33:45 client-18 xinetd[3390]: START: shell pid=28632 from=192.168.4.5
May 17 01:33:45 client-18 rshd[28633]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:33:45 client-18 xinetd[3390]: EXIT: shell status=0 pid=28632 duration=0(sec)
May 17 01:33:45 client-18 xinetd[3390]: START: shell pid=28668 from=192.168.4.5
May 17 01:33:45 client-18 rshd[28669]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:33:45 client-18 xinetd[3390]: EXIT: shell status=0 pid=28668 duration=0(sec)
May 17 01:33:46 client-18 xinetd[3390]: START: shell pid=28705 from=192.168.4.5
May 17 01:33:46 client-18 rshd[28706]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 56n: check lfs find -type l =========================================================== 01:33:45 \(1305621225\)");echo XXRETCODE:$?'
May 17 01:33:46 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 56n: check lfs find -type l =========================================================== 01:33:45 (1305621225)
May 17 01:33:46 client-18 xinetd[3390]: EXIT: shell status=0 pid=28705 duration=0(sec)
May 17 01:33:46 client-18 kernel: Lustre: Increasing default stripe size to min 1048576
May 17 01:33:46 client-18 xinetd[3390]: START: shell pid=28742 from=192.168.4.5
May 17 01:33:46 client-18 rshd[28743]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:33:46 client-18 xinetd[3390]: EXIT: shell status=0 pid=28742 duration=0(sec)
May 17 01:33:46 client-18 xinetd[3390]: START: shell pid=28778 from=192.168.4.5
May 17 01:33:46 client-18 rshd[28779]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:33:46 client-18 xinetd[3390]: EXIT: shell status=0 pid=28778 duration=0(sec)
May 17 01:33:46 client-18 xinetd[3390]: START: shell pid=28815 from=192.168.4.5
May 17 01:33:46 client-18 rshd[28816]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 56o: check lfs find -mtime for old files ============================================== 01:33:46 \(1305621226\)");echo XXRETCODE:$?'
May 17 01:33:46 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 56o: check lfs find -mtime for old files ============================================== 01:33:46 (1305621226)
May 17 01:33:46 client-18 xinetd[3390]: EXIT: shell status=0 pid=28815 duration=0(sec)
May 17 01:33:46 client-18 xinetd[3390]: START: shell pid=28850 from=192.168.4.5
May 17 01:33:46 client-18 rshd[28851]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:33:46 client-18 xinetd[3390]: EXIT: shell status=0 pid=28850 duration=0(sec)
May 17 01:33:47 client-18 xinetd[3390]: START: shell pid=28886 from=192.168.4.5
May 17 01:33:47 client-18 rshd[28887]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:33:47 client-18 xinetd[3390]: EXIT: shell status=0 pid=28886 duration=0(sec)
May 17 01:33:47 client-18 xinetd[3390]: START: shell pid=28923 from=192.168.4.5
May 17 01:33:47 client-18 rshd[28924]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 56p: check lfs find -uid and ! -uid =================================================== 01:33:47 \(1305621227\)");echo XXRETCODE:$?'
May 17 01:33:47 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 56p: check lfs find -uid and ! -uid =================================================== 01:33:47 (1305621227)
May 17 01:33:47 client-18 xinetd[3390]: EXIT: shell status=0 pid=28923 duration=0(sec)
May 17 01:33:47 client-18 xinetd[3390]: START: shell pid=28958 from=192.168.4.5
May 17 01:33:47 client-18 rshd[28959]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:33:47 client-18 xinetd[3390]: EXIT: shell status=0 pid=28958 duration=0(sec)
May 17 01:33:47 client-18 xinetd[3390]: START: shell pid=28994 from=192.168.4.5
May 17 01:33:47 client-18 rshd[28995]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:33:47 client-18 xinetd[3390]: EXIT: shell status=0 pid=28994 duration=0(sec)
May 17 01:33:47 client-18 xinetd[3390]: START: shell pid=29031 from=192.168.4.5
May 17 01:33:48 client-18 rshd[29032]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 56q: check lfs find -gid and ! -gid =================================================== 01:33:47 \(1305621227\)");echo XXRETCODE:$?'
May 17 01:33:48 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 56q: check lfs find -gid and ! -gid =================================================== 01:33:47 (1305621227)
May 17 01:33:48 client-18 xinetd[3390]: EXIT: shell status=0 pid=29031 duration=1(sec)
May 17 01:33:48 client-18 xinetd[3390]: START: shell pid=29066 from=192.168.4.5
May 17 01:33:48 client-18 rshd[29067]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:33:48 client-18 xinetd[3390]: EXIT: shell status=0 pid=29066 duration=0(sec)
May 17 01:33:48 client-18 xinetd[3390]: START: shell pid=29102 from=192.168.4.5
May 17 01:33:48 client-18 rshd[29103]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:33:48 client-18 xinetd[3390]: EXIT: shell status=0 pid=29102 duration=0(sec)
May 17 01:33:48 client-18 xinetd[3390]: START: shell pid=29139 from=192.168.4.5
May 17 01:33:48 client-18 rshd[29140]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 56r: check lfs find -size works ======================================================= 01:33:48 \(1305621228\)");echo XXRETCODE:$?'
May 17 01:33:48 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 56r: check lfs find -size works ======================================================= 01:33:48 (1305621228)
May 17 01:33:48 client-18 xinetd[3390]: EXIT: shell status=0 pid=29139 duration=0(sec)
May 17 01:33:48 client-18 xinetd[3390]: START: shell pid=29174 from=192.168.4.5
May 17 01:33:49 client-18 rshd[29175]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark  sanity test_56r: @@@@@@ FAIL: lfs find \/mnt\/lustre\/d0.sanity\/d56g -size 5 wrong: found 0, expected 1 ");echo XXRETCODE:$?'
May 17 01:33:49 client-18 kernel: Lustre: DEBUG MARKER: sanity test_56r: @@@@@@ FAIL: lfs find /mnt/lustre/d0.sanity/d56g -size 5 wrong: found 0, expected 1
May 17 01:33:49 client-18 xinetd[3390]: EXIT: shell status=0 pid=29174 duration=1(sec)
May 17 01:33:49 client-18 xinetd[3390]: START: shell pid=29209 from=192.168.4.5
May 17 01:33:49 client-18 rshd[29210]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl dk > /tmp/test_logs/2011-05-17/001301/sanity.test_56r.debug_log.\$(hostname).1305621229.log;          dmesg > /tmp/test_logs/2011-05-17/001301/sanity.test_56r.dmesg.\$(hostname).1305621229.log");echo XXRETCODE:$?'
May 17 01:33:49 client-18 xinetd[3390]: EXIT: shell status=0 pid=29209 duration=0(sec)
May 17 01:33:49 client-18 xinetd[3390]: START: shell pid=29250 from=192.168.4.5
May 17 01:33:49 client-18 rshd[29251]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rsync -az /tmp/test_logs/2011-05-17/001301/sanity.test_56r.*.1305621229.log client-5-ib:/tmp/test_logs/2011-05-17/001301");echo XXRETCODE:$?'
May 17 01:33:49 client-18 xinetd[3390]: EXIT: shell status=0 pid=29250 duration=0(sec)
May 17 01:33:52 client-18 xinetd[3390]: START: shell pid=29286 from=192.168.4.5
May 17 01:33:52 client-18 rshd[29287]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 57a: verify MDS filesystem created with large inodes ================================== 01:33:52 \(1305621232\)");echo XXRETCODE:$?'
May 17 01:33:52 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 57a: verify MDS filesystem created with large inodes ================================== 01:33:52 (1305621232)
May 17 01:33:52 client-18 xinetd[3390]: EXIT: shell status=0 pid=29286 duration=0(sec)
May 17 01:33:52 client-18 xinetd[3390]: START: shell pid=29321 from=192.168.4.5
May 17 01:33:52 client-18 rshd[29322]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:33:52 client-18 xinetd[3390]: EXIT: shell status=0 pid=29321 duration=0(sec)
May 17 01:33:52 client-18 xinetd[3390]: START: shell pid=29357 from=192.168.4.5
May 17 01:33:53 client-18 rshd[29358]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:33:53 client-18 xinetd[3390]: EXIT: shell status=0 pid=29357 duration=1(sec)
May 17 01:33:53 client-18 xinetd[3390]: START: shell pid=29394 from=192.168.4.5
May 17 01:33:53 client-18 rshd[29395]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 57b: default LOV EAs are stored inside large inodes =================================== 01:33:53 \(1305621233\)");echo XXRETCODE:$?'
May 17 01:33:53 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 57b: default LOV EAs are stored inside large inodes =================================== 01:33:53 (1305621233)
May 17 01:33:53 client-18 xinetd[3390]: EXIT: shell status=0 pid=29394 duration=0(sec)
May 17 01:33:58 client-18 xinetd[3390]: START: shell pid=29429 from=192.168.4.5
May 17 01:33:58 client-18 rshd[29430]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:33:58 client-18 xinetd[3390]: EXIT: shell status=0 pid=29429 duration=0(sec)
May 17 01:33:58 client-18 xinetd[3390]: START: shell pid=29465 from=192.168.4.5
May 17 01:33:58 client-18 rshd[29466]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:33:58 client-18 xinetd[3390]: EXIT: shell status=0 pid=29465 duration=0(sec)
May 17 01:33:58 client-18 xinetd[3390]: START: shell pid=29502 from=192.168.4.5
May 17 01:33:58 client-18 rshd[29503]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 58: verify cross-platform wire constants ============================================== 01:33:58 \(1305621238\)");echo XXRETCODE:$?'
May 17 01:33:58 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 58: verify cross-platform wire constants ============================================== 01:33:58 (1305621238)
May 17 01:33:58 client-18 xinetd[3390]: EXIT: shell status=0 pid=29502 duration=0(sec)
May 17 01:33:58 client-18 xinetd[3390]: START: shell pid=29537 from=192.168.4.5
May 17 01:33:58 client-18 rshd[29538]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:33:58 client-18 xinetd[3390]: EXIT: shell status=0 pid=29537 duration=0(sec)
May 17 01:33:58 client-18 xinetd[3390]: START: shell pid=29573 from=192.168.4.5
May 17 01:33:58 client-18 rshd[29574]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:33:59 client-18 xinetd[3390]: EXIT: shell status=0 pid=29573 duration=1(sec)
May 17 01:33:59 client-18 xinetd[3390]: START: shell pid=29610 from=192.168.4.5
May 17 01:33:59 client-18 rshd[29611]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 59: verify cancellation of llog records async ========================================= 01:33:59 \(1305621239\)");echo XXRETCODE:$?'
May 17 01:33:59 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 59: verify cancellation of llog records async ========================================= 01:33:59 (1305621239)
May 17 01:33:59 client-18 xinetd[3390]: EXIT: shell status=0 pid=29610 duration=0(sec)
May 17 01:34:01 client-18 xinetd[3390]: START: shell pid=29645 from=192.168.4.5
May 17 01:34:01 client-18 rshd[29646]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:34:02 client-18 xinetd[3390]: EXIT: shell status=0 pid=29645 duration=1(sec)
May 17 01:34:02 client-18 xinetd[3390]: START: shell pid=29681 from=192.168.4.5
May 17 01:34:02 client-18 rshd[29682]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:34:02 client-18 xinetd[3390]: EXIT: shell status=0 pid=29681 duration=0(sec)
May 17 01:34:02 client-18 xinetd[3390]: START: shell pid=29718 from=192.168.4.5
May 17 01:34:02 client-18 rshd[29719]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 60a: llog sanity tests run from kernel module ========================================= 01:34:02 \(1305621242\)");echo XXRETCODE:$?'
May 17 01:34:02 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 60a: llog sanity tests run from kernel module ========================================= 01:34:02 (1305621242)
May 17 01:34:02 client-18 xinetd[3390]: EXIT: shell status=0 pid=29718 duration=0(sec)
May 17 01:34:02 client-18 xinetd[3390]: START: shell pid=29753 from=192.168.4.5
May 17 01:34:02 client-18 rshd[29754]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark test_60 run 22840 - from kernel mode");echo XXRETCODE:$?'
May 17 01:34:02 client-18 kernel: Lustre: DEBUG MARKER: test_60 run 22840 - from kernel mode
May 17 01:34:02 client-18 xinetd[3390]: EXIT: shell status=0 pid=29753 duration=0(sec)
May 17 01:34:02 client-18 xinetd[3390]: START: shell pid=29788 from=192.168.4.5
May 17 01:34:02 client-18 rshd[29789]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:34:02 client-18 xinetd[3390]: EXIT: shell status=0 pid=29788 duration=0(sec)
May 17 01:34:02 client-18 xinetd[3390]: START: shell pid=29824 from=192.168.4.5
May 17 01:34:02 client-18 rshd[29825]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:34:02 client-18 xinetd[3390]: EXIT: shell status=0 pid=29824 duration=0(sec)
May 17 01:34:02 client-18 xinetd[3390]: START: shell pid=29861 from=192.168.4.5
May 17 01:34:03 client-18 rshd[29862]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 60b: limit repeated messages from CERROR\/CWARN ======================================== 01:34:02 \(1305621242\)");echo XXRETCODE:$?'
May 17 01:34:03 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 60b: limit repeated messages from CERROR/CWARN ======================================== 01:34:02 (1305621242)
May 17 01:34:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=29861 duration=1(sec)
May 17 01:34:03 client-18 xinetd[3390]: START: shell pid=29896 from=192.168.4.5
May 17 01:34:03 client-18 rshd[29897]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:34:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=29896 duration=0(sec)
May 17 01:34:03 client-18 xinetd[3390]: START: shell pid=29932 from=192.168.4.5
May 17 01:34:03 client-18 rshd[29933]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:34:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=29932 duration=0(sec)
May 17 01:34:03 client-18 xinetd[3390]: START: shell pid=29969 from=192.168.4.5
May 17 01:34:03 client-18 rshd[29970]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 60c: unlink file when mds full ======================================================== 01:34:03 \(1305621243\)");echo XXRETCODE:$?'
May 17 01:34:03 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 60c: unlink file when mds full ======================================================== 01:34:03 (1305621243)
May 17 01:34:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=29969 duration=0(sec)
May 17 01:34:28 client-18 xinetd[3390]: START: shell pid=30004 from=192.168.4.5
May 17 01:34:28 client-18 rshd[30005]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:34:28 client-18 xinetd[3390]: EXIT: shell status=0 pid=30004 duration=0(sec)
May 17 01:34:28 client-18 xinetd[3390]: START: shell pid=30040 from=192.168.4.5
May 17 01:34:28 client-18 rshd[30041]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:34:28 client-18 xinetd[3390]: EXIT: shell status=0 pid=30040 duration=0(sec)
May 17 01:34:28 client-18 xinetd[3390]: START: shell pid=30077 from=192.168.4.5
May 17 01:34:28 client-18 rshd[30078]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 60d: test printk console message masking ============================================== 01:34:28 \(1305621268\)");echo XXRETCODE:$?'
May 17 01:34:28 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 60d: test printk console message masking ============================================== 01:34:28 (1305621268)
May 17 01:34:28 client-18 xinetd[3390]: EXIT: shell status=0 pid=30077 duration=0(sec)
May 17 01:34:28 client-18 xinetd[3390]: START: shell pid=30112 from=192.168.4.5
May 17 01:34:28 client-18 rshd[30113]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:34:29 client-18 xinetd[3390]: EXIT: shell status=0 pid=30112 duration=1(sec)
May 17 01:34:29 client-18 xinetd[3390]: START: shell pid=30148 from=192.168.4.5
May 17 01:34:29 client-18 rshd[30149]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:34:29 client-18 xinetd[3390]: EXIT: shell status=0 pid=30148 duration=0(sec)
May 17 01:34:29 client-18 xinetd[3390]: START: shell pid=30185 from=192.168.4.5
May 17 01:34:29 client-18 rshd[30186]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 61: mmap\(\) writes don\'t make sync hang ================================================ 01:34:29 \(1305621269\)");echo XXRETCODE:$?'
May 17 01:34:29 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 61: mmap() writes don't make sync hang ================================================ 01:34:29 (1305621269)
May 17 01:34:29 client-18 xinetd[3390]: EXIT: shell status=0 pid=30185 duration=0(sec)
May 17 01:34:29 client-18 xinetd[3390]: START: shell pid=30220 from=192.168.4.5
May 17 01:34:29 client-18 rshd[30221]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:34:29 client-18 xinetd[3390]: EXIT: shell status=0 pid=30220 duration=0(sec)
May 17 01:34:29 client-18 xinetd[3390]: START: shell pid=30256 from=192.168.4.5
May 17 01:34:29 client-18 rshd[30257]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:34:30 client-18 xinetd[3390]: EXIT: shell status=0 pid=30256 duration=1(sec)
May 17 01:34:30 client-18 xinetd[3390]: START: shell pid=30293 from=192.168.4.5
May 17 01:34:30 client-18 rshd[30294]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 63a: Verify oig_wait interruption does not crash ====================================== 01:34:30 \(1305621270\)");echo XXRETCODE:$?'
May 17 01:34:30 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 63a: Verify oig_wait interruption does not crash ====================================== 01:34:30 (1305621270)
May 17 01:34:30 client-18 xinetd[3390]: EXIT: shell status=0 pid=30293 duration=0(sec)
May 17 01:35:30 client-18 xinetd[3390]: START: shell pid=30332 from=192.168.4.5
May 17 01:35:30 client-18 rshd[30333]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:35:30 client-18 xinetd[3390]: EXIT: shell status=0 pid=30332 duration=0(sec)
May 17 01:35:30 client-18 xinetd[3390]: START: shell pid=30368 from=192.168.4.5
May 17 01:35:30 client-18 rshd[30369]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:35:30 client-18 xinetd[3390]: EXIT: shell status=0 pid=30368 duration=0(sec)
May 17 01:35:30 client-18 xinetd[3390]: START: shell pid=30405 from=192.168.4.5
May 17 01:35:30 client-18 rshd[30406]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 63b: async write errors should be returned to fsync =================================== 01:35:30 \(1305621330\)");echo XXRETCODE:$?'
May 17 01:35:30 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 63b: async write errors should be returned to fsync =================================== 01:35:30 (1305621330)
May 17 01:35:30 client-18 xinetd[3390]: EXIT: shell status=0 pid=30405 duration=0(sec)
May 17 01:35:33 client-18 xinetd[3390]: START: shell pid=30440 from=192.168.4.5
May 17 01:35:33 client-18 rshd[30441]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl set_param debug=\"trace inode super ext2 malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec\";");echo XXRETCODE:$?'
May 17 01:35:33 client-18 xinetd[3390]: EXIT: shell status=0 pid=30440 duration=0(sec)
May 17 01:35:33 client-18 xinetd[3390]: START: shell pid=30475 from=192.168.4.5
May 17 01:35:33 client-18 rshd[30476]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:35:33 client-18 xinetd[3390]: EXIT: shell status=0 pid=30475 duration=0(sec)
May 17 01:35:33 client-18 xinetd[3390]: START: shell pid=30511 from=192.168.4.5
May 17 01:35:33 client-18 rshd[30512]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:35:33 client-18 xinetd[3390]: EXIT: shell status=0 pid=30511 duration=0(sec)
May 17 01:35:34 client-18 xinetd[3390]: START: shell pid=30548 from=192.168.4.5
May 17 01:35:34 client-18 rshd[30549]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 64a: verify filter grant calculations \(in kernel\) ===================================== 01:35:33 \(1305621333\)");echo XXRETCODE:$?'
May 17 01:35:34 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 64a: verify filter grant calculations (in kernel) ===================================== 01:35:33 (1305621333)
May 17 01:35:34 client-18 xinetd[3390]: EXIT: shell status=0 pid=30548 duration=0(sec)
May 17 01:35:34 client-18 xinetd[3390]: START: shell pid=30583 from=192.168.4.5
May 17 01:35:34 client-18 rshd[30584]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:35:34 client-18 xinetd[3390]: EXIT: shell status=0 pid=30583 duration=0(sec)
May 17 01:35:34 client-18 xinetd[3390]: START: shell pid=30619 from=192.168.4.5
May 17 01:35:34 client-18 rshd[30620]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:35:34 client-18 xinetd[3390]: EXIT: shell status=0 pid=30619 duration=0(sec)
May 17 01:35:34 client-18 xinetd[3390]: START: shell pid=30656 from=192.168.4.5
May 17 01:35:34 client-18 rshd[30657]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 64b: check out-of-space detection on client =========================================== 01:35:34 \(1305621334\)");echo XXRETCODE:$?'
May 17 01:35:34 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 64b: check out-of-space detection on client =========================================== 01:35:34 (1305621334)
May 17 01:35:34 client-18 xinetd[3390]: EXIT: shell status=0 pid=30656 duration=0(sec)
May 17 01:35:35 client-18 xinetd[3390]: START: shell pid=30691 from=192.168.4.5
May 17 01:35:35 client-18 rshd[30692]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark  SKIP: sanity test_64b oos.sh: 53523596kB free gt MAXFREE 2400000kB, increase 2400000 \(or reduce test fs size\) to proceed");echo XXRETCODE:$?'
May 17 01:35:35 client-18 kernel: Lustre: DEBUG MARKER: SKIP: sanity test_64b oos.sh: 53523596kB free gt MAXFREE 2400000kB, increase 2400000 (or reduce test fs size) to proceed
May 17 01:35:35 client-18 xinetd[3390]: EXIT: shell status=0 pid=30691 duration=0(sec)
May 17 01:35:35 client-18 xinetd[3390]: START: shell pid=30726 from=192.168.4.5
May 17 01:35:36 client-18 rshd[30727]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:35:36 client-18 xinetd[3390]: EXIT: shell status=0 pid=30726 duration=1(sec)
May 17 01:35:36 client-18 xinetd[3390]: START: shell pid=30762 from=192.168.4.5
May 17 01:35:36 client-18 rshd[30763]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:35:36 client-18 xinetd[3390]: EXIT: shell status=0 pid=30762 duration=0(sec)
May 17 01:35:36 client-18 xinetd[3390]: START: shell pid=30799 from=192.168.4.5
May 17 01:35:36 client-18 rshd[30800]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark  SKIP: sanity test_65a skipping ALWAYS excluded test 65a");echo XXRETCODE:$?'
May 17 01:35:36 client-18 kernel: Lustre: DEBUG MARKER: SKIP: sanity test_65a skipping ALWAYS excluded test 65a
May 17 01:35:36 client-18 xinetd[3390]: EXIT: shell status=0 pid=30799 duration=0(sec)
May 17 01:35:36 client-18 xinetd[3390]: START: shell pid=30834 from=192.168.4.5
May 17 01:35:36 client-18 rshd[30835]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 65b: directory setstripe 2097152 0 1 ================================================== 01:35:36 \(1305621336\)");echo XXRETCODE:$?'
May 17 01:35:36 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 65b: directory setstripe 2097152 0 1 ================================================== 01:35:36 (1305621336)
May 17 01:35:36 client-18 xinetd[3390]: EXIT: shell status=0 pid=30834 duration=0(sec)
May 17 01:35:36 client-18 xinetd[3390]: START: shell pid=30869 from=192.168.4.5
May 17 01:35:36 client-18 rshd[30870]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:35:36 client-18 xinetd[3390]: EXIT: shell status=0 pid=30869 duration=0(sec)
May 17 01:35:36 client-18 xinetd[3390]: START: shell pid=30905 from=192.168.4.5
May 17 01:35:36 client-18 rshd[30906]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:35:37 client-18 xinetd[3390]: EXIT: shell status=0 pid=30905 duration=1(sec)
May 17 01:35:37 client-18 xinetd[3390]: START: shell pid=30942 from=192.168.4.5
May 17 01:35:37 client-18 rshd[30943]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 65c: directory setstripe 4194304 1 5 ================================================== 01:35:37 \(1305621337\)");echo XXRETCODE:$?'
May 17 01:35:37 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 65c: directory setstripe 4194304 1 5 ================================================== 01:35:37 (1305621337)
May 17 01:35:37 client-18 xinetd[3390]: EXIT: shell status=0 pid=30942 duration=0(sec)
May 17 01:35:37 client-18 xinetd[3390]: START: shell pid=30977 from=192.168.4.5
May 17 01:35:37 client-18 rshd[30978]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:35:37 client-18 xinetd[3390]: EXIT: shell status=0 pid=30977 duration=0(sec)
May 17 01:35:37 client-18 xinetd[3390]: START: shell pid=31013 from=192.168.4.5
May 17 01:35:37 client-18 rshd[31014]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:35:37 client-18 xinetd[3390]: EXIT: shell status=0 pid=31013 duration=0(sec)
May 17 01:35:37 client-18 xinetd[3390]: START: shell pid=31050 from=192.168.4.5
May 17 01:35:37 client-18 rshd[31051]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 65d: directory setstripe 1048576 -1 stripe_count ====================================== 01:35:37 \(1305621337\)");echo XXRETCODE:$?'
May 17 01:35:37 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 65d: directory setstripe 1048576 -1 stripe_count ====================================== 01:35:37 (1305621337)
May 17 01:35:37 client-18 xinetd[3390]: EXIT: shell status=0 pid=31050 duration=0(sec)
May 17 01:35:37 client-18 xinetd[3390]: START: shell pid=31085 from=192.168.4.5
May 17 01:35:37 client-18 rshd[31086]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:35:37 client-18 xinetd[3390]: EXIT: shell status=0 pid=31085 duration=0(sec)
May 17 01:35:38 client-18 xinetd[3390]: START: shell pid=31121 from=192.168.4.5
May 17 01:35:38 client-18 rshd[31122]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:35:38 client-18 xinetd[3390]: EXIT: shell status=0 pid=31121 duration=0(sec)
May 17 01:35:38 client-18 xinetd[3390]: START: shell pid=31158 from=192.168.4.5
May 17 01:35:38 client-18 rshd[31159]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark  SKIP: sanity test_65e skipping ALWAYS excluded test 65e");echo XXRETCODE:$?'
May 17 01:35:38 client-18 kernel: Lustre: DEBUG MARKER: SKIP: sanity test_65e skipping ALWAYS excluded test 65e
May 17 01:35:38 client-18 xinetd[3390]: EXIT: shell status=0 pid=31158 duration=0(sec)
May 17 01:35:38 client-18 xinetd[3390]: START: shell pid=31193 from=192.168.4.5
May 17 01:35:38 client-18 rshd[31194]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 65f: dir setstripe permission \(should return error\) =================================== 01:35:38 \(1305621338\)");echo XXRETCODE:$?'
May 17 01:35:38 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 65f: dir setstripe permission (should return error) =================================== 01:35:38 (1305621338)
May 17 01:35:38 client-18 xinetd[3390]: EXIT: shell status=0 pid=31193 duration=0(sec)
May 17 01:35:38 client-18 xinetd[3390]: START: shell pid=31228 from=192.168.4.5
May 17 01:35:38 client-18 rshd[31229]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:35:38 client-18 xinetd[3390]: EXIT: shell status=0 pid=31228 duration=0(sec)
May 17 01:35:38 client-18 xinetd[3390]: START: shell pid=31264 from=192.168.4.5
May 17 01:35:38 client-18 rshd[31265]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:35:38 client-18 xinetd[3390]: EXIT: shell status=0 pid=31264 duration=0(sec)
May 17 01:35:38 client-18 xinetd[3390]: START: shell pid=31301 from=192.168.4.5
May 17 01:35:38 client-18 rshd[31302]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 65g: directory setstripe -d =========================================================== 01:35:38 \(1305621338\)");echo XXRETCODE:$?'
May 17 01:35:39 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 65g: directory setstripe -d =========================================================== 01:35:38 (1305621338)
May 17 01:35:39 client-18 xinetd[3390]: EXIT: shell status=0 pid=31301 duration=1(sec)
May 17 01:35:39 client-18 xinetd[3390]: START: shell pid=31336 from=192.168.4.5
May 17 01:35:39 client-18 rshd[31337]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:35:39 client-18 xinetd[3390]: EXIT: shell status=0 pid=31336 duration=0(sec)
May 17 01:35:39 client-18 xinetd[3390]: START: shell pid=31372 from=192.168.4.5
May 17 01:35:39 client-18 rshd[31373]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:35:39 client-18 xinetd[3390]: EXIT: shell status=0 pid=31372 duration=0(sec)
May 17 01:35:39 client-18 xinetd[3390]: START: shell pid=31409 from=192.168.4.5
May 17 01:35:39 client-18 rshd[31410]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 65h: directory stripe info inherit ==================================================== 01:35:39 \(1305621339\)");echo XXRETCODE:$?'
May 17 01:35:39 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 65h: directory stripe info inherit ==================================================== 01:35:39 (1305621339)
May 17 01:35:39 client-18 xinetd[3390]: EXIT: shell status=0 pid=31409 duration=0(sec)
May 17 01:35:39 client-18 xinetd[3390]: START: shell pid=31444 from=192.168.4.5
May 17 01:35:39 client-18 rshd[31445]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:35:39 client-18 xinetd[3390]: EXIT: shell status=0 pid=31444 duration=0(sec)
May 17 01:35:39 client-18 xinetd[3390]: START: shell pid=31480 from=192.168.4.5
May 17 01:35:39 client-18 rshd[31481]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:35:39 client-18 xinetd[3390]: EXIT: shell status=0 pid=31480 duration=0(sec)
May 17 01:35:40 client-18 xinetd[3390]: START: shell pid=31517 from=192.168.4.5
May 17 01:35:40 client-18 rshd[31518]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 65i: set non-default striping on root directory \(bug 6367\)============================= 01:35:39 \(1305621339\)");echo XXRETCODE:$?'
May 17 01:35:40 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 65i: set non-default striping on root directory (bug 6367)============================= 01:35:39 (1305621339)
May 17 01:35:40 client-18 xinetd[3390]: EXIT: shell status=0 pid=31517 duration=0(sec)
May 17 01:35:40 client-18 xinetd[3390]: START: shell pid=31553 from=192.168.4.5
May 17 01:35:40 client-18 rshd[31554]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:35:40 client-18 xinetd[3390]: EXIT: shell status=0 pid=31553 duration=0(sec)
May 17 01:35:40 client-18 xinetd[3390]: START: shell pid=31589 from=192.168.4.5
May 17 01:35:40 client-18 rshd[31590]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:35:40 client-18 xinetd[3390]: EXIT: shell status=0 pid=31589 duration=0(sec)
May 17 01:35:40 client-18 xinetd[3390]: START: shell pid=31626 from=192.168.4.5
May 17 01:35:40 client-18 rshd[31627]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 65ia: getstripe on -1 default directory striping ====================================== 01:35:40 \(1305621340\)");echo XXRETCODE:$?'
May 17 01:35:40 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 65ia: getstripe on -1 default directory striping ====================================== 01:35:40 (1305621340)
May 17 01:35:40 client-18 xinetd[3390]: EXIT: shell status=0 pid=31626 duration=0(sec)
May 17 01:35:41 client-18 xinetd[3390]: START: shell pid=31661 from=192.168.4.5
May 17 01:35:41 client-18 rshd[31662]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:35:41 client-18 xinetd[3390]: EXIT: shell status=0 pid=31661 duration=0(sec)
May 17 01:35:41 client-18 xinetd[3390]: START: shell pid=31697 from=192.168.4.5
May 17 01:35:41 client-18 rshd[31698]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:35:41 client-18 xinetd[3390]: EXIT: shell status=0 pid=31697 duration=0(sec)
May 17 01:35:41 client-18 xinetd[3390]: START: shell pid=31734 from=192.168.4.5
May 17 01:35:41 client-18 rshd[31735]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 65ib: getstripe -v on -1 default directory striping =================================== 01:35:41 \(1305621341\)");echo XXRETCODE:$?'
May 17 01:35:41 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 65ib: getstripe -v on -1 default directory striping =================================== 01:35:41 (1305621341)
May 17 01:35:41 client-18 xinetd[3390]: EXIT: shell status=0 pid=31734 duration=0(sec)
May 17 01:35:41 client-18 xinetd[3390]: START: shell pid=31769 from=192.168.4.5
May 17 01:35:41 client-18 rshd[31770]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:35:42 client-18 xinetd[3390]: EXIT: shell status=0 pid=31769 duration=1(sec)
May 17 01:35:42 client-18 xinetd[3390]: START: shell pid=31805 from=192.168.4.5
May 17 01:35:42 client-18 rshd[31806]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:35:42 client-18 xinetd[3390]: EXIT: shell status=0 pid=31805 duration=0(sec)
May 17 01:35:42 client-18 xinetd[3390]: START: shell pid=31842 from=192.168.4.5
May 17 01:35:42 client-18 rshd[31843]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 65ic: new find on -1 default directory striping ======================================= 01:35:42 \(1305621342\)");echo XXRETCODE:$?'
May 17 01:35:42 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 65ic: new find on -1 default directory striping ======================================= 01:35:42 (1305621342)
May 17 01:35:42 client-18 xinetd[3390]: EXIT: shell status=0 pid=31842 duration=0(sec)
May 17 01:35:44 client-18 xinetd[3390]: START: shell pid=31877 from=192.168.4.5
May 17 01:35:44 client-18 rshd[31878]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:35:44 client-18 xinetd[3390]: EXIT: shell status=0 pid=31877 duration=0(sec)
May 17 01:35:44 client-18 xinetd[3390]: START: shell pid=31913 from=192.168.4.5
May 17 01:35:44 client-18 rshd[31914]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:35:44 client-18 xinetd[3390]: EXIT: shell status=0 pid=31913 duration=0(sec)
May 17 01:35:44 client-18 xinetd[3390]: START: shell pid=31950 from=192.168.4.5
May 17 01:35:44 client-18 rshd[31951]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 65j: set default striping on root directory \(bug 6367\)================================= 01:35:44 \(1305621344\)");echo XXRETCODE:$?'
May 17 01:35:44 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 65j: set default striping on root directory (bug 6367)================================= 01:35:44 (1305621344)
May 17 01:35:44 client-18 xinetd[3390]: EXIT: shell status=0 pid=31950 duration=0(sec)
May 17 01:35:45 client-18 xinetd[3390]: START: shell pid=31985 from=192.168.4.5
May 17 01:35:45 client-18 rshd[31986]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:35:45 client-18 kernel: Lustre: Increasing default stripe size to min 1048576
May 17 01:35:45 client-18 xinetd[3390]: EXIT: shell status=0 pid=31985 duration=0(sec)
May 17 01:35:45 client-18 xinetd[3390]: START: shell pid=32023 from=192.168.4.5
May 17 01:35:45 client-18 rshd[32024]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:35:45 client-18 xinetd[3390]: EXIT: shell status=0 pid=32023 duration=0(sec)
May 17 01:35:45 client-18 xinetd[3390]: START: shell pid=32060 from=192.168.4.5
May 17 01:35:45 client-18 rshd[32061]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 65k: validate manual striping works properly with deactivated OSCs ==================== 01:35:45 \(1305621345\)");echo XXRETCODE:$?'
May 17 01:35:46 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 65k: validate manual striping works properly with deactivated OSCs ==================== 01:35:45 (1305621345)
May 17 01:35:46 client-18 xinetd[3390]: EXIT: shell status=0 pid=32060 duration=1(sec)
May 17 01:35:53 client-18 xinetd[3390]: START: shell pid=32095 from=192.168.4.5
May 17 01:35:53 client-18 rshd[32096]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:35:53 client-18 xinetd[3390]: EXIT: shell status=0 pid=32095 duration=0(sec)
May 17 01:35:53 client-18 xinetd[3390]: START: shell pid=32131 from=192.168.4.5
May 17 01:35:53 client-18 rshd[32132]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:35:53 client-18 xinetd[3390]: EXIT: shell status=0 pid=32131 duration=0(sec)
May 17 01:35:53 client-18 xinetd[3390]: START: shell pid=32168 from=192.168.4.5
May 17 01:35:53 client-18 rshd[32169]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 65l: lfs find on -1 stripe dir ======================================================== 01:35:53 \(1305621353\)");echo XXRETCODE:$?'
May 17 01:35:53 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 65l: lfs find on -1 stripe dir ======================================================== 01:35:53 (1305621353)
May 17 01:35:53 client-18 xinetd[3390]: EXIT: shell status=0 pid=32168 duration=0(sec)
May 17 01:35:53 client-18 xinetd[3390]: START: shell pid=32203 from=192.168.4.5
May 17 01:35:53 client-18 rshd[32204]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:35:54 client-18 xinetd[3390]: EXIT: shell status=0 pid=32203 duration=1(sec)
May 17 01:35:54 client-18 xinetd[3390]: START: shell pid=32239 from=192.168.4.5
May 17 01:35:54 client-18 rshd[32240]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:35:54 client-18 xinetd[3390]: EXIT: shell status=0 pid=32239 duration=0(sec)
May 17 01:35:54 client-18 xinetd[3390]: START: shell pid=32276 from=192.168.4.5
May 17 01:35:54 client-18 rshd[32277]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 66: update inode blocks count on client =============================================== 01:35:54 \(1305621354\)");echo XXRETCODE:$?'
May 17 01:35:54 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 66: update inode blocks count on client =============================================== 01:35:54 (1305621354)
May 17 01:35:54 client-18 xinetd[3390]: EXIT: shell status=0 pid=32276 duration=0(sec)
May 17 01:35:55 client-18 xinetd[3390]: START: shell pid=32311 from=192.168.4.5
May 17 01:35:55 client-18 rshd[32312]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:35:55 client-18 xinetd[3390]: EXIT: shell status=0 pid=32311 duration=0(sec)
May 17 01:35:55 client-18 xinetd[3390]: START: shell pid=32347 from=192.168.4.5
May 17 01:35:55 client-18 rshd[32348]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:35:55 client-18 xinetd[3390]: EXIT: shell status=0 pid=32347 duration=0(sec)
May 17 01:35:55 client-18 xinetd[3390]: START: shell pid=32384 from=192.168.4.5
May 17 01:35:55 client-18 rshd[32385]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 68a: lloop driver - basic test ======================================================== 01:35:55 \(1305621355\)");echo XXRETCODE:$?'
May 17 01:35:55 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 68a: lloop driver - basic test ======================================================== 01:35:55 (1305621355)
May 17 01:35:55 client-18 xinetd[3390]: EXIT: shell status=0 pid=32384 duration=0(sec)
May 17 01:35:57 client-18 xinetd[3390]: START: shell pid=32419 from=192.168.4.5
May 17 01:35:57 client-18 rshd[32420]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:35:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=32419 duration=0(sec)
May 17 01:35:57 client-18 xinetd[3390]: START: shell pid=32455 from=192.168.4.5
May 17 01:35:57 client-18 rshd[32456]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:35:58 client-18 xinetd[3390]: EXIT: shell status=0 pid=32455 duration=1(sec)
May 17 01:35:58 client-18 xinetd[3390]: START: shell pid=32492 from=192.168.4.5
May 17 01:35:58 client-18 rshd[32493]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark  SKIP: sanity test_68b skipping ALWAYS excluded test 68b");echo XXRETCODE:$?'
May 17 01:35:58 client-18 kernel: Lustre: DEBUG MARKER: SKIP: sanity test_68b skipping ALWAYS excluded test 68b
May 17 01:35:58 client-18 xinetd[3390]: EXIT: shell status=0 pid=32492 duration=0(sec)
May 17 01:35:58 client-18 xinetd[3390]: START: shell pid=32527 from=192.168.4.5
May 17 01:35:58 client-18 rshd[32528]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 69: verify oa2dentry return -ENOENT doesn\'t LBUG ====================================== 01:35:58 \(1305621358\)");echo XXRETCODE:$?'
May 17 01:35:58 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 69: verify oa2dentry return -ENOENT doesn't LBUG ====================================== 01:35:58 (1305621358)
May 17 01:35:58 client-18 xinetd[3390]: EXIT: shell status=0 pid=32527 duration=0(sec)
May 17 01:35:59 client-18 xinetd[3390]: START: shell pid=32562 from=192.168.4.5
May 17 01:36:00 client-18 rshd[32563]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:36:00 client-18 xinetd[3390]: EXIT: shell status=0 pid=32562 duration=1(sec)
May 17 01:36:00 client-18 xinetd[3390]: START: shell pid=32598 from=192.168.4.5
May 17 01:36:00 client-18 rshd[32599]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:36:00 client-18 xinetd[3390]: EXIT: shell status=0 pid=32598 duration=0(sec)
May 17 01:36:00 client-18 xinetd[3390]: START: shell pid=32635 from=192.168.4.5
May 17 01:36:00 client-18 rshd[32636]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 71: Running dbench on lustre \(don\'t segment fault\) ==================================== 01:36:00 \(1305621360\)");echo XXRETCODE:$?'
May 17 01:36:00 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 71: Running dbench on lustre (don't segment fault) ==================================== 01:36:00 (1305621360)
May 17 01:36:00 client-18 xinetd[3390]: EXIT: shell status=0 pid=32635 duration=0(sec)
May 17 01:48:05 client-18 xinetd[3390]: START: shell pid=32710 from=192.168.4.5
May 17 01:48:05 client-18 rshd[32711]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:48:05 client-18 xinetd[3390]: EXIT: shell status=0 pid=32710 duration=0(sec)
May 17 01:48:05 client-18 xinetd[3390]: START: shell pid=32746 from=192.168.4.5
May 17 01:48:05 client-18 rshd[32747]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:48:05 client-18 xinetd[3390]: EXIT: shell status=0 pid=32746 duration=0(sec)
May 17 01:48:05 client-18 xinetd[3390]: START: shell pid=317 from=192.168.4.5
May 17 01:48:05 client-18 rshd[318]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 72a: Test that remove suid works properly \(bug5695\) =================================== 01:48:05 \(1305622085\)");echo XXRETCODE:$?'
May 17 01:48:06 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 72a: Test that remove suid works properly (bug5695) =================================== 01:48:05 (1305622085)
May 17 01:48:06 client-18 xinetd[3390]: EXIT: shell status=0 pid=317 duration=1(sec)
May 17 01:48:09 client-18 xinetd[3390]: START: shell pid=352 from=192.168.4.5
May 17 01:48:09 client-18 rshd[353]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:48:09 client-18 xinetd[3390]: EXIT: shell status=0 pid=352 duration=0(sec)
May 17 01:48:09 client-18 xinetd[3390]: START: shell pid=388 from=192.168.4.5
May 17 01:48:09 client-18 rshd[389]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:48:09 client-18 xinetd[3390]: EXIT: shell status=0 pid=388 duration=0(sec)
May 17 01:48:09 client-18 xinetd[3390]: START: shell pid=433 from=192.168.4.5
May 17 01:48:09 client-18 rshd[434]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 72b: Test that we keep mode setting if without file data changed \(bug 24226\) ========== 01:48:09 \(1305622089\)");echo XXRETCODE:$?'
May 17 01:48:09 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 72b: Test that we keep mode setting if without file data changed (bug 24226) ========== 01:48:09 (1305622089)
May 17 01:48:09 client-18 xinetd[3390]: EXIT: shell status=0 pid=433 duration=0(sec)
May 17 01:48:09 client-18 xinetd[3390]: START: shell pid=468 from=192.168.4.5
May 17 01:48:09 client-18 rshd[469]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:48:10 client-18 xinetd[3390]: EXIT: shell status=0 pid=468 duration=1(sec)
May 17 01:48:10 client-18 xinetd[3390]: START: shell pid=504 from=192.168.4.5
May 17 01:48:10 client-18 rshd[505]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:48:10 client-18 xinetd[3390]: EXIT: shell status=0 pid=504 duration=0(sec)
May 17 01:48:10 client-18 xinetd[3390]: START: shell pid=541 from=192.168.4.5
May 17 01:48:10 client-18 rshd[542]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 73: multiple MDC requests \(should not deadlock\) ======================================= 01:48:10 \(1305622090\)");echo XXRETCODE:$?'
May 17 01:48:10 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 73: multiple MDC requests (should not deadlock) ======================================= 01:48:10 (1305622090)
May 17 01:48:10 client-18 xinetd[3390]: EXIT: shell status=0 pid=541 duration=0(sec)
May 17 01:48:36 client-18 xinetd[3390]: START: shell pid=581 from=192.168.4.5
May 17 01:48:36 client-18 rshd[582]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:48:36 client-18 xinetd[3390]: EXIT: shell status=0 pid=581 duration=0(sec)
May 17 01:48:36 client-18 xinetd[3390]: START: shell pid=629 from=192.168.4.5
May 17 01:48:36 client-18 rshd[630]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:48:36 client-18 xinetd[3390]: EXIT: shell status=0 pid=629 duration=0(sec)
May 17 01:48:36 client-18 xinetd[3390]: START: shell pid=668 from=192.168.4.5
May 17 01:48:36 client-18 rshd[669]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 74a: ldlm_enqueue freed-export error path, ls \(shouldn\'t LBUG\) ======================== 01:48:36 \(1305622116\)");echo XXRETCODE:$?'
May 17 01:48:36 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 74a: ldlm_enqueue freed-export error path, ls (shouldn't LBUG) ======================== 01:48:36 (1305622116)
May 17 01:48:36 client-18 xinetd[3390]: EXIT: shell status=0 pid=668 duration=0(sec)
May 17 01:48:37 client-18 xinetd[3390]: START: shell pid=704 from=192.168.4.5
May 17 01:48:37 client-18 rshd[705]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:48:37 client-18 xinetd[3390]: EXIT: shell status=0 pid=704 duration=0(sec)
May 17 01:48:37 client-18 xinetd[3390]: START: shell pid=740 from=192.168.4.5
May 17 01:48:37 client-18 rshd[741]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:48:37 client-18 xinetd[3390]: EXIT: shell status=0 pid=740 duration=0(sec)
May 17 01:48:37 client-18 xinetd[3390]: START: shell pid=777 from=192.168.4.5
May 17 01:48:37 client-18 rshd[778]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 74b: ldlm_enqueue freed-export error path, touch \(shouldn\'t LBUG\) ===================== 01:48:37 \(1305622117\)");echo XXRETCODE:$?'
May 17 01:48:37 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 74b: ldlm_enqueue freed-export error path, touch (shouldn't LBUG) ===================== 01:48:37 (1305622117)
May 17 01:48:37 client-18 xinetd[3390]: EXIT: shell status=0 pid=777 duration=0(sec)
May 17 01:48:37 client-18 xinetd[3390]: START: shell pid=812 from=192.168.4.5
May 17 01:48:37 client-18 rshd[813]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:48:37 client-18 xinetd[3390]: EXIT: shell status=0 pid=812 duration=0(sec)
May 17 01:48:37 client-18 xinetd[3390]: START: shell pid=848 from=192.168.4.5
May 17 01:48:37 client-18 rshd[849]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:48:37 client-18 xinetd[3390]: EXIT: shell status=0 pid=848 duration=0(sec)
May 17 01:48:37 client-18 xinetd[3390]: START: shell pid=885 from=192.168.4.5
May 17 01:48:37 client-18 rshd[886]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 74c: ldlm_lock_create error path, \(shouldn\'t LBUG\) ==================================== 01:48:37 \(1305622117\)");echo XXRETCODE:$?'
May 17 01:48:38 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 74c: ldlm_lock_create error path, (shouldn't LBUG) ==================================== 01:48:37 (1305622117)
May 17 01:48:38 client-18 xinetd[3390]: EXIT: shell status=0 pid=885 duration=1(sec)
May 17 01:48:38 client-18 xinetd[3390]: START: shell pid=920 from=192.168.4.5
May 17 01:48:38 client-18 rshd[921]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:48:38 client-18 xinetd[3390]: EXIT: shell status=0 pid=920 duration=0(sec)
May 17 01:48:38 client-18 xinetd[3390]: START: shell pid=956 from=192.168.4.5
May 17 01:48:38 client-18 rshd[957]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:48:38 client-18 xinetd[3390]: EXIT: shell status=0 pid=956 duration=0(sec)
May 17 01:48:38 client-18 xinetd[3390]: START: shell pid=993 from=192.168.4.5
May 17 01:48:38 client-18 rshd[994]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 76: confirm clients recycle inodes properly =========================================== 01:48:38 \(1305622118\)");echo XXRETCODE:$?'
May 17 01:48:38 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 76: confirm clients recycle inodes properly =========================================== 01:48:38 (1305622118)
May 17 01:48:38 client-18 xinetd[3390]: EXIT: shell status=0 pid=993 duration=0(sec)
May 17 01:48:50 client-18 xinetd[3390]: START: shell pid=1028 from=192.168.4.5
May 17 01:48:50 client-18 rshd[1029]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:48:50 client-18 xinetd[3390]: EXIT: shell status=0 pid=1028 duration=0(sec)
May 17 01:48:50 client-18 xinetd[3390]: START: shell pid=1064 from=192.168.4.5
May 17 01:48:50 client-18 rshd[1065]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:48:50 client-18 xinetd[3390]: EXIT: shell status=0 pid=1064 duration=0(sec)
May 17 01:48:50 client-18 xinetd[3390]: START: shell pid=1101 from=192.168.4.5
May 17 01:48:50 client-18 rshd[1102]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 77a: normal checksum read\/write operation ============================================= 01:48:50 \(1305622130\)");echo XXRETCODE:$?'
May 17 01:48:50 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 77a: normal checksum read/write operation ============================================= 01:48:50 (1305622130)
May 17 01:48:50 client-18 xinetd[3390]: EXIT: shell status=0 pid=1101 duration=0(sec)
May 17 01:48:52 client-18 xinetd[3390]: START: shell pid=1136 from=192.168.4.5
May 17 01:48:52 client-18 rshd[1137]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:48:52 client-18 xinetd[3390]: EXIT: shell status=0 pid=1136 duration=0(sec)
May 17 01:48:52 client-18 xinetd[3390]: START: shell pid=1172 from=192.168.4.5
May 17 01:48:52 client-18 rshd[1173]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:48:52 client-18 xinetd[3390]: EXIT: shell status=0 pid=1172 duration=0(sec)
May 17 01:48:52 client-18 xinetd[3390]: START: shell pid=1209 from=192.168.4.5
May 17 01:48:52 client-18 rshd[1210]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 77b: checksum error on client write =================================================== 01:48:52 \(1305622132\)");echo XXRETCODE:$?'
May 17 01:48:53 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 77b: checksum error on client write =================================================== 01:48:52 (1305622132)
May 17 01:48:53 client-18 xinetd[3390]: EXIT: shell status=0 pid=1209 duration=1(sec)
May 17 01:48:53 client-18 xinetd[3390]: START: shell pid=1244 from=192.168.4.5
May 17 01:48:53 client-18 rshd[1245]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:48:53 client-18 xinetd[3390]: EXIT: shell status=0 pid=1244 duration=0(sec)
May 17 01:48:53 client-18 xinetd[3390]: START: shell pid=1280 from=192.168.4.5
May 17 01:48:53 client-18 rshd[1281]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:48:53 client-18 xinetd[3390]: EXIT: shell status=0 pid=1280 duration=0(sec)
May 17 01:48:54 client-18 xinetd[3390]: START: shell pid=1317 from=192.168.4.5
May 17 01:48:54 client-18 rshd[1318]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 77c: checksum error on client read ==================================================== 01:48:54 \(1305622134\)");echo XXRETCODE:$?'
May 17 01:48:54 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 77c: checksum error on client read ==================================================== 01:48:54 (1305622134)
May 17 01:48:54 client-18 xinetd[3390]: EXIT: shell status=0 pid=1317 duration=0(sec)
May 17 01:48:54 client-18 xinetd[3390]: START: shell pid=1352 from=192.168.4.5
May 17 01:48:54 client-18 rshd[1353]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark set checksum type to crc32");echo XXRETCODE:$?'
May 17 01:48:54 client-18 kernel: Lustre: DEBUG MARKER: set checksum type to crc32
May 17 01:48:54 client-18 xinetd[3390]: EXIT: shell status=0 pid=1352 duration=0(sec)
May 17 01:49:01 client-18 xinetd[3390]: START: shell pid=1387 from=192.168.4.5
May 17 01:49:01 client-18 rshd[1388]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark set checksum type to adler");echo XXRETCODE:$?'
May 17 01:49:01 client-18 kernel: Lustre: DEBUG MARKER: set checksum type to adler
May 17 01:49:01 client-18 xinetd[3390]: EXIT: shell status=0 pid=1387 duration=0(sec)
May 17 01:49:08 client-18 xinetd[3390]: START: shell pid=1422 from=192.168.4.5
May 17 01:49:08 client-18 rshd[1423]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark set checksum type to adler");echo XXRETCODE:$?'
May 17 01:49:08 client-18 kernel: Lustre: DEBUG MARKER: set checksum type to adler
May 17 01:49:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=1422 duration=0(sec)
May 17 01:49:08 client-18 xinetd[3390]: START: shell pid=1458 from=192.168.4.5
May 17 01:49:08 client-18 rshd[1459]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:49:09 client-18 xinetd[3390]: EXIT: shell status=0 pid=1458 duration=1(sec)
May 17 01:49:09 client-18 xinetd[3390]: START: shell pid=1494 from=192.168.4.5
May 17 01:49:09 client-18 rshd[1495]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:49:09 client-18 xinetd[3390]: EXIT: shell status=0 pid=1494 duration=0(sec)
May 17 01:49:09 client-18 xinetd[3390]: START: shell pid=1531 from=192.168.4.5
May 17 01:49:09 client-18 rshd[1532]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 77d: checksum error on OST direct write =============================================== 01:49:09 \(1305622149\)");echo XXRETCODE:$?'
May 17 01:49:09 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 77d: checksum error on OST direct write =============================================== 01:49:09 (1305622149)
May 17 01:49:09 client-18 xinetd[3390]: EXIT: shell status=0 pid=1531 duration=0(sec)
May 17 01:49:16 client-18 xinetd[3390]: START: shell pid=1566 from=192.168.4.5
May 17 01:49:16 client-18 rshd[1567]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:49:16 client-18 xinetd[3390]: EXIT: shell status=0 pid=1566 duration=0(sec)
May 17 01:49:16 client-18 xinetd[3390]: START: shell pid=1602 from=192.168.4.5
May 17 01:49:16 client-18 rshd[1603]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:49:16 client-18 xinetd[3390]: EXIT: shell status=0 pid=1602 duration=0(sec)
May 17 01:49:16 client-18 xinetd[3390]: START: shell pid=1639 from=192.168.4.5
May 17 01:49:16 client-18 rshd[1640]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 77e: checksum error on OST direct read ================================================ 01:49:16 \(1305622156\)");echo XXRETCODE:$?'
May 17 01:49:16 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 77e: checksum error on OST direct read ================================================ 01:49:16 (1305622156)
May 17 01:49:16 client-18 xinetd[3390]: EXIT: shell status=0 pid=1639 duration=0(sec)
May 17 01:49:18 client-18 xinetd[3390]: START: shell pid=1675 from=192.168.4.5
May 17 01:49:18 client-18 rshd[1676]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:49:18 client-18 xinetd[3390]: EXIT: shell status=0 pid=1675 duration=0(sec)
May 17 01:49:18 client-18 xinetd[3390]: START: shell pid=1713 from=192.168.4.5
May 17 01:49:18 client-18 rshd[1714]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:49:18 client-18 xinetd[3390]: EXIT: shell status=0 pid=1713 duration=0(sec)
May 17 01:49:18 client-18 xinetd[3390]: START: shell pid=1750 from=192.168.4.5
May 17 01:49:18 client-18 rshd[1751]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 77f: repeat checksum error on write \(expect error\) ==================================== 01:49:18 \(1305622158\)");echo XXRETCODE:$?'
May 17 01:49:18 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 77f: repeat checksum error on write (expect error) ==================================== 01:49:18 (1305622158)
May 17 01:49:18 client-18 xinetd[3390]: EXIT: shell status=0 pid=1750 duration=0(sec)
May 17 01:49:18 client-18 xinetd[3390]: START: shell pid=1785 from=192.168.4.5
May 17 01:49:18 client-18 rshd[1786]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark set checksum type to crc32");echo XXRETCODE:$?'
May 17 01:49:18 client-18 kernel: Lustre: DEBUG MARKER: set checksum type to crc32
May 17 01:49:18 client-18 xinetd[3390]: EXIT: shell status=0 pid=1785 duration=0(sec)
May 17 01:50:20 client-18 xinetd[3390]: START: shell pid=1824 from=192.168.4.5
May 17 01:50:20 client-18 rshd[1825]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark set checksum type to adler");echo XXRETCODE:$?'
May 17 01:50:20 client-18 kernel: Lustre: DEBUG MARKER: set checksum type to adler
May 17 01:50:20 client-18 xinetd[3390]: EXIT: shell status=0 pid=1824 duration=0(sec)
May 17 01:51:17 client-18 xinetd[3390]: START: shell pid=1863 from=192.168.4.5
May 17 01:51:17 client-18 rshd[1864]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark set checksum type to adler");echo XXRETCODE:$?'
May 17 01:51:17 client-18 kernel: Lustre: DEBUG MARKER: set checksum type to adler
May 17 01:51:17 client-18 xinetd[3390]: EXIT: shell status=0 pid=1863 duration=0(sec)
May 17 01:51:17 client-18 xinetd[3390]: START: shell pid=1903 from=192.168.4.5
May 17 01:51:17 client-18 rshd[1904]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:51:17 client-18 xinetd[3390]: EXIT: shell status=0 pid=1903 duration=0(sec)
May 17 01:51:17 client-18 xinetd[3390]: START: shell pid=1939 from=192.168.4.5
May 17 01:51:17 client-18 rshd[1940]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:51:17 client-18 xinetd[3390]: EXIT: shell status=0 pid=1939 duration=0(sec)
May 17 01:51:17 client-18 xinetd[3390]: START: shell pid=1976 from=192.168.4.5
May 17 01:51:17 client-18 rshd[1977]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 77g: checksum error on OST write ====================================================== 01:51:17 \(1305622277\)");echo XXRETCODE:$?'
May 17 01:51:17 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 77g: checksum error on OST write ====================================================== 01:51:17 (1305622277)
May 17 01:51:17 client-18 xinetd[3390]: EXIT: shell status=0 pid=1976 duration=0(sec)
May 17 01:51:24 client-18 xinetd[3390]: START: shell pid=2011 from=192.168.4.5
May 17 01:51:24 client-18 rshd[2012]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:51:24 client-18 xinetd[3390]: EXIT: shell status=0 pid=2011 duration=0(sec)
May 17 01:51:24 client-18 xinetd[3390]: START: shell pid=2048 from=192.168.4.5
May 17 01:51:24 client-18 rshd[2049]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:51:24 client-18 xinetd[3390]: EXIT: shell status=0 pid=2048 duration=0(sec)
May 17 01:51:24 client-18 xinetd[3390]: START: shell pid=2085 from=192.168.4.5
May 17 01:51:24 client-18 rshd[2086]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 77h: checksum error on OST read ======================================================= 01:51:24 \(1305622284\)");echo XXRETCODE:$?'
May 17 01:51:24 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 77h: checksum error on OST read ======================================================= 01:51:24 (1305622284)
May 17 01:51:24 client-18 xinetd[3390]: EXIT: shell status=0 pid=2085 duration=0(sec)
May 17 01:51:32 client-18 xinetd[3390]: START: shell pid=2121 from=192.168.4.5
May 17 01:51:32 client-18 rshd[2122]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:51:32 client-18 xinetd[3390]: EXIT: shell status=0 pid=2121 duration=0(sec)
May 17 01:51:32 client-18 xinetd[3390]: START: shell pid=2158 from=192.168.4.5
May 17 01:51:32 client-18 rshd[2159]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:51:32 client-18 xinetd[3390]: EXIT: shell status=0 pid=2158 duration=0(sec)
May 17 01:51:32 client-18 xinetd[3390]: START: shell pid=2199 from=192.168.4.5
May 17 01:51:32 client-18 rshd[2200]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 77i: client not supporting OSD_CONNECT_CKSUM ========================================== 01:51:32 \(1305622292\)");echo XXRETCODE:$?'
May 17 01:51:32 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 77i: client not supporting OSD_CONNECT_CKSUM ========================================== 01:51:32 (1305622292)
May 17 01:51:32 client-18 xinetd[3390]: EXIT: shell status=0 pid=2199 duration=0(sec)
May 17 01:51:34 client-18 xinetd[3390]: START: shell pid=2234 from=192.168.4.5
May 17 01:51:34 client-18 rshd[2235]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:51:34 client-18 xinetd[3390]: EXIT: shell status=0 pid=2234 duration=0(sec)
May 17 01:51:34 client-18 xinetd[3390]: START: shell pid=2278 from=192.168.4.5
May 17 01:51:34 client-18 rshd[2279]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:51:34 client-18 xinetd[3390]: EXIT: shell status=0 pid=2278 duration=0(sec)
May 17 01:51:34 client-18 xinetd[3390]: START: shell pid=2315 from=192.168.4.5
May 17 01:51:34 client-18 rshd[2316]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 77j: client only supporting ADLER32 =================================================== 01:51:34 \(1305622294\)");echo XXRETCODE:$?'
May 17 01:51:34 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 77j: client only supporting ADLER32 =================================================== 01:51:34 (1305622294)
May 17 01:51:34 client-18 xinetd[3390]: EXIT: shell status=0 pid=2315 duration=0(sec)
May 17 01:51:37 client-18 xinetd[3390]: START: shell pid=2350 from=192.168.4.5
May 17 01:51:37 client-18 rshd[2351]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:51:37 client-18 xinetd[3390]: EXIT: shell status=0 pid=2350 duration=0(sec)
May 17 01:51:37 client-18 xinetd[3390]: START: shell pid=2387 from=192.168.4.5
May 17 01:51:37 client-18 rshd[2389]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:51:38 client-18 xinetd[3390]: EXIT: shell status=0 pid=2387 duration=1(sec)
May 17 01:51:38 client-18 xinetd[3390]: START: shell pid=2434 from=192.168.4.5
May 17 01:51:38 client-18 rshd[2435]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 78: handle large O_DIRECT writes correctly ============================================ 01:51:38 \(1305622298\)");echo XXRETCODE:$?'
May 17 01:51:38 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 78: handle large O_DIRECT writes correctly ============================================ 01:51:38 (1305622298)
May 17 01:51:38 client-18 xinetd[3390]: EXIT: shell status=0 pid=2434 duration=0(sec)
May 17 01:52:04 client-18 xinetd[3390]: START: shell pid=2469 from=192.168.4.5
May 17 01:52:04 client-18 rshd[2470]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:52:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=2469 duration=0(sec)
May 17 01:52:04 client-18 xinetd[3390]: START: shell pid=2506 from=192.168.4.5
May 17 01:52:04 client-18 rshd[2507]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:52:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=2506 duration=0(sec)
May 17 01:52:04 client-18 xinetd[3390]: START: shell pid=2548 from=192.168.4.5
May 17 01:52:04 client-18 rshd[2549]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 79: df report consistency check ======================================================= 01:52:04 \(1305622324\)");echo XXRETCODE:$?'
May 17 01:52:04 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 79: df report consistency check ======================================================= 01:52:04 (1305622324)
May 17 01:52:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=2548 duration=0(sec)
May 17 01:52:05 client-18 xinetd[3390]: START: shell pid=2583 from=192.168.4.5
May 17 01:52:05 client-18 rshd[2584]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:52:05 client-18 xinetd[3390]: EXIT: shell status=0 pid=2583 duration=0(sec)
May 17 01:52:05 client-18 xinetd[3390]: START: shell pid=2619 from=192.168.4.5
May 17 01:52:05 client-18 rshd[2620]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:52:05 client-18 xinetd[3390]: EXIT: shell status=0 pid=2619 duration=0(sec)
May 17 01:52:05 client-18 xinetd[3390]: START: shell pid=2656 from=192.168.4.5
May 17 01:52:05 client-18 rshd[2657]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 80: Page eviction is equally fast at high offsets too  ================================ 01:52:05 \(1305622325\)");echo XXRETCODE:$?'
May 17 01:52:06 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 80: Page eviction is equally fast at high offsets too ================================ 01:52:05 (1305622325)
May 17 01:52:06 client-18 xinetd[3390]: EXIT: shell status=0 pid=2656 duration=1(sec)
May 17 01:52:07 client-18 xinetd[3390]: START: shell pid=2691 from=192.168.4.5
May 17 01:52:07 client-18 rshd[2692]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:52:07 client-18 xinetd[3390]: EXIT: shell status=0 pid=2691 duration=0(sec)
May 17 01:52:07 client-18 xinetd[3390]: START: shell pid=2727 from=192.168.4.5
May 17 01:52:07 client-18 rshd[2728]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:52:07 client-18 xinetd[3390]: EXIT: shell status=0 pid=2727 duration=0(sec)
May 17 01:52:07 client-18 xinetd[3390]: START: shell pid=2764 from=192.168.4.5
May 17 01:52:07 client-18 rshd[2765]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 99a: cvs init ========================================================================= 01:52:07 \(1305622327\)");echo XXRETCODE:$?'
May 17 01:52:07 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 99a: cvs init ========================================================================= 01:52:07 (1305622327)
May 17 01:52:07 client-18 xinetd[3390]: EXIT: shell status=0 pid=2764 duration=0(sec)
May 17 01:52:08 client-18 xinetd[3390]: START: shell pid=2799 from=192.168.4.5
May 17 01:52:08 client-18 rshd[2800]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:52:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=2799 duration=0(sec)
May 17 01:52:08 client-18 xinetd[3390]: START: shell pid=2835 from=192.168.4.5
May 17 01:52:08 client-18 rshd[2836]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:52:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=2835 duration=0(sec)
May 17 01:52:08 client-18 xinetd[3390]: START: shell pid=2872 from=192.168.4.5
May 17 01:52:08 client-18 rshd[2873]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 99b: cvs import ======================================================================= 01:52:08 \(1305622328\)");echo XXRETCODE:$?'
May 17 01:52:08 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 99b: cvs import ======================================================================= 01:52:08 (1305622328)
May 17 01:52:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=2872 duration=0(sec)
May 17 01:52:09 client-18 xinetd[3390]: START: shell pid=2907 from=192.168.4.5
May 17 01:52:09 client-18 rshd[2908]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:52:09 client-18 xinetd[3390]: EXIT: shell status=0 pid=2907 duration=0(sec)
May 17 01:52:09 client-18 xinetd[3390]: START: shell pid=2943 from=192.168.4.5
May 17 01:52:09 client-18 rshd[2944]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:52:09 client-18 xinetd[3390]: EXIT: shell status=0 pid=2943 duration=0(sec)
May 17 01:52:10 client-18 xinetd[3390]: START: shell pid=2980 from=192.168.4.5
May 17 01:52:10 client-18 rshd[2981]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 99c: cvs checkout ===================================================================== 01:52:10 \(1305622330\)");echo XXRETCODE:$?'
May 17 01:52:10 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 99c: cvs checkout ===================================================================== 01:52:10 (1305622330)
May 17 01:52:10 client-18 xinetd[3390]: EXIT: shell status=0 pid=2980 duration=0(sec)
May 17 01:52:13 client-18 xinetd[3390]: START: shell pid=3021 from=192.168.4.5
May 17 01:52:13 client-18 rshd[3022]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:52:13 client-18 xinetd[3390]: EXIT: shell status=0 pid=3021 duration=0(sec)
May 17 01:52:13 client-18 xinetd[3390]: START: shell pid=3057 from=192.168.4.5
May 17 01:52:13 client-18 rshd[3058]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:52:13 client-18 xinetd[3390]: EXIT: shell status=0 pid=3057 duration=0(sec)
May 17 01:52:13 client-18 xinetd[3390]: START: shell pid=3099 from=192.168.4.5
May 17 01:52:13 client-18 rshd[3100]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 99d: cvs add ========================================================================== 01:52:13 \(1305622333\)");echo XXRETCODE:$?'
May 17 01:52:13 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 99d: cvs add ========================================================================== 01:52:13 (1305622333)
May 17 01:52:13 client-18 xinetd[3390]: EXIT: shell status=0 pid=3099 duration=0(sec)
May 17 01:52:13 client-18 xinetd[3390]: START: shell pid=3139 from=192.168.4.5
May 17 01:52:13 client-18 rshd[3140]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:52:13 client-18 xinetd[3390]: EXIT: shell status=0 pid=3139 duration=0(sec)
May 17 01:52:13 client-18 xinetd[3390]: START: shell pid=3178 from=192.168.4.5
May 17 01:52:13 client-18 rshd[3179]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:52:13 client-18 xinetd[3390]: EXIT: shell status=0 pid=3178 duration=0(sec)
May 17 01:52:13 client-18 xinetd[3390]: START: shell pid=3217 from=192.168.4.5
May 17 01:52:14 client-18 rshd[3218]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 99e: cvs update ======================================================================= 01:52:13 \(1305622333\)");echo XXRETCODE:$?'
May 17 01:52:14 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 99e: cvs update ======================================================================= 01:52:13 (1305622333)
May 17 01:52:14 client-18 xinetd[3390]: EXIT: shell status=0 pid=3217 duration=1(sec)
May 17 01:52:14 client-18 xinetd[3390]: START: shell pid=3253 from=192.168.4.5
May 17 01:52:14 client-18 rshd[3254]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:52:14 client-18 xinetd[3390]: EXIT: shell status=0 pid=3253 duration=0(sec)
May 17 01:52:14 client-18 xinetd[3390]: START: shell pid=3296 from=192.168.4.5
May 17 01:52:14 client-18 rshd[3297]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:52:14 client-18 xinetd[3390]: EXIT: shell status=0 pid=3296 duration=0(sec)
May 17 01:52:14 client-18 xinetd[3390]: START: shell pid=3334 from=192.168.4.5
May 17 01:52:14 client-18 rshd[3335]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 99f: cvs commit ======================================================================= 01:52:14 \(1305622334\)");echo XXRETCODE:$?'
May 17 01:52:15 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 99f: cvs commit ======================================================================= 01:52:14 (1305622334)
May 17 01:52:15 client-18 xinetd[3390]: EXIT: shell status=0 pid=3334 duration=1(sec)
May 17 01:52:16 client-18 xinetd[3390]: START: shell pid=3379 from=192.168.4.5
May 17 01:52:16 client-18 rshd[3380]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:52:16 client-18 xinetd[3390]: EXIT: shell status=0 pid=3379 duration=0(sec)
May 17 01:52:16 client-18 xinetd[3390]: START: shell pid=3420 from=192.168.4.5
May 17 01:52:16 client-18 rshd[3421]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:52:16 client-18 xinetd[3390]: EXIT: shell status=0 pid=3420 duration=0(sec)
May 17 01:52:16 client-18 xinetd[3390]: START: shell pid=3460 from=192.168.4.5
May 17 01:52:16 client-18 rshd[3461]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 100: check local port using privileged port =========================================== 01:52:16 \(1305622336\)");echo XXRETCODE:$?'
May 17 01:52:16 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 100: check local port using privileged port =========================================== 01:52:16 (1305622336)
May 17 01:52:16 client-18 xinetd[3390]: EXIT: shell status=0 pid=3460 duration=0(sec)
May 17 01:52:16 client-18 xinetd[3390]: START: shell pid=3498 from=192.168.4.5
May 17 01:52:16 client-18 rshd[3499]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark  SKIP: sanity test_100 TCP secure port test, not useful for NETTYPE=o2ib");echo XXRETCODE:$?'
May 17 01:52:16 client-18 kernel: Lustre: DEBUG MARKER: SKIP: sanity test_100 TCP secure port test, not useful for NETTYPE=o2ib
May 17 01:52:16 client-18 xinetd[3390]: EXIT: shell status=0 pid=3498 duration=0(sec)
May 17 01:52:16 client-18 xinetd[3390]: START: shell pid=3542 from=192.168.4.5
May 17 01:52:16 client-18 rshd[3543]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:52:17 client-18 xinetd[3390]: EXIT: shell status=0 pid=3542 duration=1(sec)
May 17 01:52:17 client-18 xinetd[3390]: START: shell pid=3578 from=192.168.4.5
May 17 01:52:17 client-18 rshd[3582]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:52:17 client-18 xinetd[3390]: EXIT: shell status=0 pid=3578 duration=0(sec)
May 17 01:52:17 client-18 xinetd[3390]: START: shell pid=3618 from=192.168.4.5
May 17 01:52:17 client-18 rshd[3619]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 101: check read-ahead for random reads ================================================ 01:52:17 \(1305622337\)");echo XXRETCODE:$?'
May 17 01:52:17 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 101: check read-ahead for random reads ================================================ 01:52:17 (1305622337)
May 17 01:52:17 client-18 xinetd[3390]: EXIT: shell status=0 pid=3618 duration=0(sec)
May 17 01:52:19 client-18 xinetd[3390]: START: shell pid=3657 from=192.168.4.5
May 17 01:52:19 client-18 rshd[3658]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:52:19 client-18 xinetd[3390]: EXIT: shell status=0 pid=3657 duration=0(sec)
May 17 01:52:19 client-18 xinetd[3390]: START: shell pid=3693 from=192.168.4.5
May 17 01:52:19 client-18 rshd[3694]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:52:19 client-18 xinetd[3390]: EXIT: shell status=0 pid=3693 duration=0(sec)
May 17 01:52:19 client-18 xinetd[3390]: START: shell pid=3730 from=192.168.4.5
May 17 01:52:19 client-18 rshd[3731]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 101b: check stride-io mode read-ahead ================================================= 01:52:19 \(1305622339\)");echo XXRETCODE:$?'
May 17 01:52:19 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 101b: check stride-io mode read-ahead ================================================= 01:52:19 (1305622339)
May 17 01:52:19 client-18 xinetd[3390]: EXIT: shell status=0 pid=3730 duration=0(sec)
May 17 01:52:23 client-18 xinetd[3390]: START: shell pid=3765 from=192.168.4.5
May 17 01:52:23 client-18 rshd[3766]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:52:23 client-18 xinetd[3390]: EXIT: shell status=0 pid=3765 duration=0(sec)
May 17 01:52:23 client-18 xinetd[3390]: START: shell pid=3806 from=192.168.4.5
May 17 01:52:23 client-18 rshd[3807]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:52:23 client-18 xinetd[3390]: EXIT: shell status=0 pid=3806 duration=0(sec)
May 17 01:52:23 client-18 xinetd[3390]: START: shell pid=3869 from=192.168.4.5
May 17 01:52:23 client-18 rshd[3870]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 101d: file read with and without read-ahead enabled  ================================== 01:52:23 \(1305622343\)");echo XXRETCODE:$?'
May 17 01:52:24 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 101d: file read with and without read-ahead enabled ================================== 01:52:23 (1305622343)
May 17 01:52:24 client-18 xinetd[3390]: EXIT: shell status=0 pid=3869 duration=1(sec)
May 17 01:53:25 client-18 xinetd[3390]: START: shell pid=3905 from=192.168.4.5
May 17 01:53:25 client-18 rshd[3906]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:53:25 client-18 xinetd[3390]: EXIT: shell status=0 pid=3905 duration=0(sec)
May 17 01:53:25 client-18 xinetd[3390]: START: shell pid=3951 from=192.168.4.5
May 17 01:53:25 client-18 rshd[3952]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:53:25 client-18 xinetd[3390]: EXIT: shell status=0 pid=3951 duration=0(sec)
May 17 01:53:25 client-18 xinetd[3390]: START: shell pid=3988 from=192.168.4.5
May 17 01:53:25 client-18 rshd[3989]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 102a: user xattr test ================================================================= 01:53:25 \(1305622405\)");echo XXRETCODE:$?'
May 17 01:53:25 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 102a: user xattr test ================================================================= 01:53:25 (1305622405)
May 17 01:53:25 client-18 xinetd[3390]: EXIT: shell status=0 pid=3988 duration=0(sec)
May 17 01:53:25 client-18 xinetd[3390]: START: shell pid=4023 from=192.168.4.5
May 17 01:53:25 client-18 rshd[4024]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:53:25 client-18 xinetd[3390]: EXIT: shell status=0 pid=4023 duration=0(sec)
May 17 01:53:25 client-18 xinetd[3390]: START: shell pid=4059 from=192.168.4.5
May 17 01:53:25 client-18 rshd[4060]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:53:25 client-18 xinetd[3390]: EXIT: shell status=0 pid=4059 duration=0(sec)
May 17 01:53:26 client-18 xinetd[3390]: START: shell pid=4096 from=192.168.4.5
May 17 01:53:26 client-18 rshd[4097]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 102b: getfattr\/setfattr for trusted.lov EAs =========================================== 01:53:26 \(1305622406\)");echo XXRETCODE:$?'
May 17 01:53:26 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 102b: getfattr/setfattr for trusted.lov EAs =========================================== 01:53:26 (1305622406)
May 17 01:53:26 client-18 xinetd[3390]: EXIT: shell status=0 pid=4096 duration=0(sec)
May 17 01:53:26 client-18 xinetd[3390]: START: shell pid=4131 from=192.168.4.5
May 17 01:53:26 client-18 rshd[4132]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:53:26 client-18 xinetd[3390]: EXIT: shell status=0 pid=4131 duration=0(sec)
May 17 01:53:26 client-18 xinetd[3390]: START: shell pid=4167 from=192.168.4.5
May 17 01:53:26 client-18 rshd[4168]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:53:26 client-18 xinetd[3390]: EXIT: shell status=0 pid=4167 duration=0(sec)
May 17 01:53:26 client-18 xinetd[3390]: START: shell pid=4204 from=192.168.4.5
May 17 01:53:26 client-18 rshd[4205]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 102c: non-root getfattr\/setfattr for lustre.lov EAs =================================== 01:53:26 \(1305622406\)");echo XXRETCODE:$?'
May 17 01:53:26 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 102c: non-root getfattr/setfattr for lustre.lov EAs =================================== 01:53:26 (1305622406)
May 17 01:53:26 client-18 xinetd[3390]: EXIT: shell status=0 pid=4204 duration=0(sec)
May 17 01:53:26 client-18 xinetd[3390]: START: shell pid=4239 from=192.168.4.5
May 17 01:53:26 client-18 rshd[4240]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:53:27 client-18 xinetd[3390]: EXIT: shell status=0 pid=4239 duration=1(sec)
May 17 01:53:27 client-18 xinetd[3390]: START: shell pid=4275 from=192.168.4.5
May 17 01:53:27 client-18 rshd[4276]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:53:27 client-18 xinetd[3390]: EXIT: shell status=0 pid=4275 duration=0(sec)
May 17 01:53:27 client-18 xinetd[3390]: START: shell pid=4312 from=192.168.4.5
May 17 01:53:27 client-18 rshd[4313]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 102d: tar restore stripe info from tarfile,not keep osts ============================== 01:53:27 \(1305622407\)");echo XXRETCODE:$?'
May 17 01:53:27 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 102d: tar restore stripe info from tarfile,not keep osts ============================== 01:53:27 (1305622407)
May 17 01:53:27 client-18 xinetd[3390]: EXIT: shell status=0 pid=4312 duration=0(sec)
May 17 01:53:27 client-18 xinetd[3390]: START: shell pid=4347 from=192.168.4.5
May 17 01:53:27 client-18 rshd[4348]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark  SKIP: sanity test_102d lustre-aware tar is not installed");echo XXRETCODE:$?'
May 17 01:53:27 client-18 kernel: Lustre: DEBUG MARKER: SKIP: sanity test_102d lustre-aware tar is not installed
May 17 01:53:27 client-18 xinetd[3390]: EXIT: shell status=0 pid=4347 duration=0(sec)
May 17 01:53:27 client-18 xinetd[3390]: START: shell pid=4382 from=192.168.4.5
May 17 01:53:27 client-18 rshd[4383]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:53:27 client-18 xinetd[3390]: EXIT: shell status=0 pid=4382 duration=0(sec)
May 17 01:53:27 client-18 xinetd[3390]: START: shell pid=4418 from=192.168.4.5
May 17 01:53:27 client-18 rshd[4419]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:53:27 client-18 xinetd[3390]: EXIT: shell status=0 pid=4418 duration=0(sec)
May 17 01:53:28 client-18 xinetd[3390]: START: shell pid=4455 from=192.168.4.5
May 17 01:53:28 client-18 rshd[4456]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 102f: tar copy files, not keep osts =================================================== 01:53:27 \(1305622407\)");echo XXRETCODE:$?'
May 17 01:53:28 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 102f: tar copy files, not keep osts =================================================== 01:53:27 (1305622407)
May 17 01:53:28 client-18 xinetd[3390]: EXIT: shell status=0 pid=4455 duration=0(sec)
May 17 01:53:28 client-18 xinetd[3390]: START: shell pid=4490 from=192.168.4.5
May 17 01:53:28 client-18 rshd[4491]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark  SKIP: sanity test_102f lustre-aware tar is not installed");echo XXRETCODE:$?'
May 17 01:53:28 client-18 kernel: Lustre: DEBUG MARKER: SKIP: sanity test_102f lustre-aware tar is not installed
May 17 01:53:28 client-18 xinetd[3390]: EXIT: shell status=0 pid=4490 duration=0(sec)
May 17 01:53:28 client-18 xinetd[3390]: START: shell pid=4525 from=192.168.4.5
May 17 01:53:28 client-18 rshd[4526]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:53:28 client-18 xinetd[3390]: EXIT: shell status=0 pid=4525 duration=0(sec)
May 17 01:53:28 client-18 xinetd[3390]: START: shell pid=4561 from=192.168.4.5
May 17 01:53:28 client-18 rshd[4562]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:53:28 client-18 xinetd[3390]: EXIT: shell status=0 pid=4561 duration=0(sec)
May 17 01:53:28 client-18 xinetd[3390]: START: shell pid=4598 from=192.168.4.5
May 17 01:53:28 client-18 rshd[4599]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 102h: grow xattr from inside inode to external block ================================== 01:53:28 \(1305622408\)");echo XXRETCODE:$?'
May 17 01:53:28 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 102h: grow xattr from inside inode to external block ================================== 01:53:28 (1305622408)
May 17 01:53:28 client-18 xinetd[3390]: EXIT: shell status=0 pid=4598 duration=0(sec)
May 17 01:53:28 client-18 xinetd[3390]: START: shell pid=4633 from=192.168.4.5
May 17 01:53:28 client-18 rshd[4634]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark save trusted.big on \/mnt\/lustre\/f102h");echo XXRETCODE:$?'
May 17 01:53:29 client-18 kernel: Lustre: DEBUG MARKER: save trusted.big on /mnt/lustre/f102h
May 17 01:53:29 client-18 xinetd[3390]: EXIT: shell status=0 pid=4633 duration=1(sec)
May 17 01:53:29 client-18 xinetd[3390]: START: shell pid=4668 from=192.168.4.5
May 17 01:53:29 client-18 rshd[4669]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark save trusted.sml on \/mnt\/lustre\/f102h");echo XXRETCODE:$?'
May 17 01:53:29 client-18 kernel: Lustre: DEBUG MARKER: save trusted.sml on /mnt/lustre/f102h
May 17 01:53:29 client-18 xinetd[3390]: EXIT: shell status=0 pid=4668 duration=0(sec)
May 17 01:53:29 client-18 xinetd[3390]: START: shell pid=4703 from=192.168.4.5
May 17 01:53:29 client-18 rshd[4704]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark grow trusted.sml on \/mnt\/lustre\/f102h");echo XXRETCODE:$?'
May 17 01:53:29 client-18 kernel: Lustre: DEBUG MARKER: grow trusted.sml on /mnt/lustre/f102h
May 17 01:53:29 client-18 xinetd[3390]: EXIT: shell status=0 pid=4703 duration=0(sec)
May 17 01:53:29 client-18 xinetd[3390]: START: shell pid=4738 from=192.168.4.5
May 17 01:53:29 client-18 rshd[4739]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark trusted.big still valid after growing trusted.sml");echo XXRETCODE:$?'
May 17 01:53:29 client-18 kernel: Lustre: DEBUG MARKER: trusted.big still valid after growing trusted.sml
May 17 01:53:29 client-18 xinetd[3390]: EXIT: shell status=0 pid=4738 duration=0(sec)
May 17 01:53:29 client-18 xinetd[3390]: START: shell pid=4773 from=192.168.4.5
May 17 01:53:29 client-18 rshd[4774]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:53:29 client-18 xinetd[3390]: EXIT: shell status=0 pid=4773 duration=0(sec)
May 17 01:53:29 client-18 xinetd[3390]: START: shell pid=4809 from=192.168.4.5
May 17 01:53:29 client-18 rshd[4810]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:53:29 client-18 xinetd[3390]: EXIT: shell status=0 pid=4809 duration=0(sec)
May 17 01:53:29 client-18 xinetd[3390]: START: shell pid=4846 from=192.168.4.5
May 17 01:53:30 client-18 rshd[4847]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 102i: lgetxattr test on symbolic link ================================================= 01:53:29 \(1305622409\)");echo XXRETCODE:$?'
May 17 01:53:30 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 102i: lgetxattr test on symbolic link ================================================= 01:53:29 (1305622409)
May 17 01:53:30 client-18 xinetd[3390]: EXIT: shell status=0 pid=4846 duration=1(sec)
May 17 01:53:30 client-18 xinetd[3390]: START: shell pid=4881 from=192.168.4.5
May 17 01:53:30 client-18 rshd[4882]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:53:30 client-18 xinetd[3390]: EXIT: shell status=0 pid=4881 duration=0(sec)
May 17 01:53:30 client-18 xinetd[3390]: START: shell pid=4917 from=192.168.4.5
May 17 01:53:30 client-18 rshd[4918]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:53:30 client-18 xinetd[3390]: EXIT: shell status=0 pid=4917 duration=0(sec)
May 17 01:53:30 client-18 xinetd[3390]: START: shell pid=4954 from=192.168.4.5
May 17 01:53:30 client-18 rshd[4955]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 102j: non-root tar restore stripe info from tarfile, not keep osts ==================== 01:53:30 \(1305622410\)");echo XXRETCODE:$?'
May 17 01:53:30 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 102j: non-root tar restore stripe info from tarfile, not keep osts ==================== 01:53:30 (1305622410)
May 17 01:53:30 client-18 xinetd[3390]: EXIT: shell status=0 pid=4954 duration=0(sec)
May 17 01:53:30 client-18 xinetd[3390]: START: shell pid=4989 from=192.168.4.5
May 17 01:53:30 client-18 rshd[4990]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark  SKIP: sanity test_102j lustre-aware tar is not installed");echo XXRETCODE:$?'
May 17 01:53:30 client-18 kernel: Lustre: DEBUG MARKER: SKIP: sanity test_102j lustre-aware tar is not installed
May 17 01:53:30 client-18 xinetd[3390]: EXIT: shell status=0 pid=4989 duration=0(sec)
May 17 01:53:30 client-18 xinetd[3390]: START: shell pid=5024 from=192.168.4.5
May 17 01:53:30 client-18 rshd[5025]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:53:31 client-18 xinetd[3390]: EXIT: shell status=0 pid=5024 duration=1(sec)
May 17 01:53:31 client-18 xinetd[3390]: START: shell pid=5060 from=192.168.4.5
May 17 01:53:31 client-18 rshd[5061]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:53:31 client-18 xinetd[3390]: EXIT: shell status=0 pid=5060 duration=0(sec)
May 17 01:53:31 client-18 xinetd[3390]: START: shell pid=5097 from=192.168.4.5
May 17 01:53:31 client-18 rshd[5098]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 102k: setfattr without parameter of value shouldn\'t cause a crash ===================== 01:53:31 \(1305622411\)");echo XXRETCODE:$?'
May 17 01:53:31 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 102k: setfattr without parameter of value shouldn't cause a crash ===================== 01:53:31 (1305622411)
May 17 01:53:31 client-18 xinetd[3390]: EXIT: shell status=0 pid=5097 duration=0(sec)
May 17 01:53:31 client-18 xinetd[3390]: START: shell pid=5132 from=192.168.4.5
May 17 01:53:31 client-18 rshd[5133]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:53:31 client-18 xinetd[3390]: EXIT: shell status=0 pid=5132 duration=0(sec)
May 17 01:53:31 client-18 xinetd[3390]: START: shell pid=5168 from=192.168.4.5
May 17 01:53:31 client-18 rshd[5169]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:53:31 client-18 xinetd[3390]: EXIT: shell status=0 pid=5168 duration=0(sec)
May 17 01:53:31 client-18 xinetd[3390]: START: shell pid=5205 from=192.168.4.5
May 17 01:53:31 client-18 rshd[5206]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 103: acl test ========================================================================= 01:53:31 \(1305622411\)");echo XXRETCODE:$?'
May 17 01:53:31 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 103: acl test ========================================================================= 01:53:31 (1305622411)
May 17 01:53:31 client-18 xinetd[3390]: EXIT: shell status=0 pid=5205 duration=0(sec)
May 17 01:53:42 client-18 xinetd[3390]: START: shell pid=5244 from=192.168.4.5
May 17 01:53:42 client-18 rshd[5245]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:53:42 client-18 xinetd[3390]: EXIT: shell status=0 pid=5244 duration=0(sec)
May 17 01:53:42 client-18 xinetd[3390]: START: shell pid=5280 from=192.168.4.5
May 17 01:53:42 client-18 rshd[5281]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:53:42 client-18 xinetd[3390]: EXIT: shell status=0 pid=5280 duration=0(sec)
May 17 01:53:42 client-18 xinetd[3390]: START: shell pid=5317 from=192.168.4.5
May 17 01:53:42 client-18 rshd[5318]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 104a: lfs df [-ih] [path] test ======================================================== 01:53:42 \(1305622422\)");echo XXRETCODE:$?'
May 17 01:53:42 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 104a: lfs df [-ih] [path] test ======================================================== 01:53:42 (1305622422)
May 17 01:53:42 client-18 xinetd[3390]: EXIT: shell status=0 pid=5317 duration=0(sec)
May 17 01:53:43 client-18 xinetd[3390]: START: shell pid=5352 from=192.168.4.5
May 17 01:53:43 client-18 rshd[5353]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:53:43 client-18 xinetd[3390]: EXIT: shell status=0 pid=5352 duration=0(sec)
May 17 01:53:43 client-18 xinetd[3390]: START: shell pid=5388 from=192.168.4.5
May 17 01:53:43 client-18 rshd[5389]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:53:43 client-18 xinetd[3390]: EXIT: shell status=0 pid=5388 duration=0(sec)
May 17 01:53:43 client-18 xinetd[3390]: START: shell pid=5425 from=192.168.4.5
May 17 01:53:43 client-18 rshd[5426]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 104b: runas -u 500 lfs check servers test ============================================= 01:53:43 \(1305622423\)");echo XXRETCODE:$?'
May 17 01:53:43 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 104b: runas -u 500 lfs check servers test ============================================= 01:53:43 (1305622423)
May 17 01:53:43 client-18 xinetd[3390]: EXIT: shell status=0 pid=5425 duration=0(sec)
May 17 01:53:43 client-18 xinetd[3390]: START: shell pid=5460 from=192.168.4.5
May 17 01:53:43 client-18 rshd[5461]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:53:43 client-18 xinetd[3390]: EXIT: shell status=0 pid=5460 duration=0(sec)
May 17 01:53:43 client-18 xinetd[3390]: START: shell pid=5496 from=192.168.4.5
May 17 01:53:43 client-18 rshd[5497]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:53:43 client-18 xinetd[3390]: EXIT: shell status=0 pid=5496 duration=0(sec)
May 17 01:53:43 client-18 xinetd[3390]: START: shell pid=5533 from=192.168.4.5
May 17 01:53:43 client-18 rshd[5534]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 105a: flock when mounted without -o flock test ======================================== 01:53:43 \(1305622423\)");echo XXRETCODE:$?'
May 17 01:53:43 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 105a: flock when mounted without -o flock test ======================================== 01:53:43 (1305622423)
May 17 01:53:43 client-18 xinetd[3390]: EXIT: shell status=0 pid=5533 duration=0(sec)
May 17 01:53:44 client-18 xinetd[3390]: START: shell pid=5568 from=192.168.4.5
May 17 01:53:44 client-18 rshd[5569]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:53:44 client-18 xinetd[3390]: EXIT: shell status=0 pid=5568 duration=0(sec)
May 17 01:53:44 client-18 xinetd[3390]: START: shell pid=5604 from=192.168.4.5
May 17 01:53:44 client-18 rshd[5605]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:53:44 client-18 xinetd[3390]: EXIT: shell status=0 pid=5604 duration=0(sec)
May 17 01:53:44 client-18 xinetd[3390]: START: shell pid=5641 from=192.168.4.5
May 17 01:53:44 client-18 rshd[5642]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 105b: fcntl when mounted without -o flock test ======================================== 01:53:44 \(1305622424\)");echo XXRETCODE:$?'
May 17 01:53:44 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 105b: fcntl when mounted without -o flock test ======================================== 01:53:44 (1305622424)
May 17 01:53:44 client-18 xinetd[3390]: EXIT: shell status=0 pid=5641 duration=0(sec)
May 17 01:53:44 client-18 xinetd[3390]: START: shell pid=5676 from=192.168.4.5
May 17 01:53:44 client-18 rshd[5677]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:53:44 client-18 xinetd[3390]: EXIT: shell status=0 pid=5676 duration=0(sec)
May 17 01:53:44 client-18 xinetd[3390]: START: shell pid=5712 from=192.168.4.5
May 17 01:53:44 client-18 rshd[5713]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:53:44 client-18 xinetd[3390]: EXIT: shell status=0 pid=5712 duration=0(sec)
May 17 01:53:44 client-18 xinetd[3390]: START: shell pid=5749 from=192.168.4.5
May 17 01:53:44 client-18 rshd[5750]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 105c: lockf when mounted without -o flock test ======================================== 01:53:44 \(1305622424\)");echo XXRETCODE:$?'
May 17 01:53:45 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 105c: lockf when mounted without -o flock test ======================================== 01:53:44 (1305622424)
May 17 01:53:45 client-18 xinetd[3390]: EXIT: shell status=0 pid=5749 duration=1(sec)
May 17 01:53:45 client-18 xinetd[3390]: START: shell pid=5784 from=192.168.4.5
May 17 01:53:45 client-18 rshd[5785]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:53:45 client-18 xinetd[3390]: EXIT: shell status=0 pid=5784 duration=0(sec)
May 17 01:53:45 client-18 xinetd[3390]: START: shell pid=5820 from=192.168.4.5
May 17 01:53:45 client-18 rshd[5821]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:53:45 client-18 xinetd[3390]: EXIT: shell status=0 pid=5820 duration=0(sec)
May 17 01:53:45 client-18 xinetd[3390]: START: shell pid=5857 from=192.168.4.5
May 17 01:53:45 client-18 rshd[5858]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 105d: flock race \(should not freeze\) ================================================== 01:53:45 \(1305622425\)");echo XXRETCODE:$?'
May 17 01:53:45 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 105d: flock race (should not freeze) ================================================== 01:53:45 (1305622425)
May 17 01:53:45 client-18 xinetd[3390]: EXIT: shell status=0 pid=5857 duration=0(sec)
May 17 01:53:55 client-18 xinetd[3390]: START: shell pid=5892 from=192.168.4.5
May 17 01:53:55 client-18 rshd[5893]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:53:55 client-18 xinetd[3390]: EXIT: shell status=0 pid=5892 duration=0(sec)
May 17 01:53:55 client-18 xinetd[3390]: START: shell pid=5928 from=192.168.4.5
May 17 01:53:55 client-18 rshd[5929]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:53:55 client-18 xinetd[3390]: EXIT: shell status=0 pid=5928 duration=0(sec)
May 17 01:53:56 client-18 xinetd[3390]: START: shell pid=5965 from=192.168.4.5
May 17 01:53:56 client-18 rshd[5966]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 105e: Two conflicting flocks from same process ======================================== 01:53:56 \(1305622436\)");echo XXRETCODE:$?'
May 17 01:53:56 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 105e: Two conflicting flocks from same process ======================================== 01:53:56 (1305622436)
May 17 01:53:56 client-18 xinetd[3390]: EXIT: shell status=0 pid=5965 duration=0(sec)
May 17 01:53:56 client-18 xinetd[3390]: START: shell pid=6000 from=192.168.4.5
May 17 01:53:56 client-18 rshd[6001]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:53:56 client-18 xinetd[3390]: EXIT: shell status=0 pid=6000 duration=0(sec)
May 17 01:53:56 client-18 xinetd[3390]: START: shell pid=6036 from=192.168.4.5
May 17 01:53:56 client-18 rshd[6037]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:53:56 client-18 xinetd[3390]: EXIT: shell status=0 pid=6036 duration=0(sec)
May 17 01:53:56 client-18 xinetd[3390]: START: shell pid=6073 from=192.168.4.5
May 17 01:53:56 client-18 rshd[6074]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 106: attempt exec of dir followed by chown of that dir ================================ 01:53:56 \(1305622436\)");echo XXRETCODE:$?'
May 17 01:53:56 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 106: attempt exec of dir followed by chown of that dir ================================ 01:53:56 (1305622436)
May 17 01:53:56 client-18 xinetd[3390]: EXIT: shell status=0 pid=6073 duration=0(sec)
May 17 01:53:56 client-18 xinetd[3390]: START: shell pid=6108 from=192.168.4.5
May 17 01:53:56 client-18 rshd[6109]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:53:56 client-18 xinetd[3390]: EXIT: shell status=0 pid=6108 duration=0(sec)
May 17 01:53:56 client-18 xinetd[3390]: START: shell pid=6144 from=192.168.4.5
May 17 01:53:56 client-18 rshd[6145]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:53:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=6144 duration=1(sec)
May 17 01:53:57 client-18 xinetd[3390]: START: shell pid=6181 from=192.168.4.5
May 17 01:53:57 client-18 rshd[6182]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 107: Coredump on SIG ================================================================== 01:53:57 \(1305622437\)");echo XXRETCODE:$?'
May 17 01:53:57 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 107: Coredump on SIG ================================================================== 01:53:57 (1305622437)
May 17 01:53:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=6181 duration=0(sec)
May 17 01:53:58 client-18 xinetd[3390]: START: shell pid=6216 from=192.168.4.5
May 17 01:53:58 client-18 rshd[6217]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:53:58 client-18 xinetd[3390]: EXIT: shell status=0 pid=6216 duration=0(sec)
May 17 01:53:58 client-18 xinetd[3390]: START: shell pid=6252 from=192.168.4.5
May 17 01:53:58 client-18 rshd[6253]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:53:58 client-18 xinetd[3390]: EXIT: shell status=0 pid=6252 duration=0(sec)
May 17 01:53:58 client-18 xinetd[3390]: START: shell pid=6289 from=192.168.4.5
May 17 01:53:58 client-18 rshd[6290]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 110: filename length checking ========================================================= 01:53:58 \(1305622438\)");echo XXRETCODE:$?'
May 17 01:53:58 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 110: filename length checking ========================================================= 01:53:58 (1305622438)
May 17 01:53:58 client-18 xinetd[3390]: EXIT: shell status=0 pid=6289 duration=0(sec)
May 17 01:53:58 client-18 xinetd[3390]: START: shell pid=6324 from=192.168.4.5
May 17 01:53:58 client-18 rshd[6325]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:53:59 client-18 xinetd[3390]: EXIT: shell status=0 pid=6324 duration=1(sec)
May 17 01:53:59 client-18 xinetd[3390]: START: shell pid=6360 from=192.168.4.5
May 17 01:53:59 client-18 rshd[6361]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:53:59 client-18 xinetd[3390]: EXIT: shell status=0 pid=6360 duration=0(sec)
May 17 01:53:59 client-18 xinetd[3390]: START: shell pid=6397 from=192.168.4.5
May 17 01:53:59 client-18 rshd[6398]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 115: verify dynamic thread creation==================================================== 01:53:59 \(1305622439\)");echo XXRETCODE:$?'
May 17 01:53:59 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 115: verify dynamic thread creation==================================================== 01:53:59 (1305622439)
May 17 01:53:59 client-18 xinetd[3390]: EXIT: shell status=0 pid=6397 duration=0(sec)
May 17 01:53:59 client-18 xinetd[3390]: START: shell pid=6432 from=192.168.4.5
May 17 01:53:59 client-18 rshd[6433]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark  SKIP: sanity test_115 no OSS threads");echo XXRETCODE:$?'
May 17 01:53:59 client-18 kernel: Lustre: DEBUG MARKER: SKIP: sanity test_115 no OSS threads
May 17 01:53:59 client-18 xinetd[3390]: EXIT: shell status=0 pid=6432 duration=0(sec)
May 17 01:53:59 client-18 xinetd[3390]: START: shell pid=6467 from=192.168.4.5
May 17 01:53:59 client-18 rshd[6468]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:53:59 client-18 xinetd[3390]: EXIT: shell status=0 pid=6467 duration=0(sec)
May 17 01:53:59 client-18 xinetd[3390]: START: shell pid=6503 from=192.168.4.5
May 17 01:53:59 client-18 rshd[6504]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:53:59 client-18 xinetd[3390]: EXIT: shell status=0 pid=6503 duration=0(sec)
May 17 01:53:59 client-18 xinetd[3390]: START: shell pid=6540 from=192.168.4.5
May 17 01:54:00 client-18 rshd[6541]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 116: stripe QOS: free space balance =================================================== 01:53:59 \(1305622439\)");echo XXRETCODE:$?'
May 17 01:54:00 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 116: stripe QOS: free space balance =================================================== 01:53:59 (1305622439)
May 17 01:54:00 client-18 xinetd[3390]: EXIT: shell status=0 pid=6540 duration=1(sec)
May 17 01:54:01 client-18 xinetd[3390]: START: shell pid=6575 from=192.168.4.5
May 17 01:54:01 client-18 rshd[6576]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark  SKIP: sanity test_116 too much free space in OST0, skip");echo XXRETCODE:$?'
May 17 01:54:01 client-18 kernel: Lustre: DEBUG MARKER: SKIP: sanity test_116 too much free space in OST0, skip
May 17 01:54:01 client-18 xinetd[3390]: EXIT: shell status=0 pid=6575 duration=0(sec)
May 17 01:54:01 client-18 xinetd[3390]: START: shell pid=6610 from=192.168.4.5
May 17 01:54:01 client-18 rshd[6611]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:54:01 client-18 xinetd[3390]: EXIT: shell status=0 pid=6610 duration=0(sec)
May 17 01:54:01 client-18 xinetd[3390]: START: shell pid=6646 from=192.168.4.5
May 17 01:54:01 client-18 rshd[6647]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:54:01 client-18 xinetd[3390]: EXIT: shell status=0 pid=6646 duration=0(sec)
May 17 01:54:01 client-18 xinetd[3390]: START: shell pid=6683 from=192.168.4.5
May 17 01:54:01 client-18 rshd[6684]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 117: verify fsfilt_extend ============================================================= 01:54:01 \(1305622441\)");echo XXRETCODE:$?'
May 17 01:54:01 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 117: verify fsfilt_extend ============================================================= 01:54:01 (1305622441)
May 17 01:54:01 client-18 xinetd[3390]: EXIT: shell status=0 pid=6683 duration=0(sec)
May 17 01:54:01 client-18 xinetd[3390]: START: shell pid=6718 from=192.168.4.5
May 17 01:54:01 client-18 rshd[6719]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:54:01 client-18 xinetd[3390]: EXIT: shell status=0 pid=6718 duration=0(sec)
May 17 01:54:02 client-18 xinetd[3390]: START: shell pid=6754 from=192.168.4.5
May 17 01:54:02 client-18 rshd[6755]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:54:02 client-18 xinetd[3390]: EXIT: shell status=0 pid=6754 duration=0(sec)
May 17 01:54:02 client-18 xinetd[3390]: START: shell pid=6791 from=192.168.4.5
May 17 01:54:02 client-18 rshd[6792]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 118a: verify O_SYNC works ============================================================= 01:54:02 \(1305622442\)");echo XXRETCODE:$?'
May 17 01:54:02 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 118a: verify O_SYNC works ============================================================= 01:54:02 (1305622442)
May 17 01:54:02 client-18 xinetd[3390]: EXIT: shell status=0 pid=6791 duration=0(sec)
May 17 01:54:03 client-18 xinetd[3390]: START: shell pid=6826 from=192.168.4.5
May 17 01:54:03 client-18 rshd[6827]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:54:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=6826 duration=0(sec)
May 17 01:54:03 client-18 xinetd[3390]: START: shell pid=6862 from=192.168.4.5
May 17 01:54:03 client-18 rshd[6863]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:54:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=6862 duration=0(sec)
May 17 01:54:03 client-18 xinetd[3390]: START: shell pid=6899 from=192.168.4.5
May 17 01:54:03 client-18 rshd[6900]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 118b: Reclaim dirty pages on fatal error ============================================== 01:54:03 \(1305622443\)");echo XXRETCODE:$?'
May 17 01:54:04 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 118b: Reclaim dirty pages on fatal error ============================================== 01:54:03 (1305622443)
May 17 01:54:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=6899 duration=1(sec)
May 17 01:54:05 client-18 xinetd[3390]: START: shell pid=6934 from=192.168.4.5
May 17 01:54:05 client-18 rshd[6935]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:54:05 client-18 xinetd[3390]: EXIT: shell status=0 pid=6934 duration=0(sec)
May 17 01:54:05 client-18 xinetd[3390]: START: shell pid=6970 from=192.168.4.5
May 17 01:54:05 client-18 rshd[6971]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:54:05 client-18 xinetd[3390]: EXIT: shell status=0 pid=6970 duration=0(sec)
May 17 01:54:05 client-18 xinetd[3390]: START: shell pid=7007 from=192.168.4.5
May 17 01:54:05 client-18 rshd[7008]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 118c: Fsync blocks on EROFS until dirty pages are flushed ============================= 01:54:05 \(1305622445\)");echo XXRETCODE:$?'
May 17 01:54:05 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 118c: Fsync blocks on EROFS until dirty pages are flushed ============================= 01:54:05 (1305622445)
May 17 01:54:05 client-18 xinetd[3390]: EXIT: shell status=0 pid=7007 duration=0(sec)
May 17 01:54:50 client-18 xinetd[3390]: START: shell pid=7046 from=192.168.4.5
May 17 01:54:50 client-18 rshd[7047]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:54:50 client-18 xinetd[3390]: EXIT: shell status=0 pid=7046 duration=0(sec)
May 17 01:54:50 client-18 xinetd[3390]: START: shell pid=7082 from=192.168.4.5
May 17 01:54:50 client-18 rshd[7083]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:54:50 client-18 xinetd[3390]: EXIT: shell status=0 pid=7082 duration=0(sec)
May 17 01:54:50 client-18 xinetd[3390]: START: shell pid=7119 from=192.168.4.5
May 17 01:54:50 client-18 rshd[7120]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 118d: Fsync validation inject a delay of the bulk ===================================== 01:54:50 \(1305622490\)");echo XXRETCODE:$?'
May 17 01:54:50 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 118d: Fsync validation inject a delay of the bulk ===================================== 01:54:50 (1305622490)
May 17 01:54:50 client-18 xinetd[3390]: EXIT: shell status=0 pid=7119 duration=0(sec)
May 17 01:54:57 client-18 xinetd[3390]: START: shell pid=7154 from=192.168.4.5
May 17 01:54:57 client-18 rshd[7155]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:54:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=7154 duration=0(sec)
May 17 01:54:57 client-18 xinetd[3390]: START: shell pid=7190 from=192.168.4.5
May 17 01:54:57 client-18 rshd[7191]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:54:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=7190 duration=0(sec)
May 17 01:54:57 client-18 xinetd[3390]: START: shell pid=7227 from=192.168.4.5
May 17 01:54:57 client-18 rshd[7228]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 118f: Simulate unrecoverable OSC side error =========================================== 01:54:57 \(1305622497\)");echo XXRETCODE:$?'
May 17 01:54:57 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 118f: Simulate unrecoverable OSC side error =========================================== 01:54:57 (1305622497)
May 17 01:54:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=7227 duration=0(sec)
May 17 01:54:59 client-18 xinetd[3390]: START: shell pid=7262 from=192.168.4.5
May 17 01:54:59 client-18 rshd[7263]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:54:59 client-18 xinetd[3390]: EXIT: shell status=0 pid=7262 duration=0(sec)
May 17 01:54:59 client-18 xinetd[3390]: START: shell pid=7298 from=192.168.4.5
May 17 01:54:59 client-18 rshd[7299]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:54:59 client-18 xinetd[3390]: EXIT: shell status=0 pid=7298 duration=0(sec)
May 17 01:54:59 client-18 xinetd[3390]: START: shell pid=7335 from=192.168.4.5
May 17 01:54:59 client-18 rshd[7336]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 118g: Don\'t stay in wait if we got local -ENOMEM  ===================================== 01:54:59 \(1305622499\)");echo XXRETCODE:$?'
May 17 01:54:59 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 118g: Don't stay in wait if we got local -ENOMEM ===================================== 01:54:59 (1305622499)
May 17 01:54:59 client-18 xinetd[3390]: EXIT: shell status=0 pid=7335 duration=0(sec)
May 17 01:55:01 client-18 xinetd[3390]: START: shell pid=7370 from=192.168.4.5
May 17 01:55:01 client-18 rshd[7371]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:55:01 client-18 xinetd[3390]: EXIT: shell status=0 pid=7370 duration=0(sec)
May 17 01:55:01 client-18 xinetd[3390]: START: shell pid=7406 from=192.168.4.5
May 17 01:55:01 client-18 rshd[7407]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:55:01 client-18 xinetd[3390]: EXIT: shell status=0 pid=7406 duration=0(sec)
May 17 01:55:01 client-18 xinetd[3390]: START: shell pid=7443 from=192.168.4.5
May 17 01:55:01 client-18 rshd[7444]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 118h: Verify timeout in handling recoverables errors  ================================= 01:55:01 \(1305622501\)");echo XXRETCODE:$?'
May 17 01:55:01 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 118h: Verify timeout in handling recoverables errors ================================= 01:55:01 (1305622501)
May 17 01:55:01 client-18 xinetd[3390]: EXIT: shell status=0 pid=7443 duration=0(sec)
May 17 01:56:41 client-18 xinetd[3390]: START: shell pid=7482 from=192.168.4.5
May 17 01:56:41 client-18 rshd[7483]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:56:41 client-18 xinetd[3390]: EXIT: shell status=0 pid=7482 duration=0(sec)
May 17 01:56:41 client-18 xinetd[3390]: START: shell pid=7518 from=192.168.4.5
May 17 01:56:41 client-18 rshd[7519]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:56:41 client-18 xinetd[3390]: EXIT: shell status=0 pid=7518 duration=0(sec)
May 17 01:56:41 client-18 xinetd[3390]: START: shell pid=7555 from=192.168.4.5
May 17 01:56:41 client-18 rshd[7556]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 118i: Fix error before timeout in recoverable error  ================================== 01:56:41 \(1305622601\)");echo XXRETCODE:$?'
May 17 01:56:41 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 118i: Fix error before timeout in recoverable error ================================== 01:56:41 (1305622601)
May 17 01:56:41 client-18 xinetd[3390]: EXIT: shell status=0 pid=7555 duration=0(sec)
May 17 01:57:27 client-18 xinetd[3390]: START: shell pid=7594 from=192.168.4.5
May 17 01:57:27 client-18 rshd[7595]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:57:27 client-18 xinetd[3390]: EXIT: shell status=0 pid=7594 duration=0(sec)
May 17 01:57:27 client-18 xinetd[3390]: START: shell pid=7630 from=192.168.4.5
May 17 01:57:27 client-18 rshd[7631]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:57:27 client-18 xinetd[3390]: EXIT: shell status=0 pid=7630 duration=0(sec)
May 17 01:57:27 client-18 xinetd[3390]: START: shell pid=7667 from=192.168.4.5
May 17 01:57:27 client-18 rshd[7668]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 118j: Simulate unrecoverable OST side error =========================================== 01:57:27 \(1305622647\)");echo XXRETCODE:$?'
May 17 01:57:27 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 118j: Simulate unrecoverable OST side error =========================================== 01:57:27 (1305622647)
May 17 01:57:27 client-18 xinetd[3390]: EXIT: shell status=0 pid=7667 duration=0(sec)
May 17 01:57:30 client-18 xinetd[3390]: START: shell pid=7702 from=192.168.4.5
May 17 01:57:30 client-18 rshd[7703]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:57:30 client-18 xinetd[3390]: EXIT: shell status=0 pid=7702 duration=0(sec)
May 17 01:57:30 client-18 xinetd[3390]: START: shell pid=7738 from=192.168.4.5
May 17 01:57:30 client-18 rshd[7739]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:57:30 client-18 xinetd[3390]: EXIT: shell status=0 pid=7738 duration=0(sec)
May 17 01:57:30 client-18 xinetd[3390]: START: shell pid=7775 from=192.168.4.5
May 17 01:57:30 client-18 rshd[7776]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 118k: bio alloc -ENOMEM and IO TERM handling ========================================== 01:57:30 \(1305622650\)");echo XXRETCODE:$?'
May 17 01:57:30 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 118k: bio alloc -ENOMEM and IO TERM handling ========================================== 01:57:30 (1305622650)
May 17 01:57:30 client-18 xinetd[3390]: EXIT: shell status=0 pid=7775 duration=0(sec)
May 17 01:57:39 client-18 xinetd[3390]: START: shell pid=7810 from=192.168.4.5
May 17 01:57:39 client-18 rshd[7811]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:57:39 client-18 xinetd[3390]: EXIT: shell status=0 pid=7810 duration=0(sec)
May 17 01:57:39 client-18 xinetd[3390]: START: shell pid=7846 from=192.168.4.5
May 17 01:57:39 client-18 rshd[7847]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:57:39 client-18 xinetd[3390]: EXIT: shell status=0 pid=7846 duration=0(sec)
May 17 01:57:39 client-18 xinetd[3390]: START: shell pid=7883 from=192.168.4.5
May 17 01:57:39 client-18 rshd[7884]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 119a: Short directIO read must return actual read amount ============================== 01:57:39 \(1305622659\)");echo XXRETCODE:$?'
May 17 01:57:39 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 119a: Short directIO read must return actual read amount ============================== 01:57:39 (1305622659)
May 17 01:57:39 client-18 xinetd[3390]: EXIT: shell status=0 pid=7883 duration=0(sec)
May 17 01:57:39 client-18 xinetd[3390]: START: shell pid=7918 from=192.168.4.5
May 17 01:57:39 client-18 rshd[7919]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:57:39 client-18 xinetd[3390]: EXIT: shell status=0 pid=7918 duration=0(sec)
May 17 01:57:39 client-18 xinetd[3390]: START: shell pid=7954 from=192.168.4.5
May 17 01:57:39 client-18 rshd[7955]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:57:39 client-18 xinetd[3390]: EXIT: shell status=0 pid=7954 duration=0(sec)
May 17 01:57:39 client-18 xinetd[3390]: START: shell pid=7991 from=192.168.4.5
May 17 01:57:39 client-18 rshd[7992]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 119b: Sparse directIO read must return actual read amount ============================= 01:57:39 \(1305622659\)");echo XXRETCODE:$?'
May 17 01:57:40 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 119b: Sparse directIO read must return actual read amount ============================= 01:57:39 (1305622659)
May 17 01:57:40 client-18 xinetd[3390]: EXIT: shell status=0 pid=7991 duration=1(sec)
May 17 01:57:40 client-18 xinetd[3390]: START: shell pid=8026 from=192.168.4.5
May 17 01:57:40 client-18 rshd[8027]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:57:40 client-18 xinetd[3390]: EXIT: shell status=0 pid=8026 duration=0(sec)
May 17 01:57:40 client-18 xinetd[3390]: START: shell pid=8062 from=192.168.4.5
May 17 01:57:40 client-18 rshd[8063]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:57:40 client-18 xinetd[3390]: EXIT: shell status=0 pid=8062 duration=0(sec)
May 17 01:57:40 client-18 xinetd[3390]: START: shell pid=8099 from=192.168.4.5
May 17 01:57:40 client-18 rshd[8100]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 119c: Testing for direct read hitting hole ============================================ 01:57:40 \(1305622660\)");echo XXRETCODE:$?'
May 17 01:57:40 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 119c: Testing for direct read hitting hole ============================================ 01:57:40 (1305622660)
May 17 01:57:40 client-18 xinetd[3390]: EXIT: shell status=0 pid=8099 duration=0(sec)
May 17 01:57:40 client-18 xinetd[3390]: START: shell pid=8134 from=192.168.4.5
May 17 01:57:40 client-18 rshd[8135]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:57:40 client-18 xinetd[3390]: EXIT: shell status=0 pid=8134 duration=0(sec)
May 17 01:57:40 client-18 xinetd[3390]: START: shell pid=8170 from=192.168.4.5
May 17 01:57:40 client-18 rshd[8171]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:57:41 client-18 xinetd[3390]: EXIT: shell status=0 pid=8170 duration=1(sec)
May 17 01:57:41 client-18 xinetd[3390]: START: shell pid=8207 from=192.168.4.5
May 17 01:57:41 client-18 rshd[8208]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 119d: The DIO path should try to send a new rpc once one is completed ================= 01:57:41 \(1305622661\)");echo XXRETCODE:$?'
May 17 01:57:41 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 119d: The DIO path should try to send a new rpc once one is completed ================= 01:57:41 (1305622661)
May 17 01:57:41 client-18 xinetd[3390]: EXIT: shell status=0 pid=8207 duration=0(sec)
May 17 01:57:42 client-18 xinetd[3390]: START: shell pid=8242 from=192.168.4.5
May 17 01:57:42 client-18 rshd[8243]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark the DIO writes have completed, now wait for the reads \(should not block very long\)");echo XXRETCODE:$?'
May 17 01:57:42 client-18 kernel: Lustre: DEBUG MARKER: the DIO writes have completed, now wait for the reads (should not block very long)
May 17 01:57:42 client-18 xinetd[3390]: EXIT: shell status=0 pid=8242 duration=0(sec)
May 17 01:57:44 client-18 xinetd[3390]: START: shell pid=8277 from=192.168.4.5
May 17 01:57:44 client-18 rshd[8278]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:57:44 client-18 xinetd[3390]: EXIT: shell status=0 pid=8277 duration=0(sec)
May 17 01:57:44 client-18 xinetd[3390]: START: shell pid=8313 from=192.168.4.5
May 17 01:57:44 client-18 rshd[8314]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:57:44 client-18 xinetd[3390]: EXIT: shell status=0 pid=8313 duration=0(sec)
May 17 01:57:44 client-18 xinetd[3390]: START: shell pid=8350 from=192.168.4.5
May 17 01:57:44 client-18 rshd[8351]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 120a: Early Lock Cancel: mkdir test =================================================== 01:57:44 \(1305622664\)");echo XXRETCODE:$?'
May 17 01:57:44 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 120a: Early Lock Cancel: mkdir test =================================================== 01:57:44 (1305622664)
May 17 01:57:44 client-18 xinetd[3390]: EXIT: shell status=0 pid=8350 duration=0(sec)
May 17 01:57:45 client-18 xinetd[3390]: START: shell pid=8385 from=192.168.4.5
May 17 01:57:45 client-18 rshd[8386]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:57:45 client-18 xinetd[3390]: EXIT: shell status=0 pid=8385 duration=0(sec)
May 17 01:57:45 client-18 xinetd[3390]: START: shell pid=8421 from=192.168.4.5
May 17 01:57:45 client-18 rshd[8422]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:57:45 client-18 xinetd[3390]: EXIT: shell status=0 pid=8421 duration=0(sec)
May 17 01:57:45 client-18 xinetd[3390]: START: shell pid=8458 from=192.168.4.5
May 17 01:57:45 client-18 rshd[8459]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 120b: Early Lock Cancel: create test ================================================== 01:57:45 \(1305622665\)");echo XXRETCODE:$?'
May 17 01:57:45 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 120b: Early Lock Cancel: create test ================================================== 01:57:45 (1305622665)
May 17 01:57:45 client-18 xinetd[3390]: EXIT: shell status=0 pid=8458 duration=0(sec)
May 17 01:57:45 client-18 xinetd[3390]: START: shell pid=8493 from=192.168.4.5
May 17 01:57:45 client-18 rshd[8494]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:57:45 client-18 xinetd[3390]: EXIT: shell status=0 pid=8493 duration=0(sec)
May 17 01:57:45 client-18 xinetd[3390]: START: shell pid=8529 from=192.168.4.5
May 17 01:57:45 client-18 rshd[8530]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:57:45 client-18 xinetd[3390]: EXIT: shell status=0 pid=8529 duration=0(sec)
May 17 01:57:46 client-18 xinetd[3390]: START: shell pid=8566 from=192.168.4.5
May 17 01:57:46 client-18 rshd[8567]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 120c: Early Lock Cancel: link test ==================================================== 01:57:46 \(1305622666\)");echo XXRETCODE:$?'
May 17 01:57:46 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 120c: Early Lock Cancel: link test ==================================================== 01:57:46 (1305622666)
May 17 01:57:46 client-18 xinetd[3390]: EXIT: shell status=0 pid=8566 duration=0(sec)
May 17 01:57:46 client-18 xinetd[3390]: START: shell pid=8601 from=192.168.4.5
May 17 01:57:46 client-18 rshd[8602]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:57:46 client-18 xinetd[3390]: EXIT: shell status=0 pid=8601 duration=0(sec)
May 17 01:57:46 client-18 xinetd[3390]: START: shell pid=8637 from=192.168.4.5
May 17 01:57:46 client-18 rshd[8638]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:57:46 client-18 xinetd[3390]: EXIT: shell status=0 pid=8637 duration=0(sec)
May 17 01:57:46 client-18 xinetd[3390]: START: shell pid=8674 from=192.168.4.5
May 17 01:57:46 client-18 rshd[8675]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 120d: Early Lock Cancel: setattr test ================================================= 01:57:46 \(1305622666\)");echo XXRETCODE:$?'
May 17 01:57:46 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 120d: Early Lock Cancel: setattr test ================================================= 01:57:46 (1305622666)
May 17 01:57:46 client-18 xinetd[3390]: EXIT: shell status=0 pid=8674 duration=0(sec)
May 17 01:57:46 client-18 xinetd[3390]: START: shell pid=8709 from=192.168.4.5
May 17 01:57:46 client-18 rshd[8710]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:57:47 client-18 xinetd[3390]: EXIT: shell status=0 pid=8709 duration=1(sec)
May 17 01:57:47 client-18 xinetd[3390]: START: shell pid=8745 from=192.168.4.5
May 17 01:57:47 client-18 rshd[8746]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:57:47 client-18 xinetd[3390]: EXIT: shell status=0 pid=8745 duration=0(sec)
May 17 01:57:47 client-18 xinetd[3390]: START: shell pid=8782 from=192.168.4.5
May 17 01:57:47 client-18 rshd[8783]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 120e: Early Lock Cancel: unlink test ================================================== 01:57:47 \(1305622667\)");echo XXRETCODE:$?'
May 17 01:57:47 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 120e: Early Lock Cancel: unlink test ================================================== 01:57:47 (1305622667)
May 17 01:57:47 client-18 xinetd[3390]: EXIT: shell status=0 pid=8782 duration=0(sec)
May 17 01:57:47 client-18 xinetd[3390]: START: shell pid=8817 from=192.168.4.5
May 17 01:57:47 client-18 rshd[8818]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:57:47 client-18 xinetd[3390]: EXIT: shell status=0 pid=8817 duration=0(sec)
May 17 01:57:47 client-18 xinetd[3390]: START: shell pid=8853 from=192.168.4.5
May 17 01:57:47 client-18 rshd[8854]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:57:47 client-18 xinetd[3390]: EXIT: shell status=0 pid=8853 duration=0(sec)
May 17 01:57:47 client-18 xinetd[3390]: START: shell pid=8890 from=192.168.4.5
May 17 01:57:48 client-18 rshd[8891]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 120f: Early Lock Cancel: rename test ================================================== 01:57:47 \(1305622667\)");echo XXRETCODE:$?'
May 17 01:57:48 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 120f: Early Lock Cancel: rename test ================================================== 01:57:47 (1305622667)
May 17 01:57:48 client-18 xinetd[3390]: EXIT: shell status=0 pid=8890 duration=1(sec)
May 17 01:57:48 client-18 xinetd[3390]: START: shell pid=8925 from=192.168.4.5
May 17 01:57:48 client-18 rshd[8926]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:57:48 client-18 xinetd[3390]: EXIT: shell status=0 pid=8925 duration=0(sec)
May 17 01:57:48 client-18 xinetd[3390]: START: shell pid=8961 from=192.168.4.5
May 17 01:57:48 client-18 rshd[8962]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:57:48 client-18 xinetd[3390]: EXIT: shell status=0 pid=8961 duration=0(sec)
May 17 01:57:48 client-18 xinetd[3390]: START: shell pid=8998 from=192.168.4.5
May 17 01:57:48 client-18 rshd[8999]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 120g: Early Lock Cancel: performance test ============================================= 01:57:48 \(1305622668\)");echo XXRETCODE:$?'
May 17 01:57:48 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 120g: Early Lock Cancel: performance test ============================================= 01:57:48 (1305622668)
May 17 01:57:48 client-18 xinetd[3390]: EXIT: shell status=0 pid=8998 duration=0(sec)
May 17 01:58:36 client-18 xinetd[3390]: START: shell pid=9037 from=192.168.4.5
May 17 01:58:36 client-18 rshd[9038]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:58:36 client-18 xinetd[3390]: EXIT: shell status=0 pid=9037 duration=0(sec)
May 17 01:58:36 client-18 xinetd[3390]: START: shell pid=9073 from=192.168.4.5
May 17 01:58:36 client-18 rshd[9074]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:58:36 client-18 xinetd[3390]: EXIT: shell status=0 pid=9073 duration=0(sec)
May 17 01:58:36 client-18 xinetd[3390]: START: shell pid=9110 from=192.168.4.5
May 17 01:58:36 client-18 rshd[9111]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 121: read cancel race ================================================================= 01:58:36 \(1305622716\)");echo XXRETCODE:$?'
May 17 01:58:36 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 121: read cancel race ================================================================= 01:58:36 (1305622716)
May 17 01:58:36 client-18 xinetd[3390]: EXIT: shell status=0 pid=9110 duration=0(sec)
May 17 01:58:36 client-18 xinetd[3390]: START: shell pid=9145 from=192.168.4.5
May 17 01:58:36 client-18 rshd[9146]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 01:58:36 client-18 xinetd[3390]: EXIT: shell status=0 pid=9145 duration=0(sec)
May 17 01:58:36 client-18 xinetd[3390]: START: shell pid=9181 from=192.168.4.5
May 17 01:58:36 client-18 rshd[9182]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 01:58:36 client-18 xinetd[3390]: EXIT: shell status=0 pid=9181 duration=0(sec)
May 17 01:58:37 client-18 xinetd[3390]: START: shell pid=9218 from=192.168.4.5
May 17 01:58:37 client-18 rshd[9219]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 123a: verify statahead work =========================================================== 01:58:36 \(1305622716\)");echo XXRETCODE:$?'
May 17 01:58:37 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 123a: verify statahead work =========================================================== 01:58:36 (1305622716)
May 17 01:58:37 client-18 xinetd[3390]: EXIT: shell status=0 pid=9218 duration=0(sec)
May 17 01:58:37 client-18 xinetd[3390]: START: shell pid=9253 from=192.168.4.5
May 17 01:58:37 client-18 rshd[9254]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark ls 100 files without statahead: 0 sec");echo XXRETCODE:$?'
May 17 01:58:37 client-18 kernel: Lustre: DEBUG MARKER: ls 100 files without statahead: 0 sec
May 17 01:58:37 client-18 xinetd[3390]: EXIT: shell status=0 pid=9253 duration=0(sec)
May 17 01:58:38 client-18 xinetd[3390]: START: shell pid=9288 from=192.168.4.5
May 17 01:58:38 client-18 rshd[9289]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark ls 100 files with statahead: 1 sec");echo XXRETCODE:$?'
May 17 01:58:38 client-18 kernel: Lustre: DEBUG MARKER: ls 100 files with statahead: 1 sec
May 17 01:58:38 client-18 xinetd[3390]: EXIT: shell status=0 pid=9288 duration=0(sec)
May 17 01:58:42 client-18 xinetd[3390]: START: shell pid=9323 from=192.168.4.5
May 17 01:58:42 client-18 rshd[9324]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark ls 1000 files without statahead: 2 sec");echo XXRETCODE:$?'
May 17 01:58:42 client-18 kernel: Lustre: DEBUG MARKER: ls 1000 files without statahead: 2 sec
May 17 01:58:42 client-18 xinetd[3390]: EXIT: shell status=0 pid=9323 duration=0(sec)
May 17 01:58:44 client-18 xinetd[3390]: START: shell pid=9358 from=192.168.4.5
May 17 01:58:44 client-18 rshd[9359]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark ls 1000 files with statahead: 1 sec");echo XXRETCODE:$?'
May 17 01:58:44 client-18 kernel: Lustre: DEBUG MARKER: ls 1000 files with statahead: 1 sec
May 17 01:58:44 client-18 xinetd[3390]: EXIT: shell status=0 pid=9358 duration=0(sec)
May 17 01:59:27 client-18 xinetd[3390]: START: shell pid=9393 from=192.168.4.5
May 17 01:59:27 client-18 rshd[9394]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark ls 10000 files without statahead: 25 sec");echo XXRETCODE:$?'
May 17 01:59:27 client-18 kernel: Lustre: DEBUG MARKER: ls 10000 files without statahead: 25 sec
May 17 01:59:27 client-18 xinetd[3390]: EXIT: shell status=0 pid=9393 duration=0(sec)
May 17 01:59:40 client-18 xinetd[3390]: START: shell pid=9428 from=192.168.4.5
May 17 01:59:40 client-18 rshd[9429]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark ls 10000 files with statahead: 11 sec");echo XXRETCODE:$?'
May 17 01:59:40 client-18 kernel: Lustre: DEBUG MARKER: ls 10000 files with statahead: 11 sec
May 17 01:59:40 client-18 xinetd[3390]: EXIT: shell status=0 pid=9428 duration=0(sec)
May 17 01:59:41 client-18 xinetd[3390]: START: shell pid=9463 from=192.168.4.5
May 17 01:59:41 client-18 rshd[9464]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark ls done");echo XXRETCODE:$?'
May 17 01:59:41 client-18 kernel: Lustre: DEBUG MARKER: ls done
May 17 01:59:41 client-18 xinetd[3390]: EXIT: shell status=0 pid=9463 duration=0(sec)
May 17 02:00:01 client-18 xinetd[3390]: START: shell pid=9502 from=192.168.4.5
May 17 02:00:01 client-18 rshd[9503]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark rm -r \/mnt\/lustre\/d0.sanity\/d123\/: 20 seconds");echo XXRETCODE:$?'
May 17 02:00:01 client-18 kernel: Lustre: DEBUG MARKER: rm -r /mnt/lustre/d0.sanity/d123/: 20 seconds
May 17 02:00:01 client-18 xinetd[3390]: EXIT: shell status=0 pid=9502 duration=0(sec)
May 17 02:00:01 client-18 xinetd[3390]: START: shell pid=9537 from=192.168.4.5
May 17 02:00:01 client-18 rshd[9538]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark rm done");echo XXRETCODE:$?'
May 17 02:00:02 client-18 kernel: Lustre: DEBUG MARKER: rm done
May 17 02:00:02 client-18 xinetd[3390]: EXIT: shell status=0 pid=9537 duration=1(sec)
May 17 02:00:02 client-18 xinetd[3390]: START: shell pid=9572 from=192.168.4.5
May 17 02:00:02 client-18 rshd[9573]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 02:00:02 client-18 xinetd[3390]: EXIT: shell status=0 pid=9572 duration=0(sec)
May 17 02:00:02 client-18 xinetd[3390]: START: shell pid=9608 from=192.168.4.5
May 17 02:00:02 client-18 rshd[9609]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 02:00:02 client-18 xinetd[3390]: EXIT: shell status=0 pid=9608 duration=0(sec)
May 17 02:00:02 client-18 xinetd[3390]: START: shell pid=9645 from=192.168.4.5
May 17 02:00:02 client-18 rshd[9646]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 123b: not panic with network error in statahead enqueue \(bug 15027\) =================== 02:00:02 \(1305622802\)");echo XXRETCODE:$?'
May 17 02:00:02 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 123b: not panic with network error in statahead enqueue (bug 15027) =================== 02:00:02 (1305622802)
May 17 02:00:02 client-18 xinetd[3390]: EXIT: shell status=0 pid=9645 duration=0(sec)
May 17 02:00:05 client-18 xinetd[3390]: START: shell pid=9680 from=192.168.4.5
May 17 02:00:05 client-18 rshd[9681]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark ls done");echo XXRETCODE:$?'
May 17 02:00:05 client-18 kernel: Lustre: DEBUG MARKER: ls done
May 17 02:00:05 client-18 xinetd[3390]: EXIT: shell status=0 pid=9680 duration=0(sec)
May 17 02:00:07 client-18 xinetd[3390]: START: shell pid=9715 from=192.168.4.5
May 17 02:00:07 client-18 rshd[9716]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 02:00:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=9715 duration=1(sec)
May 17 02:00:08 client-18 xinetd[3390]: START: shell pid=9751 from=192.168.4.5
May 17 02:00:08 client-18 rshd[9752]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 02:00:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=9751 duration=0(sec)
May 17 02:00:08 client-18 xinetd[3390]: START: shell pid=9788 from=192.168.4.5
May 17 02:00:08 client-18 rshd[9789]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 124a: lru resize ====================================================================== 02:00:08 \(1305622808\)");echo XXRETCODE:$?'
May 17 02:00:08 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 124a: lru resize ====================================================================== 02:00:08 (1305622808)
May 17 02:00:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=9788 duration=0(sec)
May 17 02:00:08 client-18 xinetd[3390]: START: shell pid=9823 from=192.168.4.5
May 17 02:00:08 client-18 rshd[9824]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark create 2000 files at \/mnt\/lustre\/d0.sanity\/d124");echo XXRETCODE:$?'
May 17 02:00:08 client-18 kernel: Lustre: DEBUG MARKER: create 2000 files at /mnt/lustre/d0.sanity/d124
May 17 02:00:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=9823 duration=0(sec)
May 17 02:00:15 client-18 xinetd[3390]: START: shell pid=9858 from=192.168.4.5
May 17 02:00:15 client-18 rshd[9859]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark NS=ldlm.namespaces.lustre-MDT0000-mdc-ffff8103201c4000");echo XXRETCODE:$?'
May 17 02:00:15 client-18 kernel: Lustre: DEBUG MARKER: NS=ldlm.namespaces.lustre-MDT0000-mdc-ffff8103201c4000
May 17 02:00:15 client-18 xinetd[3390]: EXIT: shell status=0 pid=9858 duration=0(sec)
May 17 02:00:15 client-18 xinetd[3390]: START: shell pid=9893 from=192.168.4.5
May 17 02:00:15 client-18 rshd[9894]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark LRU=2003");echo XXRETCODE:$?'
May 17 02:00:15 client-18 kernel: Lustre: DEBUG MARKER: LRU=2003
May 17 02:00:15 client-18 xinetd[3390]: EXIT: shell status=0 pid=9893 duration=0(sec)
May 17 02:00:15 client-18 xinetd[3390]: START: shell pid=9928 from=192.168.4.5
May 17 02:00:15 client-18 rshd[9929]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark LVF=766841");echo XXRETCODE:$?'
May 17 02:00:15 client-18 kernel: Lustre: DEBUG MARKER: LVF=766841
May 17 02:00:15 client-18 xinetd[3390]: EXIT: shell status=0 pid=9928 duration=0(sec)
May 17 02:00:15 client-18 xinetd[3390]: START: shell pid=9963 from=192.168.4.5
May 17 02:00:15 client-18 rshd[9964]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark Sleep 50 sec");echo XXRETCODE:$?'
May 17 02:00:15 client-18 kernel: Lustre: DEBUG MARKER: Sleep 50 sec
May 17 02:00:15 client-18 xinetd[3390]: EXIT: shell status=0 pid=9963 duration=0(sec)
May 17 02:01:05 client-18 xinetd[3390]: START: shell pid=10008 from=192.168.4.5
May 17 02:01:05 client-18 rshd[10009]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark Dropped 725 locks in 50s");echo XXRETCODE:$?'
May 17 02:01:05 client-18 kernel: Lustre: DEBUG MARKER: Dropped 725 locks in 50s
May 17 02:01:05 client-18 xinetd[3390]: EXIT: shell status=0 pid=10008 duration=0(sec)
May 17 02:01:05 client-18 xinetd[3390]: START: shell pid=10043 from=192.168.4.5
May 17 02:01:05 client-18 rshd[10044]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark unlink 2000 files at \/mnt\/lustre\/d0.sanity\/d124");echo XXRETCODE:$?'
May 17 02:01:06 client-18 kernel: Lustre: DEBUG MARKER: unlink 2000 files at /mnt/lustre/d0.sanity/d124
May 17 02:01:06 client-18 xinetd[3390]: EXIT: shell status=0 pid=10043 duration=1(sec)
May 17 02:01:09 client-18 xinetd[3390]: START: shell pid=10078 from=192.168.4.5
May 17 02:01:09 client-18 rshd[10079]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 02:01:09 client-18 xinetd[3390]: EXIT: shell status=0 pid=10078 duration=0(sec)
May 17 02:01:09 client-18 xinetd[3390]: START: shell pid=10114 from=192.168.4.5
May 17 02:01:09 client-18 rshd[10115]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 02:01:09 client-18 xinetd[3390]: EXIT: shell status=0 pid=10114 duration=0(sec)
May 17 02:01:09 client-18 xinetd[3390]: START: shell pid=10151 from=192.168.4.5
May 17 02:01:09 client-18 rshd[10152]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 124b: lru resize \(performance test\) =================================================== 02:01:09 \(1305622869\)");echo XXRETCODE:$?'
May 17 02:01:09 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 124b: lru resize (performance test) =================================================== 02:01:09 (1305622869)
May 17 02:01:09 client-18 xinetd[3390]: EXIT: shell status=0 pid=10151 duration=0(sec)
May 17 02:01:25 client-18 xinetd[3390]: START: shell pid=10186 from=192.168.4.5
May 17 02:01:25 client-18 rshd[10187]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark doing ls -la \/mnt\/lustre\/d0.sanity\/d124\/disable_lru_resize 3 times");echo XXRETCODE:$?'
May 17 02:01:25 client-18 kernel: Lustre: DEBUG MARKER: doing ls -la /mnt/lustre/d0.sanity/d124/disable_lru_resize 3 times
May 17 02:01:25 client-18 xinetd[3390]: EXIT: shell status=0 pid=10186 duration=0(sec)
May 17 02:01:51 client-18 xinetd[3390]: START: shell pid=10221 from=192.168.4.5
May 17 02:01:51 client-18 rshd[10222]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark ls -la time: 26 seconds");echo XXRETCODE:$?'
May 17 02:01:51 client-18 kernel: Lustre: DEBUG MARKER: ls -la time: 26 seconds
May 17 02:01:51 client-18 xinetd[3390]: EXIT: shell status=0 pid=10221 duration=0(sec)
May 17 02:01:51 client-18 xinetd[3390]: START: shell pid=10256 from=192.168.4.5
May 17 02:01:51 client-18 rshd[10257]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark lru_size = 400");echo XXRETCODE:$?'
May 17 02:01:51 client-18 kernel: Lustre: DEBUG MARKER: lru_size = 400
May 17 02:01:51 client-18 xinetd[3390]: EXIT: shell status=0 pid=10256 duration=0(sec)
May 17 02:02:28 client-18 xinetd[3390]: START: shell pid=10295 from=192.168.4.5
May 17 02:02:28 client-18 rshd[10296]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark doing ls -la \/mnt\/lustre\/d0.sanity\/d124\/enable_lru_resize 3 times");echo XXRETCODE:$?'
May 17 02:02:28 client-18 kernel: Lustre: DEBUG MARKER: doing ls -la /mnt/lustre/d0.sanity/d124/enable_lru_resize 3 times
May 17 02:02:28 client-18 xinetd[3390]: EXIT: shell status=0 pid=10295 duration=0(sec)
May 17 02:02:36 client-18 xinetd[3390]: START: shell pid=10330 from=192.168.4.5
May 17 02:02:36 client-18 rshd[10331]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark ls -la time: 8 seconds");echo XXRETCODE:$?'
May 17 02:02:36 client-18 kernel: Lustre: DEBUG MARKER: ls -la time: 8 seconds
May 17 02:02:36 client-18 xinetd[3390]: EXIT: shell status=0 pid=10330 duration=0(sec)
May 17 02:02:36 client-18 xinetd[3390]: START: shell pid=10365 from=192.168.4.5
May 17 02:02:36 client-18 rshd[10366]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark lru_size = 8004");echo XXRETCODE:$?'
May 17 02:02:36 client-18 kernel: Lustre: DEBUG MARKER: lru_size = 8004
May 17 02:02:36 client-18 xinetd[3390]: EXIT: shell status=0 pid=10365 duration=0(sec)
May 17 02:02:36 client-18 xinetd[3390]: START: shell pid=10400 from=192.168.4.5
May 17 02:02:36 client-18 rshd[10401]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark ls -la is 69% faster with lru resize enabled");echo XXRETCODE:$?'
May 17 02:02:36 client-18 kernel: Lustre: DEBUG MARKER: ls -la is 69% faster with lru resize enabled
May 17 02:02:36 client-18 xinetd[3390]: EXIT: shell status=0 pid=10400 duration=0(sec)
May 17 02:02:48 client-18 xinetd[3390]: START: shell pid=10435 from=192.168.4.5
May 17 02:02:48 client-18 rshd[10436]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 02:02:48 client-18 xinetd[3390]: EXIT: shell status=0 pid=10435 duration=0(sec)
May 17 02:02:48 client-18 xinetd[3390]: START: shell pid=10471 from=192.168.4.5
May 17 02:02:48 client-18 rshd[10472]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 02:02:48 client-18 xinetd[3390]: EXIT: shell status=0 pid=10471 duration=0(sec)
May 17 02:02:48 client-18 xinetd[3390]: START: shell pid=10508 from=192.168.4.5
May 17 02:02:48 client-18 rshd[10509]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 125: don\'t return EPROTO when a dir has a non-default striping and ACLs =============== 02:02:48 \(1305622968\)");echo XXRETCODE:$?'
May 17 02:02:48 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 125: don't return EPROTO when a dir has a non-default striping and ACLs =============== 02:02:48 (1305622968)
May 17 02:02:48 client-18 xinetd[3390]: EXIT: shell status=0 pid=10508 duration=0(sec)
May 17 02:02:49 client-18 xinetd[3390]: START: shell pid=10543 from=192.168.4.5
May 17 02:02:49 client-18 rshd[10544]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 02:02:49 client-18 xinetd[3390]: EXIT: shell status=0 pid=10543 duration=0(sec)
May 17 02:02:49 client-18 xinetd[3390]: START: shell pid=10579 from=192.168.4.5
May 17 02:02:49 client-18 rshd[10580]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 02:02:49 client-18 xinetd[3390]: EXIT: shell status=0 pid=10579 duration=0(sec)
May 17 02:02:49 client-18 xinetd[3390]: START: shell pid=10616 from=192.168.4.5
May 17 02:02:49 client-18 rshd[10617]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 126: check that the fsgid provided by the client is taken into account ================ 02:02:49 \(1305622969\)");echo XXRETCODE:$?'
May 17 02:02:49 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 126: check that the fsgid provided by the client is taken into account ================ 02:02:49 (1305622969)
May 17 02:02:49 client-18 xinetd[3390]: EXIT: shell status=0 pid=10616 duration=0(sec)
May 17 02:02:49 client-18 xinetd[3390]: START: shell pid=10651 from=192.168.4.5
May 17 02:02:49 client-18 rshd[10652]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 02:02:49 client-18 xinetd[3390]: EXIT: shell status=0 pid=10651 duration=0(sec)
May 17 02:02:49 client-18 xinetd[3390]: START: shell pid=10687 from=192.168.4.5
May 17 02:02:49 client-18 rshd[10688]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 02:02:49 client-18 xinetd[3390]: EXIT: shell status=0 pid=10687 duration=0(sec)
May 17 02:02:49 client-18 xinetd[3390]: START: shell pid=10724 from=192.168.4.5
May 17 02:02:50 client-18 rshd[10725]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 127: verify the client stats are sane ================================================= 02:02:49 \(1305622969\)");echo XXRETCODE:$?'
May 17 02:02:50 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 127: verify the client stats are sane ================================================= 02:02:49 (1305622969)
May 17 02:02:50 client-18 xinetd[3390]: EXIT: shell status=0 pid=10724 duration=1(sec)
May 17 02:02:50 client-18 xinetd[3390]: START: shell pid=10759 from=192.168.4.5
May 17 02:02:50 client-18 rshd[10760]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 02:02:50 client-18 xinetd[3390]: EXIT: shell status=0 pid=10759 duration=0(sec)
May 17 02:02:50 client-18 xinetd[3390]: START: shell pid=10795 from=192.168.4.5
May 17 02:02:50 client-18 rshd[10796]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 02:02:50 client-18 xinetd[3390]: EXIT: shell status=0 pid=10795 duration=0(sec)
May 17 02:02:50 client-18 xinetd[3390]: START: shell pid=10832 from=192.168.4.5
May 17 02:02:50 client-18 rshd[10833]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 128: interactive lfs for 2 consecutive find\'s ========================================= 02:02:50 \(1305622970\)");echo XXRETCODE:$?'
May 17 02:02:50 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 128: interactive lfs for 2 consecutive find's ========================================= 02:02:50 (1305622970)
May 17 02:02:50 client-18 xinetd[3390]: EXIT: shell status=0 pid=10832 duration=0(sec)
May 17 02:02:50 client-18 xinetd[3390]: START: shell pid=10867 from=192.168.4.5
May 17 02:02:50 client-18 rshd[10868]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 02:02:50 client-18 xinetd[3390]: EXIT: shell status=0 pid=10867 duration=0(sec)
May 17 02:02:51 client-18 xinetd[3390]: START: shell pid=10903 from=192.168.4.5
May 17 02:02:51 client-18 rshd[10904]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 02:02:51 client-18 xinetd[3390]: EXIT: shell status=0 pid=10903 duration=0(sec)
May 17 02:02:51 client-18 xinetd[3390]: START: shell pid=10940 from=192.168.4.5
May 17 02:02:51 client-18 rshd[10941]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 129: test directory size limit ======================================================== 02:02:51 \(1305622971\)");echo XXRETCODE:$?'
May 17 02:02:51 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 129: test directory size limit ======================================================== 02:02:51 (1305622971)
May 17 02:02:51 client-18 xinetd[3390]: EXIT: shell status=0 pid=10940 duration=0(sec)
May 17 02:02:53 client-18 xinetd[3390]: START: shell pid=10975 from=192.168.4.5
May 17 02:02:53 client-18 rshd[10976]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 02:02:54 client-18 xinetd[3390]: EXIT: shell status=0 pid=10975 duration=1(sec)
May 17 02:02:54 client-18 xinetd[3390]: START: shell pid=11011 from=192.168.4.5
May 17 02:02:54 client-18 rshd[11012]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 02:02:54 client-18 xinetd[3390]: EXIT: shell status=0 pid=11011 duration=0(sec)
May 17 02:02:54 client-18 xinetd[3390]: START: shell pid=11048 from=192.168.4.5
May 17 02:02:54 client-18 rshd[11049]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 130a: FIEMAP \(1-stripe file\) ========================================================== 02:02:54 \(1305622974\)");echo XXRETCODE:$?'
May 17 02:02:54 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 130a: FIEMAP (1-stripe file) ========================================================== 02:02:54 (1305622974)
May 17 02:02:54 client-18 xinetd[3390]: EXIT: shell status=0 pid=11048 duration=0(sec)
May 17 02:02:54 client-18 xinetd[3390]: START: shell pid=11083 from=192.168.4.5
May 17 02:02:54 client-18 rshd[11084]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark  SKIP: sanity test_130a filefrag does not support FIEMAP");echo XXRETCODE:$?'
May 17 02:02:54 client-18 kernel: Lustre: DEBUG MARKER: SKIP: sanity test_130a filefrag does not support FIEMAP
May 17 02:02:54 client-18 xinetd[3390]: EXIT: shell status=0 pid=11083 duration=0(sec)
May 17 02:02:54 client-18 xinetd[3390]: START: shell pid=11118 from=192.168.4.5
May 17 02:02:54 client-18 rshd[11119]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 02:02:54 client-18 xinetd[3390]: EXIT: shell status=0 pid=11118 duration=0(sec)
May 17 02:02:54 client-18 xinetd[3390]: START: shell pid=11154 from=192.168.4.5
May 17 02:02:54 client-18 rshd[11155]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 02:02:54 client-18 xinetd[3390]: EXIT: shell status=0 pid=11154 duration=0(sec)
May 17 02:02:55 client-18 xinetd[3390]: START: shell pid=11191 from=192.168.4.5
May 17 02:02:55 client-18 rshd[11192]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 130b: FIEMAP \(2-stripe file\) ========================================================== 02:02:54 \(1305622974\)");echo XXRETCODE:$?'
May 17 02:02:55 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 130b: FIEMAP (2-stripe file) ========================================================== 02:02:54 (1305622974)
May 17 02:02:55 client-18 xinetd[3390]: EXIT: shell status=0 pid=11191 duration=0(sec)
May 17 02:02:55 client-18 xinetd[3390]: START: shell pid=11226 from=192.168.4.5
May 17 02:02:55 client-18 rshd[11227]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark  SKIP: sanity test_130b filefrag does not support FIEMAP");echo XXRETCODE:$?'
May 17 02:02:55 client-18 kernel: Lustre: DEBUG MARKER: SKIP: sanity test_130b filefrag does not support FIEMAP
May 17 02:02:55 client-18 xinetd[3390]: EXIT: shell status=0 pid=11226 duration=0(sec)
May 17 02:02:55 client-18 xinetd[3390]: START: shell pid=11261 from=192.168.4.5
May 17 02:02:55 client-18 rshd[11262]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 02:02:55 client-18 xinetd[3390]: EXIT: shell status=0 pid=11261 duration=0(sec)
May 17 02:02:55 client-18 xinetd[3390]: START: shell pid=11297 from=192.168.4.5
May 17 02:02:55 client-18 rshd[11298]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 02:02:55 client-18 xinetd[3390]: EXIT: shell status=0 pid=11297 duration=0(sec)
May 17 02:02:55 client-18 xinetd[3390]: START: shell pid=11334 from=192.168.4.5
May 17 02:02:55 client-18 rshd[11335]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 130c: FIEMAP \(2-stripe file with hole\) ================================================ 02:02:55 \(1305622975\)");echo XXRETCODE:$?'
May 17 02:02:55 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 130c: FIEMAP (2-stripe file with hole) ================================================ 02:02:55 (1305622975)
May 17 02:02:55 client-18 xinetd[3390]: EXIT: shell status=0 pid=11334 duration=0(sec)
May 17 02:02:55 client-18 xinetd[3390]: START: shell pid=11369 from=192.168.4.5
May 17 02:02:56 client-18 rshd[11370]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark  SKIP: sanity test_130c filefrag does not support FIEMAP");echo XXRETCODE:$?'
May 17 02:02:56 client-18 kernel: Lustre: DEBUG MARKER: SKIP: sanity test_130c filefrag does not support FIEMAP
May 17 02:02:56 client-18 xinetd[3390]: EXIT: shell status=0 pid=11369 duration=1(sec)
May 17 02:02:56 client-18 xinetd[3390]: START: shell pid=11404 from=192.168.4.5
May 17 02:02:56 client-18 rshd[11405]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 02:02:56 client-18 xinetd[3390]: EXIT: shell status=0 pid=11404 duration=0(sec)
May 17 02:02:56 client-18 xinetd[3390]: START: shell pid=11440 from=192.168.4.5
May 17 02:02:56 client-18 rshd[11441]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 02:02:56 client-18 xinetd[3390]: EXIT: shell status=0 pid=11440 duration=0(sec)
May 17 02:02:56 client-18 xinetd[3390]: START: shell pid=11477 from=192.168.4.5
May 17 02:02:56 client-18 rshd[11478]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 130d: FIEMAP \(N-stripe file\) ========================================================== 02:02:56 \(1305622976\)");echo XXRETCODE:$?'
May 17 02:02:56 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 130d: FIEMAP (N-stripe file) ========================================================== 02:02:56 (1305622976)
May 17 02:02:56 client-18 xinetd[3390]: EXIT: shell status=0 pid=11477 duration=0(sec)
May 17 02:02:56 client-18 xinetd[3390]: START: shell pid=11512 from=192.168.4.5
May 17 02:02:56 client-18 rshd[11513]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark  SKIP: sanity test_130d filefrag does not support FIEMAP");echo XXRETCODE:$?'
May 17 02:02:56 client-18 kernel: Lustre: DEBUG MARKER: SKIP: sanity test_130d filefrag does not support FIEMAP
May 17 02:02:56 client-18 xinetd[3390]: EXIT: shell status=0 pid=11512 duration=0(sec)
May 17 02:02:56 client-18 xinetd[3390]: START: shell pid=11547 from=192.168.4.5
May 17 02:02:56 client-18 rshd[11548]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 02:02:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=11547 duration=1(sec)
May 17 02:02:57 client-18 xinetd[3390]: START: shell pid=11583 from=192.168.4.5
May 17 02:02:57 client-18 rshd[11584]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 02:02:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=11583 duration=0(sec)
May 17 02:02:57 client-18 xinetd[3390]: START: shell pid=11620 from=192.168.4.5
May 17 02:02:57 client-18 rshd[11621]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 130e: FIEMAP \(test continuation FIEMAP calls\) ========================================= 02:02:57 \(1305622977\)");echo XXRETCODE:$?'
May 17 02:02:57 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 130e: FIEMAP (test continuation FIEMAP calls) ========================================= 02:02:57 (1305622977)
May 17 02:02:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=11620 duration=0(sec)
May 17 02:02:57 client-18 xinetd[3390]: START: shell pid=11655 from=192.168.4.5
May 17 02:02:57 client-18 rshd[11656]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark  SKIP: sanity test_130e filefrag does not support FIEMAP");echo XXRETCODE:$?'
May 17 02:02:57 client-18 kernel: Lustre: DEBUG MARKER: SKIP: sanity test_130e filefrag does not support FIEMAP
May 17 02:02:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=11655 duration=0(sec)
May 17 02:02:57 client-18 xinetd[3390]: START: shell pid=11690 from=192.168.4.5
May 17 02:02:57 client-18 rshd[11691]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 02:02:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=11690 duration=0(sec)
May 17 02:02:57 client-18 xinetd[3390]: START: shell pid=11726 from=192.168.4.5
May 17 02:02:57 client-18 rshd[11727]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 02:02:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=11726 duration=0(sec)
May 17 02:02:57 client-18 xinetd[3390]: START: shell pid=11763 from=192.168.4.5
May 17 02:02:58 client-18 rshd[11764]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 131a: test iov\'s crossing stripe boundary for writev\/readv ============================ 02:02:57 \(1305622977\)");echo XXRETCODE:$?'
May 17 02:02:58 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 131a: test iov's crossing stripe boundary for writev/readv ============================ 02:02:57 (1305622977)
May 17 02:02:58 client-18 xinetd[3390]: EXIT: shell status=0 pid=11763 duration=1(sec)
May 17 02:02:58 client-18 xinetd[3390]: START: shell pid=11798 from=192.168.4.5
May 17 02:02:58 client-18 rshd[11799]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 02:02:58 client-18 xinetd[3390]: EXIT: shell status=0 pid=11798 duration=0(sec)
May 17 02:02:58 client-18 xinetd[3390]: START: shell pid=11834 from=192.168.4.5
May 17 02:02:58 client-18 rshd[11835]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 02:02:58 client-18 xinetd[3390]: EXIT: shell status=0 pid=11834 duration=0(sec)
May 17 02:02:58 client-18 xinetd[3390]: START: shell pid=11871 from=192.168.4.5
May 17 02:02:58 client-18 rshd[11872]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 131b: test append writev ============================================================== 02:02:58 \(1305622978\)");echo XXRETCODE:$?'
May 17 02:02:58 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 131b: test append writev ============================================================== 02:02:58 (1305622978)
May 17 02:02:58 client-18 xinetd[3390]: EXIT: shell status=0 pid=11871 duration=0(sec)
May 17 02:02:58 client-18 xinetd[3390]: START: shell pid=11906 from=192.168.4.5
May 17 02:02:58 client-18 rshd[11907]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 02:02:58 client-18 xinetd[3390]: EXIT: shell status=0 pid=11906 duration=0(sec)
May 17 02:02:58 client-18 xinetd[3390]: START: shell pid=11942 from=192.168.4.5
May 17 02:02:59 client-18 rshd[11943]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 02:02:59 client-18 xinetd[3390]: EXIT: shell status=0 pid=11942 duration=1(sec)
May 17 02:02:59 client-18 xinetd[3390]: START: shell pid=11979 from=192.168.4.5
May 17 02:02:59 client-18 rshd[11980]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 131c: test read\/write on file w\/o objects ============================================= 02:02:59 \(1305622979\)");echo XXRETCODE:$?'
May 17 02:02:59 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 131c: test read/write on file w/o objects ============================================= 02:02:59 (1305622979)
May 17 02:02:59 client-18 xinetd[3390]: EXIT: shell status=0 pid=11979 duration=0(sec)
May 17 02:02:59 client-18 xinetd[3390]: START: shell pid=12014 from=192.168.4.5
May 17 02:02:59 client-18 rshd[12015]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 02:02:59 client-18 xinetd[3390]: EXIT: shell status=0 pid=12014 duration=0(sec)
May 17 02:02:59 client-18 xinetd[3390]: START: shell pid=12050 from=192.168.4.5
May 17 02:02:59 client-18 rshd[12051]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 02:02:59 client-18 xinetd[3390]: EXIT: shell status=0 pid=12050 duration=0(sec)
May 17 02:02:59 client-18 xinetd[3390]: START: shell pid=12087 from=192.168.4.5
May 17 02:02:59 client-18 rshd[12088]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 131d: test short read ================================================================= 02:02:59 \(1305622979\)");echo XXRETCODE:$?'
May 17 02:02:59 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 131d: test short read ================================================================= 02:02:59 (1305622979)
May 17 02:02:59 client-18 xinetd[3390]: EXIT: shell status=0 pid=12087 duration=0(sec)
May 17 02:02:59 client-18 xinetd[3390]: START: shell pid=12122 from=192.168.4.5
May 17 02:02:59 client-18 rshd[12123]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 02:02:59 client-18 xinetd[3390]: EXIT: shell status=0 pid=12122 duration=0(sec)
May 17 02:03:00 client-18 xinetd[3390]: START: shell pid=12158 from=192.168.4.5
May 17 02:03:00 client-18 rshd[12159]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 02:03:00 client-18 xinetd[3390]: EXIT: shell status=0 pid=12158 duration=0(sec)
May 17 02:03:00 client-18 xinetd[3390]: START: shell pid=12195 from=192.168.4.5
May 17 02:03:00 client-18 rshd[12196]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 131e: test read hitting hole ========================================================== 02:03:00 \(1305622980\)");echo XXRETCODE:$?'
May 17 02:03:00 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 131e: test read hitting hole ========================================================== 02:03:00 (1305622980)
May 17 02:03:00 client-18 xinetd[3390]: EXIT: shell status=0 pid=12195 duration=0(sec)
May 17 02:03:00 client-18 xinetd[3390]: START: shell pid=12230 from=192.168.4.5
May 17 02:03:00 client-18 rshd[12231]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 02:03:00 client-18 xinetd[3390]: EXIT: shell status=0 pid=12230 duration=0(sec)
May 17 02:03:00 client-18 xinetd[3390]: START: shell pid=12266 from=192.168.4.5
May 17 02:03:00 client-18 rshd[12267]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 02:03:00 client-18 xinetd[3390]: EXIT: shell status=0 pid=12266 duration=0(sec)
May 17 02:03:00 client-18 xinetd[3390]: START: shell pid=12303 from=192.168.4.5
May 17 02:03:00 client-18 rshd[12304]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 132: som avoids glimpse rpc =========================================================== 02:03:00 \(1305622980\)");echo XXRETCODE:$?'
May 17 02:03:00 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 132: som avoids glimpse rpc =========================================================== 02:03:00 (1305622980)
May 17 02:03:00 client-18 xinetd[3390]: EXIT: shell status=0 pid=12303 duration=0(sec)
May 17 02:03:03 client-18 xinetd[3390]: START: shell pid=12339 from=192.168.4.5
May 17 02:03:03 client-18 rshd[12340]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:; lsof -t /mnt/lustre || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount  /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre is still busy, wait one second && sleep 1;     umount  /mnt/lustre; fi fi");echo XXRETCODE:$?'
May 17 02:03:04 client-18 kernel: LustreError: 12381:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway
May 17 02:03:04 client-18 kernel: LustreError: 12381:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108
May 17 02:03:04 client-18 kernel: Lustre: client ffff81031ffac400 umount complete
May 17 02:03:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=12339 duration=1(sec)
May 17 02:03:04 client-18 xinetd[3390]: START: shell pid=12383 from=192.168.4.5
May 17 02:03:04 client-18 rshd[12384]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:; lsof -t /mnt/lustre2 || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre2 | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount  /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre2 is still busy, wait one second && sleep 1;     umount  /mnt/lustre2; fi fi");echo XXRETCODE:$?'
May 17 02:03:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=12383 duration=0(sec)
May 17 02:03:30 client-18 xinetd[3390]: START: shell pid=12420 from=192.168.4.5
May 17 02:03:30 client-18 rshd[12421]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); mpts=\$(mount | grep -w -c /mnt/lustre); if [ \$running -ne \$mpts ]; then     echo \$(hostname) env are INSANE!;     exit 1; fi");echo XXRETCODE:$?'
May 17 02:03:30 client-18 xinetd[3390]: EXIT: shell status=0 pid=12420 duration=0(sec)
May 17 02:03:30 client-18 xinetd[3390]: START: shell pid=12460 from=192.168.4.5
May 17 02:03:30 client-18 rshd[12461]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); mpts=\$(mount | grep -w -c /mnt/lustre2); if [ \$running -ne \$mpts ]; then     echo \$(hostname) env are INSANE!;     exit 1; fi");echo XXRETCODE:$?'
May 17 02:03:30 client-18 xinetd[3390]: EXIT: shell status=0 pid=12460 duration=0(sec)
May 17 02:03:45 client-18 xinetd[3390]: START: shell pid=12504 from=192.168.4.5
May 17 02:03:45 client-18 rshd[12505]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c " running=\$(mount | grep -c /mnt/lustre' '); rc=0; if [ \$running -eq 0 ] ; then     mkdir -p /mnt/lustre;     mount -t lustre -o flock,user_xattr,acl,som_preview fat-intel-1-ib@o2ib:/lustre /mnt/lustre;     rc=\$?; fi; exit \$rc");echo XXRETCODE:$?'
May 17 02:03:45 client-18 kernel: Lustre: 12544:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null
May 17 02:03:45 client-18 kernel: Lustre: 12544:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 6 previous similar messages
May 17 02:03:45 client-18 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'.
May 17 02:03:45 client-18 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import
May 17 02:03:45 client-18 kernel: Lustre: Increasing default stripe size to min 1048576
May 17 02:03:45 client-18 kernel: Lustre: Client lustre-client has started
May 17 02:03:45 client-18 xinetd[3390]: EXIT: shell status=0 pid=12504 duration=0(sec)
May 17 02:03:45 client-18 xinetd[3390]: START: shell pid=12557 from=192.168.4.5
May 17 02:03:45 client-18 rshd[12558]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "mount | grep -w /mnt/lustre");echo XXRETCODE:$?'
May 17 02:03:45 client-18 xinetd[3390]: EXIT: shell status=0 pid=12557 duration=0(sec)
May 17 02:03:45 client-18 xinetd[3390]: START: shell pid=12594 from=192.168.4.5
May 17 02:03:45 client-18 rshd[12595]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param debug=\"0x33f0406\";         lctl set_param subsystem_debug=\"0xffb7e3ff\";         lctl set_param debug_mb=32;");echo XXRETCODE:$?'
May 17 02:03:45 client-18 kernel: Lustre: 12629:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release.
May 17 02:03:45 client-18 kernel: Lustre: 12629:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message
May 17 02:03:45 client-18 xinetd[3390]: EXIT: shell status=0 pid=12594 duration=0(sec)
May 17 02:03:46 client-18 xinetd[3390]: START: shell pid=12631 from=192.168.4.5
May 17 02:03:46 client-18 rshd[12632]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark Using TIMEOUT=20");echo XXRETCODE:$?'
May 17 02:03:46 client-18 kernel: Lustre: DEBUG MARKER: Using TIMEOUT=20
May 17 02:03:46 client-18 xinetd[3390]: EXIT: shell status=0 pid=12631 duration=0(sec)
May 17 02:03:52 client-18 xinetd[3390]: START: shell pid=12668 from=192.168.4.5
May 17 02:03:52 client-18 rshd[12669]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:; lsof -t /mnt/lustre || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount  /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre is still busy, wait one second && sleep 1;     umount  /mnt/lustre; fi fi");echo XXRETCODE:$?'
May 17 02:03:52 client-18 kernel: LustreError: 12710:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway
May 17 02:03:52 client-18 kernel: LustreError: 12710:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108
May 17 02:03:52 client-18 kernel: Lustre: client ffff8103189cd400 umount complete
May 17 02:03:52 client-18 xinetd[3390]: EXIT: shell status=0 pid=12668 duration=0(sec)
May 17 02:03:52 client-18 xinetd[3390]: START: shell pid=12712 from=192.168.4.5
May 17 02:03:52 client-18 rshd[12713]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:; lsof -t /mnt/lustre2 || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre2 | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount  /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre2 is still busy, wait one second && sleep 1;     umount  /mnt/lustre2; fi fi");echo XXRETCODE:$?'
May 17 02:03:52 client-18 xinetd[3390]: EXIT: shell status=0 pid=12712 duration=0(sec)
May 17 02:04:30 client-18 xinetd[3390]: START: shell pid=12749 from=192.168.4.5
May 17 02:04:31 client-18 rshd[12750]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); mpts=\$(mount | grep -w -c /mnt/lustre); if [ \$running -ne \$mpts ]; then     echo \$(hostname) env are INSANE!;     exit 1; fi");echo XXRETCODE:$?'
May 17 02:04:31 client-18 xinetd[3390]: EXIT: shell status=0 pid=12749 duration=1(sec)
May 17 02:04:31 client-18 xinetd[3390]: START: shell pid=12789 from=192.168.4.5
May 17 02:04:31 client-18 rshd[12790]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); mpts=\$(mount | grep -w -c /mnt/lustre2); if [ \$running -ne \$mpts ]; then     echo \$(hostname) env are INSANE!;     exit 1; fi");echo XXRETCODE:$?'
May 17 02:04:31 client-18 xinetd[3390]: EXIT: shell status=0 pid=12789 duration=0(sec)
May 17 02:04:45 client-18 xinetd[3390]: START: shell pid=12829 from=192.168.4.5
May 17 02:04:45 client-18 rshd[12830]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c " running=\$(mount | grep -c /mnt/lustre' '); rc=0; if [ \$running -eq 0 ] ; then     mkdir -p /mnt/lustre;     mount -t lustre -o flock,user_xattr,acl, fat-intel-1-ib@o2ib:/lustre /mnt/lustre;     rc=\$?; fi; exit \$rc");echo XXRETCODE:$?'
May 17 02:04:46 client-18 kernel: Lustre: 12869:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null
May 17 02:04:46 client-18 kernel: Lustre: 12869:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 7 previous similar messages
May 17 02:04:46 client-18 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'.
May 17 02:04:46 client-18 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import
May 17 02:04:46 client-18 kernel: Lustre: Increasing default stripe size to min 1048576
May 17 02:04:46 client-18 kernel: Lustre: Client lustre-client has started
May 17 02:04:46 client-18 xinetd[3390]: START: shell pid=12882 from=192.168.4.5
May 17 02:04:46 client-18 xinetd[3390]: EXIT: shell status=0 pid=12829 duration=1(sec)
May 17 02:04:46 client-18 rshd[12883]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "mount | grep -w /mnt/lustre");echo XXRETCODE:$?'
May 17 02:04:46 client-18 xinetd[3390]: START: shell pid=12919 from=192.168.4.5
May 17 02:04:46 client-18 xinetd[3390]: EXIT: shell status=0 pid=12882 duration=0(sec)
May 17 02:04:46 client-18 rshd[12920]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param debug=\"0x33f0406\";         lctl set_param subsystem_debug=\"0xffb7e3ff\";         lctl set_param debug_mb=32;");echo XXRETCODE:$?'
May 17 02:04:46 client-18 kernel: Lustre: 12954:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release.
May 17 02:04:46 client-18 kernel: Lustre: 12954:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message
May 17 02:04:46 client-18 xinetd[3390]: EXIT: shell status=0 pid=12919 duration=0(sec)
May 17 02:04:46 client-18 xinetd[3390]: START: shell pid=12956 from=192.168.4.5
May 17 02:04:46 client-18 rshd[12957]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark Using TIMEOUT=20");echo XXRETCODE:$?'
May 17 02:04:46 client-18 kernel: Lustre: DEBUG MARKER: Using TIMEOUT=20
May 17 02:04:46 client-18 xinetd[3390]: EXIT: shell status=0 pid=12956 duration=0(sec)
May 17 02:04:52 client-18 xinetd[3390]: START: shell pid=12995 from=192.168.4.5
May 17 02:04:52 client-18 rshd[12996]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 02:04:52 client-18 xinetd[3390]: EXIT: shell status=0 pid=12995 duration=0(sec)
May 17 02:04:52 client-18 xinetd[3390]: START: shell pid=13031 from=192.168.4.5
May 17 02:04:52 client-18 rshd[13032]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 02:04:52 client-18 xinetd[3390]: EXIT: shell status=0 pid=13031 duration=0(sec)
May 17 02:04:52 client-18 xinetd[3390]: START: shell pid=13068 from=192.168.4.5
May 17 02:04:52 client-18 rshd[13069]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 140: Check reasonable stack depth \(shouldn\'t LBUG\) ==================================== 02:04:52 \(1305623092\)");echo XXRETCODE:$?'
May 17 02:04:52 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 140: Check reasonable stack depth (shouldn't LBUG) ==================================== 02:04:52 (1305623092)
May 17 02:04:52 client-18 xinetd[3390]: EXIT: shell status=0 pid=13068 duration=0(sec)
May 17 02:04:52 client-18 xinetd[3390]: START: shell pid=13103 from=192.168.4.5
May 17 02:04:52 client-18 rshd[13104]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 02:04:53 client-18 xinetd[3390]: EXIT: shell status=0 pid=13103 duration=1(sec)
May 17 02:04:53 client-18 xinetd[3390]: START: shell pid=13139 from=192.168.4.5
May 17 02:04:53 client-18 rshd[13140]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 02:04:53 client-18 xinetd[3390]: EXIT: shell status=0 pid=13139 duration=0(sec)
May 17 02:04:53 client-18 xinetd[3390]: START: shell pid=13176 from=192.168.4.5
May 17 02:04:53 client-18 rshd[13177]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 150: truncate\/append tests ============================================================ 02:04:53 \(1305623093\)");echo XXRETCODE:$?'
May 17 02:04:53 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 150: truncate/append tests ============================================================ 02:04:53 (1305623093)
May 17 02:04:53 client-18 xinetd[3390]: EXIT: shell status=0 pid=13176 duration=0(sec)
May 17 02:04:54 client-18 xinetd[3390]: START: shell pid=13211 from=192.168.4.5
May 17 02:04:54 client-18 rshd[13212]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 02:04:54 client-18 xinetd[3390]: EXIT: shell status=0 pid=13211 duration=0(sec)
May 17 02:04:54 client-18 xinetd[3390]: START: shell pid=13247 from=192.168.4.5
May 17 02:04:54 client-18 rshd[13248]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 02:04:54 client-18 xinetd[3390]: EXIT: shell status=0 pid=13247 duration=0(sec)
May 17 02:04:54 client-18 xinetd[3390]: START: shell pid=13284 from=192.168.4.5
May 17 02:04:54 client-18 rshd[13285]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 151: test cache on oss and controls =================================================== 02:04:54 \(1305623094\)");echo XXRETCODE:$?'
May 17 02:04:54 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 151: test cache on oss and controls =================================================== 02:04:54 (1305623094)
May 17 02:04:54 client-18 xinetd[3390]: EXIT: shell status=0 pid=13284 duration=0(sec)
May 17 02:04:56 client-18 xinetd[3390]: START: shell pid=13319 from=192.168.4.5
May 17 02:04:56 client-18 rshd[13320]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 02:04:56 client-18 xinetd[3390]: EXIT: shell status=0 pid=13319 duration=0(sec)
May 17 02:04:56 client-18 xinetd[3390]: START: shell pid=13355 from=192.168.4.5
May 17 02:04:56 client-18 rshd[13356]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 02:04:56 client-18 xinetd[3390]: EXIT: shell status=0 pid=13355 duration=0(sec)
May 17 02:04:56 client-18 xinetd[3390]: START: shell pid=13392 from=192.168.4.5
May 17 02:04:56 client-18 rshd[13393]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 152: test read\/write with enomem ====================================================== 02:04:56 \(1305623096\)");echo XXRETCODE:$?'
May 17 02:04:56 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 152: test read/write with enomem ====================================================== 02:04:56 (1305623096)
May 17 02:04:56 client-18 xinetd[3390]: EXIT: shell status=0 pid=13392 duration=0(sec)
May 17 02:04:56 client-18 xinetd[3390]: START: shell pid=13427 from=192.168.4.5
May 17 02:04:56 client-18 rshd[13428]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 02:04:56 client-18 xinetd[3390]: EXIT: shell status=0 pid=13427 duration=0(sec)
May 17 02:04:57 client-18 xinetd[3390]: START: shell pid=13463 from=192.168.4.5
May 17 02:04:57 client-18 rshd[13464]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 02:04:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=13463 duration=0(sec)
May 17 02:04:57 client-18 xinetd[3390]: START: shell pid=13500 from=192.168.4.5
May 17 02:04:57 client-18 rshd[13501]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 153: test if fdatasync does not crash ================================================= 02:04:57 \(1305623097\)");echo XXRETCODE:$?'
May 17 02:04:57 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 153: test if fdatasync does not crash ================================================= 02:04:57 (1305623097)
May 17 02:04:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=13500 duration=0(sec)
May 17 02:04:57 client-18 xinetd[3390]: START: shell pid=13535 from=192.168.4.5
May 17 02:04:57 client-18 rshd[13536]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 02:04:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=13535 duration=0(sec)
May 17 02:04:57 client-18 xinetd[3390]: START: shell pid=13571 from=192.168.4.5
May 17 02:04:57 client-18 rshd[13572]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 02:04:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=13571 duration=0(sec)
May 17 02:04:57 client-18 xinetd[3390]: START: shell pid=13608 from=192.168.4.5
May 17 02:04:57 client-18 rshd[13609]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 154: Opening a file by FID ============================================================ 02:04:57 \(1305623097\)");echo XXRETCODE:$?'
May 17 02:04:57 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 154: Opening a file by FID ============================================================ 02:04:57 (1305623097)
May 17 02:04:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=13608 duration=0(sec)
May 17 02:04:58 client-18 xinetd[3390]: START: shell pid=13643 from=192.168.4.5
May 17 02:04:58 client-18 rshd[13644]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 02:04:58 client-18 xinetd[3390]: EXIT: shell status=0 pid=13643 duration=0(sec)
May 17 02:04:58 client-18 xinetd[3390]: START: shell pid=13679 from=192.168.4.5
May 17 02:04:58 client-18 rshd[13680]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 02:04:58 client-18 xinetd[3390]: EXIT: shell status=0 pid=13679 duration=0(sec)
May 17 02:04:58 client-18 xinetd[3390]: START: shell pid=13716 from=192.168.4.5
May 17 02:04:58 client-18 rshd[13717]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 155a: Verification of correctness: read cache:on write_cache:on ======================= 02:04:58 \(1305623098\)");echo XXRETCODE:$?'
May 17 02:04:58 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 155a: Verification of correctness: read cache:on write_cache:on ======================= 02:04:58 (1305623098)
May 17 02:04:58 client-18 xinetd[3390]: EXIT: shell status=0 pid=13716 duration=0(sec)
May 17 02:04:59 client-18 xinetd[3390]: START: shell pid=13751 from=192.168.4.5
May 17 02:04:59 client-18 rshd[13752]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark cache size on OSS is 294912 KB");echo XXRETCODE:$?'
May 17 02:04:59 client-18 kernel: Lustre: DEBUG MARKER: cache size on OSS is 294912 KB
May 17 02:04:59 client-18 xinetd[3390]: EXIT: shell status=0 pid=13751 duration=0(sec)
May 17 02:04:59 client-18 xinetd[3390]: START: shell pid=13786 from=192.168.4.5
May 17 02:04:59 client-18 rshd[13787]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark large file size is 589824 KB");echo XXRETCODE:$?'
May 17 02:04:59 client-18 kernel: Lustre: DEBUG MARKER: large file size is 589824 KB
May 17 02:04:59 client-18 xinetd[3390]: EXIT: shell status=0 pid=13786 duration=0(sec)
May 17 02:04:59 client-18 xinetd[3390]: START: shell pid=13821 from=192.168.4.5
May 17 02:04:59 client-18 rshd[13822]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark min available OST size is 8904820 KB");echo XXRETCODE:$?'
May 17 02:04:59 client-18 kernel: Lustre: DEBUG MARKER: min available OST size is 8904820 KB
May 17 02:04:59 client-18 xinetd[3390]: EXIT: shell status=0 pid=13821 duration=0(sec)
May 17 02:06:37 client-18 xinetd[3390]: START: shell pid=13860 from=192.168.4.5
May 17 02:06:37 client-18 rshd[13861]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 02:06:37 client-18 xinetd[3390]: EXIT: shell status=0 pid=13860 duration=0(sec)
May 17 02:06:37 client-18 xinetd[3390]: START: shell pid=13896 from=192.168.4.5
May 17 02:06:37 client-18 rshd[13897]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 02:06:37 client-18 xinetd[3390]: EXIT: shell status=0 pid=13896 duration=0(sec)
May 17 02:06:37 client-18 xinetd[3390]: START: shell pid=13933 from=192.168.4.5
May 17 02:06:38 client-18 rshd[13934]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 155b: Verification of correctness: read cache:on write_cache:off ====================== 02:06:37 \(1305623197\)");echo XXRETCODE:$?'
May 17 02:06:38 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 155b: Verification of correctness: read cache:on write_cache:off ====================== 02:06:37 (1305623197)
May 17 02:06:38 client-18 xinetd[3390]: EXIT: shell status=0 pid=13933 duration=1(sec)
May 17 02:06:38 client-18 xinetd[3390]: START: shell pid=13968 from=192.168.4.5
May 17 02:06:38 client-18 rshd[13969]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark cache size on OSS is 294912 KB");echo XXRETCODE:$?'
May 17 02:06:38 client-18 kernel: Lustre: DEBUG MARKER: cache size on OSS is 294912 KB
May 17 02:06:38 client-18 xinetd[3390]: EXIT: shell status=0 pid=13968 duration=0(sec)
May 17 02:06:38 client-18 xinetd[3390]: START: shell pid=14003 from=192.168.4.5
May 17 02:06:38 client-18 rshd[14004]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark large file size is 589824 KB");echo XXRETCODE:$?'
May 17 02:06:38 client-18 kernel: Lustre: DEBUG MARKER: large file size is 589824 KB
May 17 02:06:38 client-18 xinetd[3390]: EXIT: shell status=0 pid=14003 duration=0(sec)
May 17 02:06:38 client-18 xinetd[3390]: START: shell pid=14038 from=192.168.4.5
May 17 02:06:38 client-18 rshd[14039]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark min available OST size is 8904820 KB");echo XXRETCODE:$?'
May 17 02:06:39 client-18 kernel: Lustre: DEBUG MARKER: min available OST size is 8904820 KB
May 17 02:06:39 client-18 xinetd[3390]: EXIT: shell status=0 pid=14038 duration=1(sec)
May 17 02:08:22 client-18 xinetd[3390]: START: shell pid=14077 from=192.168.4.5
May 17 02:08:22 client-18 rshd[14078]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 02:08:22 client-18 xinetd[3390]: EXIT: shell status=0 pid=14077 duration=0(sec)
May 17 02:08:22 client-18 xinetd[3390]: START: shell pid=14113 from=192.168.4.5
May 17 02:08:22 client-18 rshd[14114]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 02:08:22 client-18 xinetd[3390]: EXIT: shell status=0 pid=14113 duration=0(sec)
May 17 02:08:22 client-18 xinetd[3390]: START: shell pid=14150 from=192.168.4.5
May 17 02:08:22 client-18 rshd[14151]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 155c: Verification of correctness: read cache:off write_cache:on ====================== 02:08:22 \(1305623302\)");echo XXRETCODE:$?'
May 17 02:08:22 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 155c: Verification of correctness: read cache:off write_cache:on ====================== 02:08:22 (1305623302)
May 17 02:08:22 client-18 xinetd[3390]: EXIT: shell status=0 pid=14150 duration=0(sec)
May 17 02:08:23 client-18 xinetd[3390]: START: shell pid=14185 from=192.168.4.5
May 17 02:08:23 client-18 rshd[14186]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark cache size on OSS is 294912 KB");echo XXRETCODE:$?'
May 17 02:08:23 client-18 kernel: Lustre: DEBUG MARKER: cache size on OSS is 294912 KB
May 17 02:08:23 client-18 xinetd[3390]: EXIT: shell status=0 pid=14185 duration=0(sec)
May 17 02:08:23 client-18 xinetd[3390]: START: shell pid=14220 from=192.168.4.5
May 17 02:08:23 client-18 rshd[14221]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark large file size is 589824 KB");echo XXRETCODE:$?'
May 17 02:08:23 client-18 kernel: Lustre: DEBUG MARKER: large file size is 589824 KB
May 17 02:08:23 client-18 xinetd[3390]: EXIT: shell status=0 pid=14220 duration=0(sec)
May 17 02:08:23 client-18 xinetd[3390]: START: shell pid=14255 from=192.168.4.5
May 17 02:08:23 client-18 rshd[14256]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark min available OST size is 8904820 KB");echo XXRETCODE:$?'
May 17 02:08:23 client-18 kernel: Lustre: DEBUG MARKER: min available OST size is 8904820 KB
May 17 02:08:23 client-18 xinetd[3390]: EXIT: shell status=0 pid=14255 duration=0(sec)
May 17 02:10:08 client-18 xinetd[3390]: START: shell pid=14298 from=192.168.4.5
May 17 02:10:08 client-18 rshd[14299]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 02:10:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=14298 duration=0(sec)
May 17 02:10:08 client-18 xinetd[3390]: START: shell pid=14334 from=192.168.4.5
May 17 02:10:08 client-18 rshd[14335]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 02:10:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=14334 duration=0(sec)
May 17 02:10:08 client-18 xinetd[3390]: START: shell pid=14371 from=192.168.4.5
May 17 02:10:08 client-18 rshd[14372]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 155d: Verification of correctness: read cache:off write_cache:off ===================== 02:10:08 \(1305623408\)");echo XXRETCODE:$?'
May 17 02:10:08 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 155d: Verification of correctness: read cache:off write_cache:off ===================== 02:10:08 (1305623408)
May 17 02:10:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=14371 duration=0(sec)
May 17 02:10:09 client-18 xinetd[3390]: START: shell pid=14406 from=192.168.4.5
May 17 02:10:09 client-18 rshd[14407]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark cache size on OSS is 294912 KB");echo XXRETCODE:$?'
May 17 02:10:09 client-18 kernel: Lustre: DEBUG MARKER: cache size on OSS is 294912 KB
May 17 02:10:09 client-18 xinetd[3390]: EXIT: shell status=0 pid=14406 duration=0(sec)
May 17 02:10:09 client-18 xinetd[3390]: START: shell pid=14441 from=192.168.4.5
May 17 02:10:09 client-18 rshd[14442]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark large file size is 589824 KB");echo XXRETCODE:$?'
May 17 02:10:09 client-18 kernel: Lustre: DEBUG MARKER: large file size is 589824 KB
May 17 02:10:09 client-18 xinetd[3390]: EXIT: shell status=0 pid=14441 duration=0(sec)
May 17 02:10:09 client-18 xinetd[3390]: START: shell pid=14476 from=192.168.4.5
May 17 02:10:09 client-18 rshd[14477]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark min available OST size is 8904820 KB");echo XXRETCODE:$?'
May 17 02:10:09 client-18 kernel: Lustre: DEBUG MARKER: min available OST size is 8904820 KB
May 17 02:10:09 client-18 xinetd[3390]: EXIT: shell status=0 pid=14476 duration=0(sec)
May 17 02:12:00 client-18 xinetd[3390]: START: shell pid=14518 from=192.168.4.5
May 17 02:12:00 client-18 rshd[14519]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 02:12:00 client-18 xinetd[3390]: EXIT: shell status=0 pid=14518 duration=0(sec)
May 17 02:12:00 client-18 xinetd[3390]: START: shell pid=14554 from=192.168.4.5
May 17 02:12:00 client-18 rshd[14555]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 02:12:00 client-18 xinetd[3390]: EXIT: shell status=0 pid=14554 duration=0(sec)
May 17 02:12:00 client-18 xinetd[3390]: START: shell pid=14591 from=192.168.4.5
May 17 02:12:00 client-18 rshd[14592]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 156: Verification of tunables ========================================================= 02:12:00 \(1305623520\)");echo XXRETCODE:$?'
May 17 02:12:00 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 156: Verification of tunables ========================================================= 02:12:00 (1305623520)
May 17 02:12:00 client-18 xinetd[3390]: EXIT: shell status=0 pid=14591 duration=0(sec)
May 17 02:12:00 client-18 xinetd[3390]: START: shell pid=14626 from=192.168.4.5
May 17 02:12:00 client-18 rshd[14627]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark Turn on read and write cache");echo XXRETCODE:$?'
May 17 02:12:00 client-18 kernel: Lustre: DEBUG MARKER: Turn on read and write cache
May 17 02:12:00 client-18 xinetd[3390]: EXIT: shell status=0 pid=14626 duration=0(sec)
May 17 02:12:01 client-18 xinetd[3390]: START: shell pid=14661 from=192.168.4.5
May 17 02:12:01 client-18 rshd[14662]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark Write data and read it back.");echo XXRETCODE:$?'
May 17 02:12:01 client-18 kernel: Lustre: DEBUG MARKER: Write data and read it back.
May 17 02:12:01 client-18 xinetd[3390]: EXIT: shell status=0 pid=14661 duration=0(sec)
May 17 02:12:01 client-18 xinetd[3390]: START: shell pid=14696 from=192.168.4.5
May 17 02:12:01 client-18 rshd[14697]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark Read should be satisfied from the cache.");echo XXRETCODE:$?'
May 17 02:12:01 client-18 kernel: Lustre: DEBUG MARKER: Read should be satisfied from the cache.
May 17 02:12:01 client-18 xinetd[3390]: EXIT: shell status=0 pid=14696 duration=0(sec)
May 17 02:12:01 client-18 xinetd[3390]: START: shell pid=14731 from=192.168.4.5
May 17 02:12:02 client-18 rshd[14732]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark cache hits:: before: 294940, after: 294943");echo XXRETCODE:$?'
May 17 02:12:02 client-18 kernel: Lustre: DEBUG MARKER: cache hits:: before: 294940, after: 294943
May 17 02:12:02 client-18 xinetd[3390]: EXIT: shell status=0 pid=14731 duration=1(sec)
May 17 02:12:02 client-18 xinetd[3390]: START: shell pid=14766 from=192.168.4.5
May 17 02:12:02 client-18 rshd[14767]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark Read again\; it should be satisfied from the cache.");echo XXRETCODE:$?'
May 17 02:12:02 client-18 kernel: Lustre: DEBUG MARKER: Read again; it should be satisfied from the cache.
May 17 02:12:02 client-18 xinetd[3390]: EXIT: shell status=0 pid=14766 duration=0(sec)
May 17 02:12:02 client-18 xinetd[3390]: START: shell pid=14801 from=192.168.4.5
May 17 02:12:02 client-18 rshd[14802]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark cache hits:: before: 294943, after: 294946");echo XXRETCODE:$?'
May 17 02:12:02 client-18 kernel: Lustre: DEBUG MARKER: cache hits:: before: 294943, after: 294946
May 17 02:12:02 client-18 xinetd[3390]: EXIT: shell status=0 pid=14801 duration=0(sec)
May 17 02:12:02 client-18 xinetd[3390]: START: shell pid=14836 from=192.168.4.5
May 17 02:12:02 client-18 rshd[14837]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark Turn off the read cache and turn on the write cache");echo XXRETCODE:$?'
May 17 02:12:02 client-18 kernel: Lustre: DEBUG MARKER: Turn off the read cache and turn on the write cache
May 17 02:12:02 client-18 xinetd[3390]: EXIT: shell status=0 pid=14836 duration=0(sec)
May 17 02:12:03 client-18 xinetd[3390]: START: shell pid=14871 from=192.168.4.5
May 17 02:12:03 client-18 rshd[14872]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark Read again\; it should be satisfied from the cache.");echo XXRETCODE:$?'
May 17 02:12:03 client-18 kernel: Lustre: DEBUG MARKER: Read again; it should be satisfied from the cache.
May 17 02:12:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=14871 duration=0(sec)
May 17 02:12:03 client-18 xinetd[3390]: START: shell pid=14906 from=192.168.4.5
May 17 02:12:03 client-18 rshd[14907]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark cache hits:: before: 294946, after: 294949");echo XXRETCODE:$?'
May 17 02:12:03 client-18 kernel: Lustre: DEBUG MARKER: cache hits:: before: 294946, after: 294949
May 17 02:12:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=14906 duration=0(sec)
May 17 02:12:03 client-18 xinetd[3390]: START: shell pid=14941 from=192.168.4.5
May 17 02:12:03 client-18 rshd[14942]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark Read again\; it should not be satisfied from the cache.");echo XXRETCODE:$?'
May 17 02:12:03 client-18 kernel: Lustre: DEBUG MARKER: Read again; it should not be satisfied from the cache.
May 17 02:12:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=14941 duration=0(sec)
May 17 02:12:04 client-18 xinetd[3390]: START: shell pid=14976 from=192.168.4.5
May 17 02:12:04 client-18 rshd[14977]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark cache hits:: before: 294949, after: 294949");echo XXRETCODE:$?'
May 17 02:12:04 client-18 kernel: Lustre: DEBUG MARKER: cache hits:: before: 294949, after: 294949
May 17 02:12:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=14976 duration=0(sec)
May 17 02:12:04 client-18 xinetd[3390]: START: shell pid=15011 from=192.168.4.5
May 17 02:12:04 client-18 rshd[15012]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark Write data and read it back.");echo XXRETCODE:$?'
May 17 02:12:04 client-18 kernel: Lustre: DEBUG MARKER: Write data and read it back.
May 17 02:12:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=15011 duration=0(sec)
May 17 02:12:04 client-18 xinetd[3390]: START: shell pid=15046 from=192.168.4.5
May 17 02:12:04 client-18 rshd[15047]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark Read should be satisfied from the cache.");echo XXRETCODE:$?'
May 17 02:12:04 client-18 kernel: Lustre: DEBUG MARKER: Read should be satisfied from the cache.
May 17 02:12:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=15046 duration=0(sec)
May 17 02:12:04 client-18 xinetd[3390]: START: shell pid=15081 from=192.168.4.5
May 17 02:12:04 client-18 rshd[15082]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark cache hits:: before: 294949, after: 294952");echo XXRETCODE:$?'
May 17 02:12:05 client-18 kernel: Lustre: DEBUG MARKER: cache hits:: before: 294949, after: 294952
May 17 02:12:05 client-18 xinetd[3390]: EXIT: shell status=0 pid=15081 duration=1(sec)
May 17 02:12:05 client-18 xinetd[3390]: START: shell pid=15116 from=192.168.4.5
May 17 02:12:05 client-18 rshd[15117]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark Read again\; it should not be satisfied from the cache.");echo XXRETCODE:$?'
May 17 02:12:05 client-18 kernel: Lustre: DEBUG MARKER: Read again; it should not be satisfied from the cache.
May 17 02:12:05 client-18 xinetd[3390]: EXIT: shell status=0 pid=15116 duration=0(sec)
May 17 02:12:05 client-18 xinetd[3390]: START: shell pid=15151 from=192.168.4.5
May 17 02:12:05 client-18 rshd[15152]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark cache hits:: before: 294952, after: 294952");echo XXRETCODE:$?'
May 17 02:12:05 client-18 kernel: Lustre: DEBUG MARKER: cache hits:: before: 294952, after: 294952
May 17 02:12:05 client-18 xinetd[3390]: EXIT: shell status=0 pid=15151 duration=0(sec)
May 17 02:12:05 client-18 xinetd[3390]: START: shell pid=15186 from=192.168.4.5
May 17 02:12:05 client-18 rshd[15187]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark Turn off read and write cache");echo XXRETCODE:$?'
May 17 02:12:05 client-18 kernel: Lustre: DEBUG MARKER: Turn off read and write cache
May 17 02:12:05 client-18 xinetd[3390]: EXIT: shell status=0 pid=15186 duration=0(sec)
May 17 02:12:06 client-18 xinetd[3390]: START: shell pid=15221 from=192.168.4.5
May 17 02:12:06 client-18 rshd[15222]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark Write data and read it back");echo XXRETCODE:$?'
May 17 02:12:06 client-18 kernel: Lustre: DEBUG MARKER: Write data and read it back
May 17 02:12:06 client-18 xinetd[3390]: EXIT: shell status=0 pid=15221 duration=0(sec)
May 17 02:12:06 client-18 xinetd[3390]: START: shell pid=15256 from=192.168.4.5
May 17 02:12:06 client-18 rshd[15257]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark It should not be satisfied from the cache.");echo XXRETCODE:$?'
May 17 02:12:06 client-18 kernel: Lustre: DEBUG MARKER: It should not be satisfied from the cache.
May 17 02:12:06 client-18 xinetd[3390]: EXIT: shell status=0 pid=15256 duration=0(sec)
May 17 02:12:06 client-18 xinetd[3390]: START: shell pid=15291 from=192.168.4.5
May 17 02:12:06 client-18 rshd[15292]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark cache hits:: before: 294952, after: 294952");echo XXRETCODE:$?'
May 17 02:12:06 client-18 kernel: Lustre: DEBUG MARKER: cache hits:: before: 294952, after: 294952
May 17 02:12:06 client-18 xinetd[3390]: EXIT: shell status=0 pid=15291 duration=0(sec)
May 17 02:12:06 client-18 xinetd[3390]: START: shell pid=15326 from=192.168.4.5
May 17 02:12:06 client-18 rshd[15327]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark Turn on the read cache and turn off the write cache");echo XXRETCODE:$?'
May 17 02:12:07 client-18 kernel: Lustre: DEBUG MARKER: Turn on the read cache and turn off the write cache
May 17 02:12:07 client-18 xinetd[3390]: EXIT: shell status=0 pid=15326 duration=1(sec)
May 17 02:12:07 client-18 xinetd[3390]: START: shell pid=15361 from=192.168.4.5
May 17 02:12:07 client-18 rshd[15362]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark Write data and read it back");echo XXRETCODE:$?'
May 17 02:12:07 client-18 kernel: Lustre: DEBUG MARKER: Write data and read it back
May 17 02:12:07 client-18 xinetd[3390]: EXIT: shell status=0 pid=15361 duration=0(sec)
May 17 02:12:07 client-18 xinetd[3390]: START: shell pid=15396 from=192.168.4.5
May 17 02:12:07 client-18 rshd[15397]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark It should not be satisfied from the cache.");echo XXRETCODE:$?'
May 17 02:12:07 client-18 kernel: Lustre: DEBUG MARKER: It should not be satisfied from the cache.
May 17 02:12:07 client-18 xinetd[3390]: EXIT: shell status=0 pid=15396 duration=0(sec)
May 17 02:12:08 client-18 xinetd[3390]: START: shell pid=15431 from=192.168.4.5
May 17 02:12:08 client-18 rshd[15432]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark cache hits:: before: 294952, after: 294952");echo XXRETCODE:$?'
May 17 02:12:08 client-18 kernel: Lustre: DEBUG MARKER: cache hits:: before: 294952, after: 294952
May 17 02:12:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=15431 duration=0(sec)
May 17 02:12:08 client-18 xinetd[3390]: START: shell pid=15466 from=192.168.4.5
May 17 02:12:08 client-18 rshd[15467]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark Read again\; it should be satisfied from the cache.");echo XXRETCODE:$?'
May 17 02:12:08 client-18 kernel: Lustre: DEBUG MARKER: Read again; it should be satisfied from the cache.
May 17 02:12:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=15466 duration=0(sec)
May 17 02:12:08 client-18 xinetd[3390]: START: shell pid=15501 from=192.168.4.5
May 17 02:12:08 client-18 rshd[15502]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark cache hits:: before: 294952, after: 294955");echo XXRETCODE:$?'
May 17 02:12:08 client-18 kernel: Lustre: DEBUG MARKER: cache hits:: before: 294952, after: 294955
May 17 02:12:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=15501 duration=0(sec)
May 17 02:12:08 client-18 xinetd[3390]: START: shell pid=15536 from=192.168.4.5
May 17 02:12:08 client-18 rshd[15537]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 02:12:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=15536 duration=0(sec)
May 17 02:12:09 client-18 xinetd[3390]: START: shell pid=15572 from=192.168.4.5
May 17 02:12:09 client-18 rshd[15573]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 02:12:09 client-18 xinetd[3390]: EXIT: shell status=0 pid=15572 duration=0(sec)
May 17 02:12:09 client-18 xinetd[3390]: START: shell pid=15609 from=192.168.4.5
May 17 02:12:09 client-18 rshd[15610]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 160: changelog sanity ================================================================= 02:12:09 \(1305623529\)");echo XXRETCODE:$?'
May 17 02:12:09 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 160: changelog sanity ================================================================= 02:12:09 (1305623529)
May 17 02:12:09 client-18 xinetd[3390]: EXIT: shell status=0 pid=15609 duration=0(sec)
May 17 02:12:10 client-18 xinetd[3390]: START: shell pid=15644 from=192.168.4.5
May 17 02:12:10 client-18 rshd[15645]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 02:12:10 client-18 xinetd[3390]: EXIT: shell status=0 pid=15644 duration=0(sec)
May 17 02:12:10 client-18 xinetd[3390]: START: shell pid=15680 from=192.168.4.5
May 17 02:12:10 client-18 rshd[15681]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 02:12:11 client-18 xinetd[3390]: EXIT: shell status=0 pid=15680 duration=1(sec)
May 17 02:12:11 client-18 xinetd[3390]: START: shell pid=15717 from=192.168.4.5
May 17 02:12:11 client-18 rshd[15718]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 161: link ea sanity =================================================================== 02:12:11 \(1305623531\)");echo XXRETCODE:$?'
May 17 02:12:11 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 161: link ea sanity =================================================================== 02:12:11 (1305623531)
May 17 02:12:11 client-18 xinetd[3390]: EXIT: shell status=0 pid=15717 duration=0(sec)
May 17 02:12:13 client-18 xinetd[3390]: START: shell pid=15752 from=192.168.4.5
May 17 02:12:13 client-18 rshd[15753]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 02:12:13 client-18 xinetd[3390]: EXIT: shell status=0 pid=15752 duration=0(sec)
May 17 02:12:13 client-18 xinetd[3390]: START: shell pid=15788 from=192.168.4.5
May 17 02:12:13 client-18 rshd[15789]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 02:12:13 client-18 xinetd[3390]: EXIT: shell status=0 pid=15788 duration=0(sec)
May 17 02:12:13 client-18 xinetd[3390]: START: shell pid=15825 from=192.168.4.5
May 17 02:12:13 client-18 rshd[15826]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 162: path lookup sanity =============================================================== 02:12:13 \(1305623533\)");echo XXRETCODE:$?'
May 17 02:12:13 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 162: path lookup sanity =============================================================== 02:12:13 (1305623533)
May 17 02:12:13 client-18 xinetd[3390]: EXIT: shell status=0 pid=15825 duration=0(sec)
May 17 02:12:13 client-18 xinetd[3390]: START: shell pid=15860 from=192.168.4.5
May 17 02:12:13 client-18 rshd[15861]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 02:12:14 client-18 xinetd[3390]: EXIT: shell status=0 pid=15860 duration=1(sec)
May 17 02:12:14 client-18 xinetd[3390]: START: shell pid=15896 from=192.168.4.5
May 17 02:12:14 client-18 rshd[15897]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 02:12:14 client-18 xinetd[3390]: EXIT: shell status=0 pid=15896 duration=0(sec)
May 17 02:12:14 client-18 xinetd[3390]: START: shell pid=15933 from=192.168.4.5
May 17 02:12:14 client-18 rshd[15934]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 163: kernel \<-\> userspace comms ======================================================= 02:12:14 \(1305623534\)");echo XXRETCODE:$?'
May 17 02:12:14 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 163: kernel <-> userspace comms ======================================================= 02:12:14 (1305623534)
May 17 02:12:14 client-18 xinetd[3390]: EXIT: shell status=0 pid=15933 duration=0(sec)
May 17 02:12:15 client-18 xinetd[3390]: START: shell pid=15968 from=192.168.4.5
May 17 02:12:15 client-18 rshd[15969]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 02:12:15 client-18 xinetd[3390]: EXIT: shell status=0 pid=15968 duration=0(sec)
May 17 02:12:15 client-18 xinetd[3390]: START: shell pid=16004 from=192.168.4.5
May 17 02:12:15 client-18 rshd[16005]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 02:12:15 client-18 xinetd[3390]: EXIT: shell status=0 pid=16004 duration=0(sec)
May 17 02:12:15 client-18 xinetd[3390]: START: shell pid=16041 from=192.168.4.5
May 17 02:12:15 client-18 rshd[16042]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 169: parallel read and truncate should not deadlock =================================== 02:12:15 \(1305623535\)");echo XXRETCODE:$?'
May 17 02:12:16 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 169: parallel read and truncate should not deadlock =================================== 02:12:15 (1305623535)
May 17 02:12:16 client-18 xinetd[3390]: EXIT: shell status=0 pid=16041 duration=1(sec)
May 17 02:12:16 client-18 xinetd[3390]: START: shell pid=16076 from=192.168.4.5
May 17 02:12:16 client-18 rshd[16077]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark creating a 10 Mb file");echo XXRETCODE:$?'
May 17 02:12:16 client-18 kernel: Lustre: DEBUG MARKER: creating a 10 Mb file
May 17 02:12:16 client-18 xinetd[3390]: EXIT: shell status=0 pid=16076 duration=0(sec)
May 17 02:12:16 client-18 xinetd[3390]: START: shell pid=16111 from=192.168.4.5
May 17 02:12:16 client-18 rshd[16112]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark starting reads");echo XXRETCODE:$?'
May 17 02:12:16 client-18 kernel: Lustre: DEBUG MARKER: starting reads
May 17 02:12:16 client-18 xinetd[3390]: EXIT: shell status=0 pid=16111 duration=0(sec)
May 17 02:12:16 client-18 xinetd[3390]: START: shell pid=16146 from=192.168.4.5
May 17 02:12:16 client-18 rshd[16147]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark truncating the file");echo XXRETCODE:$?'
May 17 02:12:16 client-18 kernel: Lustre: DEBUG MARKER: truncating the file
May 17 02:12:16 client-18 xinetd[3390]: EXIT: shell status=0 pid=16146 duration=0(sec)
May 17 02:12:16 client-18 xinetd[3390]: START: shell pid=16181 from=192.168.4.5
May 17 02:12:16 client-18 rshd[16182]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark killing dd");echo XXRETCODE:$?'
May 17 02:12:16 client-18 kernel: Lustre: DEBUG MARKER: killing dd
May 17 02:12:16 client-18 xinetd[3390]: EXIT: shell status=0 pid=16181 duration=0(sec)
May 17 02:12:16 client-18 xinetd[3390]: START: shell pid=16216 from=192.168.4.5
May 17 02:12:16 client-18 rshd[16217]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark removing the temporary file");echo XXRETCODE:$?'
May 17 02:12:17 client-18 kernel: Lustre: DEBUG MARKER: removing the temporary file
May 17 02:12:17 client-18 xinetd[3390]: EXIT: shell status=0 pid=16216 duration=1(sec)
May 17 02:12:17 client-18 xinetd[3390]: START: shell pid=16251 from=192.168.4.5
May 17 02:12:17 client-18 rshd[16252]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 02:12:17 client-18 xinetd[3390]: EXIT: shell status=0 pid=16251 duration=0(sec)
May 17 02:12:17 client-18 xinetd[3390]: START: shell pid=16287 from=192.168.4.5
May 17 02:12:17 client-18 rshd[16288]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 02:12:17 client-18 xinetd[3390]: EXIT: shell status=0 pid=16287 duration=0(sec)
May 17 02:12:17 client-18 xinetd[3390]: START: shell pid=16324 from=192.168.4.5
May 17 02:12:17 client-18 rshd[16325]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 170: test lctl df to handle corrupted log ============================================= 02:12:17 \(1305623537\)");echo XXRETCODE:$?'
May 17 02:12:17 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 170: test lctl df to handle corrupted log ============================================= 02:12:17 (1305623537)
May 17 02:12:17 client-18 xinetd[3390]: EXIT: shell status=0 pid=16324 duration=0(sec)
May 17 02:12:19 client-18 xinetd[3390]: START: shell pid=16363 from=192.168.4.5
May 17 02:12:19 client-18 rshd[16364]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 02:12:19 client-18 xinetd[3390]: EXIT: shell status=0 pid=16363 duration=0(sec)
May 17 02:12:19 client-18 xinetd[3390]: START: shell pid=16399 from=192.168.4.5
May 17 02:12:19 client-18 rshd[16400]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 02:12:19 client-18 xinetd[3390]: EXIT: shell status=0 pid=16399 duration=0(sec)
May 17 02:12:20 client-18 xinetd[3390]: START: shell pid=16436 from=192.168.4.5
May 17 02:12:20 client-18 rshd[16437]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 171: test libcfs_debug_dumplog_thread stuck in do_exit\(\) ============================== 02:12:19 \(1305623539\)");echo XXRETCODE:$?'
May 17 02:12:20 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 171: test libcfs_debug_dumplog_thread stuck in do_exit() ============================== 02:12:19 (1305623539)
May 17 02:12:20 client-18 xinetd[3390]: EXIT: shell status=0 pid=16436 duration=0(sec)
May 17 02:12:23 client-18 xinetd[3390]: START: shell pid=16471 from=192.168.4.5
May 17 02:12:23 client-18 rshd[16472]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 02:12:23 client-18 xinetd[3390]: EXIT: shell status=0 pid=16471 duration=0(sec)
May 17 02:12:23 client-18 xinetd[3390]: START: shell pid=16507 from=192.168.4.5
May 17 02:12:23 client-18 rshd[16508]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 02:12:23 client-18 xinetd[3390]: EXIT: shell status=0 pid=16507 duration=0(sec)
May 17 02:12:23 client-18 xinetd[3390]: START: shell pid=16544 from=192.168.4.5
May 17 02:12:23 client-18 rshd[16545]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 180a: test obdecho on osc ============================================================= 02:12:23 \(1305623543\)");echo XXRETCODE:$?'
May 17 02:12:23 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 180a: test obdecho on osc ============================================================= 02:12:23 (1305623543)
May 17 02:12:23 client-18 xinetd[3390]: EXIT: shell status=0 pid=16544 duration=0(sec)
May 17 02:12:24 client-18 xinetd[3390]: START: shell pid=16579 from=192.168.4.5
May 17 02:12:24 client-18 rshd[16580]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 02:12:24 client-18 xinetd[3390]: EXIT: shell status=0 pid=16579 duration=0(sec)
May 17 02:12:24 client-18 xinetd[3390]: START: shell pid=16615 from=192.168.4.5
May 17 02:12:24 client-18 rshd[16616]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 02:12:24 client-18 xinetd[3390]: EXIT: shell status=0 pid=16615 duration=0(sec)
May 17 02:12:24 client-18 xinetd[3390]: START: shell pid=16652 from=192.168.4.5
May 17 02:12:24 client-18 rshd[16653]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 180b: test obdecho directly on obdfilter ============================================== 02:12:24 \(1305623544\)");echo XXRETCODE:$?'
May 17 02:12:24 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 180b: test obdecho directly on obdfilter ============================================== 02:12:24 (1305623544)
May 17 02:12:24 client-18 xinetd[3390]: EXIT: shell status=0 pid=16652 duration=0(sec)
May 17 02:12:25 client-18 xinetd[3390]: START: shell pid=16687 from=192.168.4.5
May 17 02:12:25 client-18 rshd[16688]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 02:12:26 client-18 xinetd[3390]: EXIT: shell status=0 pid=16687 duration=1(sec)
May 17 02:12:26 client-18 xinetd[3390]: START: shell pid=16723 from=192.168.4.5
May 17 02:12:26 client-18 rshd[16724]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 02:12:26 client-18 xinetd[3390]: EXIT: shell status=0 pid=16723 duration=0(sec)
May 17 02:12:26 client-18 xinetd[3390]: START: shell pid=16760 from=192.168.4.5
May 17 02:12:26 client-18 rshd[16761]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 181: Test open-unlinked dir =========================================================== 02:12:26 \(1305623546\)");echo XXRETCODE:$?'
May 17 02:12:26 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 181: Test open-unlinked dir =========================================================== 02:12:26 (1305623546)
May 17 02:12:26 client-18 xinetd[3390]: EXIT: shell status=0 pid=16760 duration=0(sec)
May 17 02:12:33 client-18 xinetd[3390]: START: shell pid=16795 from=192.168.4.5
May 17 02:12:33 client-18 rshd[16796]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 02:12:34 client-18 xinetd[3390]: EXIT: shell status=0 pid=16795 duration=1(sec)
May 17 02:12:34 client-18 xinetd[3390]: START: shell pid=16831 from=192.168.4.5
May 17 02:12:34 client-18 rshd[16832]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 02:12:34 client-18 xinetd[3390]: EXIT: shell status=0 pid=16831 duration=0(sec)
May 17 02:12:34 client-18 xinetd[3390]: START: shell pid=16868 from=192.168.4.5
May 17 02:12:34 client-18 rshd[16869]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 200a: Create new pool ================================================================= 02:12:34 \(1305623554\)");echo XXRETCODE:$?'
May 17 02:12:34 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 200a: Create new pool ================================================================= 02:12:34 (1305623554)
May 17 02:12:34 client-18 xinetd[3390]: EXIT: shell status=0 pid=16868 duration=0(sec)
May 17 02:12:43 client-18 xinetd[3390]: START: shell pid=16907 from=192.168.4.5
May 17 02:12:43 client-18 rshd[16908]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 02:12:43 client-18 xinetd[3390]: EXIT: shell status=0 pid=16907 duration=0(sec)
May 17 02:12:43 client-18 xinetd[3390]: START: shell pid=16943 from=192.168.4.5
May 17 02:12:43 client-18 rshd[16944]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 02:12:43 client-18 xinetd[3390]: EXIT: shell status=0 pid=16943 duration=0(sec)
May 17 02:12:43 client-18 xinetd[3390]: START: shell pid=16980 from=192.168.4.5
May 17 02:12:44 client-18 rshd[16981]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 200b: Add targets to a pool =========================================================== 02:12:43 \(1305623563\)");echo XXRETCODE:$?'
May 17 02:12:44 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 200b: Add targets to a pool =========================================================== 02:12:43 (1305623563)
May 17 02:12:44 client-18 xinetd[3390]: EXIT: shell status=0 pid=16980 duration=1(sec)
May 17 02:12:50 client-18 xinetd[3390]: START: shell pid=17017 from=192.168.4.5
May 17 02:12:50 client-18 rshd[17018]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 02:12:50 client-18 xinetd[3390]: EXIT: shell status=0 pid=17017 duration=0(sec)
May 17 02:12:50 client-18 xinetd[3390]: START: shell pid=17053 from=192.168.4.5
May 17 02:12:50 client-18 rshd[17054]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 02:12:50 client-18 xinetd[3390]: EXIT: shell status=0 pid=17053 duration=0(sec)
May 17 02:12:50 client-18 xinetd[3390]: START: shell pid=17090 from=192.168.4.5
May 17 02:12:50 client-18 rshd[17091]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 200c: Set pool on a directory ========================================================= 02:12:50 \(1305623570\)");echo XXRETCODE:$?'
May 17 02:12:50 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 200c: Set pool on a directory ========================================================= 02:12:50 (1305623570)
May 17 02:12:50 client-18 xinetd[3390]: EXIT: shell status=0 pid=17090 duration=0(sec)
May 17 02:12:50 client-18 xinetd[3390]: START: shell pid=17125 from=192.168.4.5
May 17 02:12:50 client-18 rshd[17126]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 02:12:50 client-18 xinetd[3390]: EXIT: shell status=0 pid=17125 duration=0(sec)
May 17 02:12:51 client-18 xinetd[3390]: START: shell pid=17161 from=192.168.4.5
May 17 02:12:51 client-18 rshd[17162]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 02:12:51 client-18 xinetd[3390]: EXIT: shell status=0 pid=17161 duration=0(sec)
May 17 02:12:51 client-18 xinetd[3390]: START: shell pid=17198 from=192.168.4.5
May 17 02:12:51 client-18 rshd[17199]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 200d: Check pool on a directory ======================================================= 02:12:51 \(1305623571\)");echo XXRETCODE:$?'
May 17 02:12:51 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 200d: Check pool on a directory ======================================================= 02:12:51 (1305623571)
May 17 02:12:51 client-18 xinetd[3390]: EXIT: shell status=0 pid=17198 duration=0(sec)
May 17 02:12:51 client-18 xinetd[3390]: START: shell pid=17233 from=192.168.4.5
May 17 02:12:51 client-18 rshd[17234]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 02:12:51 client-18 xinetd[3390]: EXIT: shell status=0 pid=17233 duration=0(sec)
May 17 02:12:51 client-18 xinetd[3390]: START: shell pid=17269 from=192.168.4.5
May 17 02:12:51 client-18 rshd[17270]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 02:12:51 client-18 xinetd[3390]: EXIT: shell status=0 pid=17269 duration=0(sec)
May 17 02:12:51 client-18 xinetd[3390]: START: shell pid=17306 from=192.168.4.5
May 17 02:12:51 client-18 rshd[17307]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 200e: Check files allocation from directory pool ====================================== 02:12:51 \(1305623571\)");echo XXRETCODE:$?'
May 17 02:12:51 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 200e: Check files allocation from directory pool ====================================== 02:12:51 (1305623571)
May 17 02:12:51 client-18 xinetd[3390]: EXIT: shell status=0 pid=17306 duration=0(sec)
May 17 02:12:52 client-18 xinetd[3390]: START: shell pid=17341 from=192.168.4.5
May 17 02:12:52 client-18 rshd[17342]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 02:12:52 client-18 xinetd[3390]: EXIT: shell status=0 pid=17341 duration=0(sec)
May 17 02:12:52 client-18 xinetd[3390]: START: shell pid=17377 from=192.168.4.5
May 17 02:12:52 client-18 rshd[17378]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 02:12:52 client-18 xinetd[3390]: EXIT: shell status=0 pid=17377 duration=0(sec)
May 17 02:12:52 client-18 xinetd[3390]: START: shell pid=17414 from=192.168.4.5
May 17 02:12:52 client-18 rshd[17415]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 200f: Create files in a pool ========================================================== 02:12:52 \(1305623572\)");echo XXRETCODE:$?'
May 17 02:12:52 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 200f: Create files in a pool ========================================================== 02:12:52 (1305623572)
May 17 02:12:52 client-18 xinetd[3390]: EXIT: shell status=0 pid=17414 duration=0(sec)
May 17 02:12:53 client-18 xinetd[3390]: START: shell pid=17449 from=192.168.4.5
May 17 02:12:53 client-18 rshd[17450]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 02:12:53 client-18 xinetd[3390]: EXIT: shell status=0 pid=17449 duration=0(sec)
May 17 02:12:53 client-18 xinetd[3390]: START: shell pid=17485 from=192.168.4.5
May 17 02:12:53 client-18 rshd[17486]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 02:12:53 client-18 xinetd[3390]: EXIT: shell status=0 pid=17485 duration=0(sec)
May 17 02:12:53 client-18 xinetd[3390]: START: shell pid=17522 from=192.168.4.5
May 17 02:12:53 client-18 rshd[17523]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 200g: lfs df a pool =================================================================== 02:12:53 \(1305623573\)");echo XXRETCODE:$?'
May 17 02:12:53 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 200g: lfs df a pool =================================================================== 02:12:53 (1305623573)
May 17 02:12:53 client-18 xinetd[3390]: EXIT: shell status=0 pid=17522 duration=0(sec)
May 17 02:12:53 client-18 xinetd[3390]: START: shell pid=17557 from=192.168.4.5
May 17 02:12:53 client-18 rshd[17558]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 02:12:53 client-18 xinetd[3390]: EXIT: shell status=0 pid=17557 duration=0(sec)
May 17 02:12:53 client-18 xinetd[3390]: START: shell pid=17593 from=192.168.4.5
May 17 02:12:53 client-18 rshd[17594]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 02:12:53 client-18 xinetd[3390]: EXIT: shell status=0 pid=17593 duration=0(sec)
May 17 02:12:53 client-18 xinetd[3390]: START: shell pid=17630 from=192.168.4.5
May 17 02:12:53 client-18 rshd[17631]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 200h: Create files in a pool with relative pathname =================================== 02:12:53 \(1305623573\)");echo XXRETCODE:$?'
May 17 02:12:54 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 200h: Create files in a pool with relative pathname =================================== 02:12:53 (1305623573)
May 17 02:12:54 client-18 xinetd[3390]: EXIT: shell status=0 pid=17630 duration=1(sec)
May 17 02:12:54 client-18 xinetd[3390]: START: shell pid=17665 from=192.168.4.5
May 17 02:12:54 client-18 rshd[17666]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 02:12:54 client-18 xinetd[3390]: EXIT: shell status=0 pid=17665 duration=0(sec)
May 17 02:12:54 client-18 xinetd[3390]: START: shell pid=17701 from=192.168.4.5
May 17 02:12:54 client-18 rshd[17702]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 02:12:54 client-18 xinetd[3390]: EXIT: shell status=0 pid=17701 duration=0(sec)
May 17 02:12:54 client-18 xinetd[3390]: START: shell pid=17738 from=192.168.4.5
May 17 02:12:54 client-18 rshd[17739]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 201a: Remove a target from a pool ===================================================== 02:12:54 \(1305623574\)");echo XXRETCODE:$?'
May 17 02:12:54 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 201a: Remove a target from a pool ===================================================== 02:12:54 (1305623574)
May 17 02:12:54 client-18 xinetd[3390]: EXIT: shell status=0 pid=17738 duration=0(sec)
May 17 02:13:00 client-18 xinetd[3390]: START: shell pid=17775 from=192.168.4.5
May 17 02:13:00 client-18 rshd[17776]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 02:13:00 client-18 xinetd[3390]: EXIT: shell status=0 pid=17775 duration=0(sec)
May 17 02:13:00 client-18 xinetd[3390]: START: shell pid=17811 from=192.168.4.5
May 17 02:13:01 client-18 rshd[17812]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 02:13:01 client-18 xinetd[3390]: EXIT: shell status=0 pid=17811 duration=1(sec)
May 17 02:13:01 client-18 xinetd[3390]: START: shell pid=17848 from=192.168.4.5
May 17 02:13:01 client-18 rshd[17849]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 201b: Remove all targets from a pool ================================================== 02:13:01 \(1305623581\)");echo XXRETCODE:$?'
May 17 02:13:01 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 201b: Remove all targets from a pool ================================================== 02:13:01 (1305623581)
May 17 02:13:01 client-18 xinetd[3390]: EXIT: shell status=0 pid=17848 duration=0(sec)
May 17 02:13:09 client-18 xinetd[3390]: START: shell pid=17887 from=192.168.4.5
May 17 02:13:09 client-18 rshd[17888]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 02:13:09 client-18 xinetd[3390]: EXIT: shell status=0 pid=17887 duration=0(sec)
May 17 02:13:09 client-18 xinetd[3390]: START: shell pid=17923 from=192.168.4.5
May 17 02:13:09 client-18 rshd[17924]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 02:13:09 client-18 xinetd[3390]: EXIT: shell status=0 pid=17923 duration=0(sec)
May 17 02:13:09 client-18 xinetd[3390]: START: shell pid=17960 from=192.168.4.5
May 17 02:13:10 client-18 rshd[17961]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 201c: Remove a pool =================================================================== 02:13:09 \(1305623589\)");echo XXRETCODE:$?'
May 17 02:13:10 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 201c: Remove a pool =================================================================== 02:13:09 (1305623589)
May 17 02:13:10 client-18 xinetd[3390]: EXIT: shell status=0 pid=17960 duration=1(sec)
May 17 02:13:16 client-18 xinetd[3390]: START: shell pid=17997 from=192.168.4.5
May 17 02:13:16 client-18 rshd[17998]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 02:13:16 client-18 xinetd[3390]: EXIT: shell status=0 pid=17997 duration=0(sec)
May 17 02:13:16 client-18 xinetd[3390]: START: shell pid=18033 from=192.168.4.5
May 17 02:13:16 client-18 rshd[18034]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 02:13:16 client-18 xinetd[3390]: EXIT: shell status=0 pid=18033 duration=0(sec)
May 17 02:13:16 client-18 xinetd[3390]: START: shell pid=18070 from=192.168.4.5
May 17 02:13:16 client-18 rshd[18071]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 212: Sendfile test ==================================================================== 02:13:16 \(1305623596\)");echo XXRETCODE:$?'
May 17 02:13:16 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 212: Sendfile test ==================================================================== 02:13:16 (1305623596)
May 17 02:13:16 client-18 xinetd[3390]: EXIT: shell status=0 pid=18070 duration=0(sec)
May 17 02:13:19 client-18 xinetd[3390]: START: shell pid=18105 from=192.168.4.5
May 17 02:13:19 client-18 rshd[18106]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 02:13:19 client-18 xinetd[3390]: EXIT: shell status=0 pid=18105 duration=0(sec)
May 17 02:13:19 client-18 xinetd[3390]: START: shell pid=18141 from=192.168.4.5
May 17 02:13:19 client-18 rshd[18142]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 02:13:19 client-18 xinetd[3390]: EXIT: shell status=0 pid=18141 duration=0(sec)
May 17 02:13:19 client-18 xinetd[3390]: START: shell pid=18178 from=192.168.4.5
May 17 02:13:19 client-18 rshd[18179]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 213: OSC lock completion and cancel race don\'t crash - bug 18829 ====================== 02:13:19 \(1305623599\)");echo XXRETCODE:$?'
May 17 02:13:19 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 213: OSC lock completion and cancel race don't crash - bug 18829 ====================== 02:13:19 (1305623599)
May 17 02:13:19 client-18 xinetd[3390]: EXIT: shell status=0 pid=18178 duration=0(sec)
May 17 02:13:19 client-18 xinetd[3390]: START: shell pid=18213 from=192.168.4.5
May 17 02:13:19 client-18 rshd[18214]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 02:13:19 client-18 xinetd[3390]: EXIT: shell status=0 pid=18213 duration=0(sec)
May 17 02:13:19 client-18 xinetd[3390]: START: shell pid=18249 from=192.168.4.5
May 17 02:13:19 client-18 rshd[18250]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 02:13:19 client-18 xinetd[3390]: EXIT: shell status=0 pid=18249 duration=0(sec)
May 17 02:13:20 client-18 xinetd[3390]: START: shell pid=18286 from=192.168.4.5
May 17 02:13:20 client-18 rshd[18287]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 214: hash-indexed directory test - bug 20133 ========================================== 02:13:19 \(1305623599\)");echo XXRETCODE:$?'
May 17 02:13:20 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 214: hash-indexed directory test - bug 20133 ========================================== 02:13:19 (1305623599)
May 17 02:13:20 client-18 xinetd[3390]: EXIT: shell status=0 pid=18286 duration=0(sec)
May 17 02:13:21 client-18 xinetd[3390]: START: shell pid=18321 from=192.168.4.5
May 17 02:13:21 client-18 rshd[18322]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 02:13:21 client-18 xinetd[3390]: EXIT: shell status=0 pid=18321 duration=0(sec)
May 17 02:13:21 client-18 xinetd[3390]: START: shell pid=18357 from=192.168.4.5
May 17 02:13:21 client-18 rshd[18358]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 02:13:21 client-18 xinetd[3390]: EXIT: shell status=0 pid=18357 duration=0(sec)
May 17 02:13:21 client-18 xinetd[3390]: START: shell pid=18394 from=192.168.4.5
May 17 02:13:21 client-18 rshd[18395]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 215: \/proc\/sys\/lnet exists and has proper content - bugs 18102, 21079, 21517 ========== 02:13:21 \(1305623601\)");echo XXRETCODE:$?'
May 17 02:13:21 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 215: /proc/sys/lnet exists and has proper content - bugs 18102, 21079, 21517 ========== 02:13:21 (1305623601)
May 17 02:13:21 client-18 xinetd[3390]: EXIT: shell status=0 pid=18394 duration=0(sec)
May 17 02:13:22 client-18 xinetd[3390]: START: shell pid=18429 from=192.168.4.5
May 17 02:13:22 client-18 rshd[18430]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 02:13:22 client-18 xinetd[3390]: EXIT: shell status=0 pid=18429 duration=0(sec)
May 17 02:13:22 client-18 xinetd[3390]: START: shell pid=18465 from=192.168.4.5
May 17 02:13:22 client-18 rshd[18466]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 02:13:22 client-18 xinetd[3390]: EXIT: shell status=0 pid=18465 duration=0(sec)
May 17 02:13:22 client-18 xinetd[3390]: START: shell pid=18502 from=192.168.4.5
May 17 02:13:22 client-18 rshd[18503]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 216: check lockless direct write works and updates file size and kms correctly ======== 02:13:22 \(1305623602\)");echo XXRETCODE:$?'
May 17 02:13:22 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 216: check lockless direct write works and updates file size and kms correctly ======== 02:13:22 (1305623602)
May 17 02:13:22 client-18 xinetd[3390]: EXIT: shell status=0 pid=18502 duration=0(sec)
May 17 02:13:25 client-18 xinetd[3390]: START: shell pid=18537 from=192.168.4.5
May 17 02:13:25 client-18 rshd[18538]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 02:13:25 client-18 xinetd[3390]: EXIT: shell status=0 pid=18537 duration=0(sec)
May 17 02:13:25 client-18 xinetd[3390]: START: shell pid=18573 from=192.168.4.5
May 17 02:13:25 client-18 rshd[18574]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 02:13:25 client-18 xinetd[3390]: EXIT: shell status=0 pid=18573 duration=0(sec)
May 17 02:13:25 client-18 xinetd[3390]: START: shell pid=18610 from=192.168.4.5
May 17 02:13:25 client-18 rshd[18611]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 217: check lctl ping for hostnames with hiphen \(\'-\'\) ================================== 02:13:25 \(1305623605\)");echo XXRETCODE:$?'
May 17 02:13:25 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 217: check lctl ping for hostnames with hiphen ('-') ================================== 02:13:25 (1305623605)
May 17 02:13:25 client-18 xinetd[3390]: EXIT: shell status=0 pid=18610 duration=0(sec)
May 17 02:13:25 client-18 xinetd[3390]: START: shell pid=18645 from=192.168.4.5
May 17 02:13:25 client-18 rshd[18646]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 02:13:25 client-18 xinetd[3390]: EXIT: shell status=0 pid=18645 duration=0(sec)
May 17 02:13:25 client-18 xinetd[3390]: START: shell pid=18681 from=192.168.4.5
May 17 02:13:25 client-18 rshd[18682]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 02:13:26 client-18 xinetd[3390]: EXIT: shell status=0 pid=18681 duration=1(sec)
May 17 02:13:26 client-18 xinetd[3390]: START: shell pid=18718 from=192.168.4.5
May 17 02:13:26 client-18 rshd[18719]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 218: parallel read and truncate should not deadlock =================================== 02:13:26 \(1305623606\)");echo XXRETCODE:$?'
May 17 02:13:26 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 218: parallel read and truncate should not deadlock =================================== 02:13:26 (1305623606)
May 17 02:13:26 client-18 xinetd[3390]: EXIT: shell status=0 pid=18718 duration=0(sec)
May 17 02:13:26 client-18 xinetd[3390]: START: shell pid=18753 from=192.168.4.5
May 17 02:13:26 client-18 rshd[18754]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark creating a 10 Mb file");echo XXRETCODE:$?'
May 17 02:13:26 client-18 kernel: Lustre: DEBUG MARKER: creating a 10 Mb file
May 17 02:13:26 client-18 xinetd[3390]: EXIT: shell status=0 pid=18753 duration=0(sec)
May 17 02:13:26 client-18 xinetd[3390]: START: shell pid=18788 from=192.168.4.5
May 17 02:13:26 client-18 rshd[18789]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark starting reads");echo XXRETCODE:$?'
May 17 02:13:26 client-18 kernel: Lustre: DEBUG MARKER: starting reads
May 17 02:13:26 client-18 xinetd[3390]: EXIT: shell status=0 pid=18788 duration=0(sec)
May 17 02:13:26 client-18 xinetd[3390]: START: shell pid=18823 from=192.168.4.5
May 17 02:13:26 client-18 rshd[18824]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark truncating the file");echo XXRETCODE:$?'
May 17 02:13:26 client-18 kernel: Lustre: DEBUG MARKER: truncating the file
May 17 02:13:26 client-18 xinetd[3390]: EXIT: shell status=0 pid=18823 duration=0(sec)
May 17 02:13:26 client-18 xinetd[3390]: START: shell pid=18858 from=192.168.4.5
May 17 02:13:27 client-18 rshd[18859]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark killing dd");echo XXRETCODE:$?'
May 17 02:13:27 client-18 kernel: Lustre: DEBUG MARKER: killing dd
May 17 02:13:27 client-18 xinetd[3390]: EXIT: shell status=0 pid=18858 duration=1(sec)
May 17 02:13:27 client-18 xinetd[3390]: START: shell pid=18893 from=192.168.4.5
May 17 02:13:27 client-18 rshd[18894]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark removing the temporary file");echo XXRETCODE:$?'
May 17 02:13:27 client-18 kernel: Lustre: DEBUG MARKER: removing the temporary file
May 17 02:13:27 client-18 xinetd[3390]: EXIT: shell status=0 pid=18893 duration=0(sec)
May 17 02:13:27 client-18 xinetd[3390]: START: shell pid=18928 from=192.168.4.5
May 17 02:13:27 client-18 rshd[18929]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 02:13:27 client-18 xinetd[3390]: EXIT: shell status=0 pid=18928 duration=0(sec)
May 17 02:13:27 client-18 xinetd[3390]: START: shell pid=18964 from=192.168.4.5
May 17 02:13:27 client-18 rshd[18965]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 02:13:27 client-18 xinetd[3390]: EXIT: shell status=0 pid=18964 duration=0(sec)
May 17 02:13:27 client-18 xinetd[3390]: START: shell pid=19001 from=192.168.4.5
May 17 02:13:27 client-18 rshd[19002]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity test 900: umount should not race with any mgc requeue thread =============================== 02:13:27 \(1305623607\)");echo XXRETCODE:$?'
May 17 02:13:27 client-18 kernel: Lustre: DEBUG MARKER: == sanity test 900: umount should not race with any mgc requeue thread =============================== 02:13:27 (1305623607)
May 17 02:13:27 client-18 xinetd[3390]: EXIT: shell status=0 pid=19001 duration=0(sec)
May 17 02:13:27 client-18 xinetd[3390]: START: shell pid=19036 from=192.168.4.5
May 17 02:13:27 client-18 rshd[19037]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:; lsof -t /mnt/lustre || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount  /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre is still busy, wait one second && sleep 1;     umount  /mnt/lustre; fi fi");echo XXRETCODE:$?'
May 17 02:13:28 client-18 kernel: LustreError: 19078:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway
May 17 02:13:28 client-18 kernel: LustreError: 19078:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108
May 17 02:13:28 client-18 kernel: Lustre: client ffff8103183da000 umount complete
May 17 02:13:28 client-18 xinetd[3390]: EXIT: shell status=0 pid=19036 duration=1(sec)
May 17 02:13:28 client-18 xinetd[3390]: START: shell pid=19080 from=192.168.4.5
May 17 02:13:28 client-18 rshd[19081]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:; lsof -t /mnt/lustre2 || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre2 | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount  /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre2 is still busy, wait one second && sleep 1;     umount  /mnt/lustre2; fi fi");echo XXRETCODE:$?'
May 17 02:13:28 client-18 xinetd[3390]: EXIT: shell status=0 pid=19080 duration=0(sec)
May 17 02:14:04 client-18 xinetd[3390]: START: shell pid=19121 from=192.168.4.5
May 17 02:14:04 client-18 rshd[19122]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); mpts=\$(mount | grep -w -c /mnt/lustre); if [ \$running -ne \$mpts ]; then     echo \$(hostname) env are INSANE!;     exit 1; fi");echo XXRETCODE:$?'
May 17 02:14:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=19121 duration=0(sec)
May 17 02:14:04 client-18 xinetd[3390]: START: shell pid=19161 from=192.168.4.5
May 17 02:14:04 client-18 rshd[19162]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); mpts=\$(mount | grep -w -c /mnt/lustre2); if [ \$running -ne \$mpts ]; then     echo \$(hostname) env are INSANE!;     exit 1; fi");echo XXRETCODE:$?'
May 17 02:14:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=19161 duration=0(sec)
May 17 02:14:19 client-18 xinetd[3390]: START: shell pid=19201 from=192.168.4.5
May 17 02:14:19 client-18 rshd[19202]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c " running=\$(mount | grep -c /mnt/lustre' '); rc=0; if [ \$running -eq 0 ] ; then     mkdir -p /mnt/lustre;     mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre;     rc=\$?; fi; exit \$rc");echo XXRETCODE:$?'
May 17 02:14:19 client-18 kernel: Lustre: 19241:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null
May 17 02:14:19 client-18 kernel: Lustre: 19241:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 7 previous similar messages
May 17 02:14:19 client-18 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'.
May 17 02:14:19 client-18 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import
May 17 02:14:19 client-18 kernel: Lustre: Increasing default stripe size to min 1048576
May 17 02:14:19 client-18 kernel: Lustre: Client lustre-client has started
May 17 02:14:19 client-18 xinetd[3390]: EXIT: shell status=0 pid=19201 duration=0(sec)
May 17 02:14:19 client-18 xinetd[3390]: START: shell pid=19254 from=192.168.4.5
May 17 02:14:19 client-18 rshd[19255]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "mount | grep -w /mnt/lustre");echo XXRETCODE:$?'
May 17 02:14:19 client-18 xinetd[3390]: EXIT: shell status=0 pid=19254 duration=0(sec)
May 17 02:14:19 client-18 xinetd[3390]: START: shell pid=19291 from=192.168.4.5
May 17 02:14:20 client-18 rshd[19292]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param debug=\"0x33f0406\";         lctl set_param subsystem_debug=\"0xffb7e3ff\";         lctl set_param debug_mb=32;");echo XXRETCODE:$?'
May 17 02:14:20 client-18 kernel: Lustre: 19326:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release.
May 17 02:14:20 client-18 kernel: Lustre: 19326:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message
May 17 02:14:20 client-18 xinetd[3390]: EXIT: shell status=0 pid=19291 duration=1(sec)
May 17 02:14:20 client-18 xinetd[3390]: START: shell pid=19328 from=192.168.4.5
May 17 02:14:20 client-18 rshd[19329]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark Using TIMEOUT=20");echo XXRETCODE:$?'
May 17 02:14:20 client-18 kernel: Lustre: DEBUG MARKER: Using TIMEOUT=20
May 17 02:14:20 client-18 xinetd[3390]: EXIT: shell status=0 pid=19328 duration=0(sec)
May 17 02:14:25 client-18 xinetd[3390]: START: shell pid=19363 from=192.168.4.5
May 17 02:14:25 client-18 rshd[19364]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 02:14:26 client-18 xinetd[3390]: EXIT: shell status=0 pid=19363 duration=1(sec)
May 17 02:14:26 client-18 xinetd[3390]: START: shell pid=19399 from=192.168.4.5
May 17 02:14:26 client-18 rshd[19400]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 02:14:26 client-18 xinetd[3390]: EXIT: shell status=0 pid=19399 duration=0(sec)
May 17 02:14:26 client-18 xinetd[3390]: START: shell pid=19436 from=192.168.4.5
May 17 02:14:26 client-18 rshd[19437]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity sanity.sh test complete, duration 7201 sec ================================================= 02:14:26 \(1305623666\)");echo XXRETCODE:$?'
May 17 02:14:26 client-18 kernel: Lustre: DEBUG MARKER: == sanity sanity.sh test complete, duration 7201 sec ================================================= 02:14:26 (1305623666)
May 17 02:14:26 client-18 xinetd[3390]: EXIT: shell status=0 pid=19436 duration=0(sec)
May 17 02:14:27 client-18 xinetd[3390]: START: shell pid=19471 from=192.168.4.5
May 17 02:14:27 client-18 rshd[19472]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark -----============= acceptance-small: metadata-updates ============----- Tue May 17 02:14:27 PDT 2011");echo XXRETCODE:$?'
May 17 02:14:27 client-18 kernel: Lustre: DEBUG MARKER: -----============= acceptance-small: metadata-updates ============----- Tue May 17 02:14:27 PDT 2011
May 17 02:14:27 client-18 xinetd[3390]: EXIT: shell status=0 pid=19471 duration=0(sec)
May 17 02:14:27 client-18 xinetd[3390]: START: shell pid=19506 from=192.168.4.5
May 17 02:14:27 client-18 rshd[19507]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "echo \$(hostname)");echo XXRETCODE:$?'
May 17 02:14:27 client-18 xinetd[3390]: EXIT: shell status=0 pid=19506 duration=0(sec)
May 17 02:14:27 client-18 xinetd[3390]: START: shell pid=19543 from=192.168.4.5
May 17 02:14:27 client-18 rshd[19544]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark ===== \/usr\/lib64\/lustre\/tests\/metadata-updates.sh ====== ");echo XXRETCODE:$?'
May 17 02:14:27 client-18 kernel: Lustre: DEBUG MARKER: ===== /usr/lib64/lustre/tests/metadata-updates.sh ======
May 17 02:14:27 client-18 xinetd[3390]: EXIT: shell status=0 pid=19543 duration=0(sec)
May 17 02:14:27 client-18 xinetd[3390]: START: shell pid=19578 from=192.168.4.5
May 17 02:14:27 client-18 rshd[19579]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/tests:/usr/lib64/openmpi/1.4-gcc/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/usr/lib64/lustre/utils:/usr/lib64/lustre/utils/gss:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey::/usr/lib64/lustre/tests/mpi NAME=sarah sh rpc.sh check_config_client /mnt/lustre ");echo XXRETCODE:$?'
May 17 02:14:27 client-18 xinetd[3390]: EXIT: shell status=0 pid=19578 duration=0(sec)
May 17 02:14:28 client-18 xinetd[3390]: START: shell pid=19672 from=192.168.4.5
May 17 02:14:28 client-18 rshd[19673]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); mpts=\$(mount | grep -w -c /mnt/lustre); if [ \$running -ne \$mpts ]; then     echo \$(hostname) env are INSANE!;     exit 1; fi");echo XXRETCODE:$?'
May 17 02:14:28 client-18 xinetd[3390]: EXIT: shell status=0 pid=19672 duration=0(sec)
May 17 02:14:28 client-18 xinetd[3390]: START: shell pid=19712 from=192.168.4.5
May 17 02:14:28 client-18 rshd[19713]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); mpts=\$(mount | grep -w -c /mnt/lustre2); if [ \$running -ne \$mpts ]; then     echo \$(hostname) env are INSANE!;     exit 1; fi");echo XXRETCODE:$?'
May 17 02:14:28 client-18 xinetd[3390]: EXIT: shell status=0 pid=19712 duration=0(sec)
May 17 02:14:30 client-18 xinetd[3390]: START: shell pid=19752 from=192.168.4.5
May 17 02:14:30 client-18 rshd[19753]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark Using TIMEOUT=20");echo XXRETCODE:$?'
May 17 02:14:30 client-18 kernel: Lustre: DEBUG MARKER: Using TIMEOUT=20
May 17 02:14:30 client-18 xinetd[3390]: EXIT: shell status=0 pid=19752 duration=0(sec)
May 17 02:14:31 client-18 xinetd[3390]: START: shell pid=19787 from=192.168.4.5
May 17 02:14:31 client-18 rshd[19788]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param debug=\"0x33f0406\";             lctl set_param subsystem_debug=\"0xffb7e3ff\";             lctl set_param debug_mb=32;             sync");echo XXRETCODE:$?'
May 17 02:14:31 client-18 kernel: Lustre: 19822:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release.
May 17 02:14:31 client-18 kernel: Lustre: 19822:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message
May 17 02:14:31 client-18 xinetd[3390]: EXIT: shell status=0 pid=19787 duration=0(sec)
May 17 02:14:31 client-18 xinetd[3390]: START: shell pid=19825 from=192.168.4.5
May 17 02:14:31 client-18 rshd[19826]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "cat /proc/mounts");echo XXRETCODE:$?'
May 17 02:14:31 client-18 xinetd[3390]: EXIT: shell status=0 pid=19825 duration=0(sec)
May 17 02:14:32 client-18 xinetd[3390]: START: shell pid=19860 from=192.168.4.5
May 17 02:14:32 client-18 rshd[19861]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl get_param -n osc.*OST*-osc-[^M][^D][^T]*.srpc_info");echo XXRETCODE:$?'
May 17 02:14:32 client-18 xinetd[3390]: EXIT: shell status=0 pid=19860 duration=0(sec)
May 17 02:14:32 client-18 xinetd[3390]: START: shell pid=19895 from=192.168.4.5
May 17 02:14:32 client-18 rshd[19896]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl get_param -n mdc.*-MDT*-mdc-*.srpc_info");echo XXRETCODE:$?'
May 17 02:14:32 client-18 xinetd[3390]: EXIT: shell status=0 pid=19895 duration=0(sec)
May 17 02:14:32 client-18 xinetd[3390]: START: shell pid=19930 from=192.168.4.5
May 17 02:14:32 client-18 rshd[19931]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "set +x; DIR=/mnt/lustre/d0.metadata-updates/\$(hostname); TESTFILE=\$DIR/testfile; rm -f \$TESTFILE; rm -f /mnt/lustre/d0.metadata-updates/mdsum; rmdir \$DIR 2>/dev/null; mkdir -p \$DIR");echo XXRETCODE:$?'
May 17 02:14:32 client-18 xinetd[3390]: EXIT: shell status=0 pid=19930 duration=0(sec)
May 17 02:14:32 client-18 xinetd[3390]: START: shell pid=19970 from=192.168.4.5
May 17 02:14:32 client-18 rshd[19971]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "set +x; TESTFILE=/mnt/lustre/d0.metadata-updates/\$(hostname)/testfile; mcreate \$TESTFILE; ");echo XXRETCODE:$?'
May 17 02:14:32 client-18 xinetd[3390]: EXIT: shell status=0 pid=19970 duration=0(sec)
May 17 02:14:32 client-18 xinetd[3390]: START: shell pid=20007 from=192.168.4.5
May 17 02:14:32 client-18 rshd[20008]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "set +x; TESTFILE=/mnt/lustre/d0.metadata-updates/\$(hostname)/testfile; dd if=/dev/zero of=\$TESTFILE bs=1024 count=1 2>/dev/null || exit 54; echo \$(hostname) | dd of=\$TESTFILE conv=notrunc 2>/dev/null || exit 55;  md5sum \$TESTFILE >> /mnt/lustre/d0.metadata-updates/mdsum; ");echo XXRETCODE:$?'
May 17 02:14:32 client-18 xinetd[3390]: EXIT: shell status=0 pid=20007 duration=0(sec)
May 17 02:14:32 client-18 xinetd[3390]: START: shell pid=20050 from=192.168.4.5
May 17 02:14:32 client-18 rshd[20051]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "md5sum --check /mnt/lustre/d0.metadata-updates/mdsum");echo XXRETCODE:$?'
May 17 02:14:33 client-18 xinetd[3390]: EXIT: shell status=0 pid=20050 duration=1(sec)
May 17 02:14:33 client-18 xinetd[3390]: START: shell pid=20085 from=192.168.4.5
May 17 02:14:33 client-18 rshd[20086]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "set +x; for HOST in client-18.lab.whamcloud.com client-5-ib  ; do     TESTFILE=/mnt/lustre/d0.metadata-updates/\$HOST/testfile;     tmp=\$(stat -c \"%u %g %s 0%a\" \$TESTFILE);     echo \"\$TESTFILE [ uid gid size mode ] expected : 0 0 1024 0644 ;  got : \$tmp \";     if [ x\"\$tmp\" != x\"0 0 1024 0644\" ] ; then         echo \"Wrong file attributes\";         exit 56;     fi; done ");echo XXRETCODE:$?'
May 17 02:14:33 client-18 xinetd[3390]: EXIT: shell status=0 pid=20085 duration=0(sec)
May 17 02:14:33 client-18 xinetd[3390]: START: shell pid=20124 from=192.168.4.5
May 17 02:14:33 client-18 rshd[20125]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "set +x; TESTFILE=/mnt/lustre/d0.metadata-updates/\$(hostname)/testfile; chmod 0222 \$TESTFILE");echo XXRETCODE:$?'
May 17 02:14:33 client-18 xinetd[3390]: EXIT: shell status=0 pid=20124 duration=0(sec)
May 17 02:14:33 client-18 xinetd[3390]: START: shell pid=20161 from=192.168.4.5
May 17 02:14:33 client-18 rshd[20162]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "set +x; for HOST in client-18.lab.whamcloud.com client-5-ib  ; do     TESTFILE=/mnt/lustre/d0.metadata-updates/\$HOST/testfile;     tmp=\$(stat -c \"%u %g %s 0%a\" \$TESTFILE);     echo \"\$TESTFILE [ uid gid size mode ] expected : 0 0 1024 0222 ;  got : \$tmp \";     if [ x\"\$tmp\" != x\"0 0 1024 0222\" ] ; then         echo \"Wrong file attributes\";         exit 56;     fi; done ");echo XXRETCODE:$?'
May 17 02:14:33 client-18 xinetd[3390]: EXIT: shell status=0 pid=20161 duration=0(sec)
May 17 02:14:33 client-18 xinetd[3390]: START: shell pid=20200 from=192.168.4.5
May 17 02:14:33 client-18 rshd[20201]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c " set +x; TESTFILE=/mnt/lustre/d0.metadata-updates/\$(hostname)/testfile; touch -c --date=\"2001-01-01 GMT\" -a \$TESTFILE; touch -c --date=\"2005-05-05 GMT\" -m \$TESTFILE ");echo XXRETCODE:$?'
May 17 02:14:33 client-18 xinetd[3390]: EXIT: shell status=0 pid=20200 duration=0(sec)
May 17 02:14:33 client-18 xinetd[3390]: START: shell pid=20238 from=192.168.4.5
May 17 02:14:33 client-18 rshd[20239]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "set +x; for HOST in client-18.lab.whamcloud.com client-5-ib  ; do     TESTFILE=/mnt/lustre/d0.metadata-updates/\$HOST/testfile;     tmp=\$(stat -c \"%X %Y\" \$TESTFILE);     if [ x\"\$tmp\" != x\"978307200 1115251200\" ] ; then        echo \"\$TESTFILE [ atime mtime ] expected : 978307200 1115251200 ;  got : \$tmp \";        RC=57;     fi; done; exit \$RC");echo XXRETCODE:$?'
May 17 02:14:33 client-18 xinetd[3390]: EXIT: shell status=0 pid=20238 duration=0(sec)
May 17 02:14:33 client-18 xinetd[3390]: START: shell pid=20277 from=192.168.4.5
May 17 02:14:33 client-18 rshd[20278]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "set +x; TESTFILE=/mnt/lustre/d0.metadata-updates/\$(hostname)/testfile; /usr/lib64/lustre/tests/truncate \$TESTFILE 0");echo XXRETCODE:$?'
May 17 02:14:33 client-18 xinetd[3390]: EXIT: shell status=0 pid=20277 duration=0(sec)
May 17 02:14:33 client-18 xinetd[3390]: START: shell pid=20314 from=192.168.4.5
May 17 02:14:33 client-18 rshd[20315]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "set +x; for HOST in client-18.lab.whamcloud.com client-5-ib  ; do     TESTFILE=/mnt/lustre/d0.metadata-updates/\$HOST/testfile;     tmp=\$(stat -c \"%u %g %s 0%a\" \$TESTFILE);     echo \"\$TESTFILE [ uid gid size mode ] expected : 0 0 0 0222 ;  got : \$tmp \";     if [ x\"\$tmp\" != x\"0 0 0 0222\" ] ; then         echo \"Wrong file attributes\";         exit 56;     fi; done ");echo XXRETCODE:$?'
May 17 02:14:33 client-18 xinetd[3390]: EXIT: shell status=0 pid=20314 duration=0(sec)
May 17 02:14:33 client-18 xinetd[3390]: START: shell pid=20353 from=192.168.4.5
May 17 02:14:33 client-18 rshd[20354]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "set +x; TESTFILE=/mnt/lustre/d0.metadata-updates/\$(hostname)/testfile; rm -f \$TESTFILE; DIR=/mnt/lustre/d0.metadata-updates/\$(hostname); for i in \$(seq 1000) ; do     touch \$DIR/f\$i; done ");echo XXRETCODE:$?'
May 17 02:14:36 client-18 xinetd[3390]: EXIT: shell status=0 pid=20353 duration=3(sec)
May 17 02:14:36 client-18 xinetd[3390]: START: shell pid=21395 from=192.168.4.5
May 17 02:14:36 client-18 rshd[21396]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "set +x; for HOST in client-18.lab.whamcloud.com client-5-ib  ; do     DIR=/mnt/lustre/d0.metadata-updates/\$HOST;     for i in \$(seq 1000 -1 1) ; do         if ! [ -f \$DIR/f\$i ] ; then             echo \"ERROR: file \$DIR/f\$i should exist\";             RC=1;         fi;     done;     for i in \$(seq 0) ; do         if [ -f \$DIR/f\$i ] ; then             echo \"ERROR: deleted file \$DIR/f\$i exists\";             RC=1;         fi;     done; done; exit \$RC ");echo XXRETCODE:$?'
May 17 02:14:39 client-18 xinetd[3390]: EXIT: shell status=0 pid=21395 duration=3(sec)
May 17 02:14:39 client-18 xinetd[3390]: START: shell pid=21438 from=192.168.4.5
May 17 02:14:39 client-18 rshd[21439]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "set +x; DIR=/mnt/lustre/d0.metadata-updates/\$(hostname); for i in \$(seq 500) ; do     if ! rm -f \$DIR/f\$i ; then         exit 1;     fi; done ");echo XXRETCODE:$?'
May 17 02:14:40 client-18 xinetd[3390]: EXIT: shell status=0 pid=21438 duration=1(sec)
May 17 02:14:40 client-18 xinetd[3390]: START: shell pid=21978 from=192.168.4.5
May 17 02:14:40 client-18 rshd[21979]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "set +x; for HOST in client-18.lab.whamcloud.com client-5-ib  ; do     DIR=/mnt/lustre/d0.metadata-updates/\$HOST;     for i in \$(seq 1000 -1 501) ; do         if ! [ -f \$DIR/f\$i ] ; then             echo \"ERROR: file \$DIR/f\$i should exist\";             RC=1;         fi;     done;     for i in \$(seq 500) ; do         if [ -f \$DIR/f\$i ] ; then             echo \"ERROR: deleted file \$DIR/f\$i exists\";             RC=1;         fi;     done; done; exit \$RC ");echo XXRETCODE:$?'
May 17 02:14:40 client-18 xinetd[3390]: EXIT: shell status=0 pid=21978 duration=0(sec)
May 17 02:16:52 client-18 init: Trying to re-exec init
May 17 12:31:41 client-18 xinetd[3390]: START: shell pid=25696 from=192.168.4.5
May 17 12:31:41 client-18 rshd[25697]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == metadata-updates metadata-updates.sh test complete, duration 37034 sec ============================ 12:31:41 \(1305660701\)");echo XXRETCODE:$?'
May 17 12:31:41 client-18 kernel: Lustre: DEBUG MARKER: == metadata-updates metadata-updates.sh test complete, duration 37034 sec ============================ 12:31:41 (1305660701)
May 17 12:31:41 client-18 xinetd[3390]: EXIT: shell status=0 pid=25696 duration=0(sec)
May 17 12:31:42 client-18 xinetd[3390]: START: shell pid=25732 from=192.168.4.5
May 17 12:31:42 client-18 rshd[25733]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark -----============= acceptance-small: sanity-benchmark ============----- Tue May 17 12:31:42 PDT 2011");echo XXRETCODE:$?'
May 17 12:31:42 client-18 kernel: Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-benchmark ============----- Tue May 17 12:31:42 PDT 2011
May 17 12:31:42 client-18 xinetd[3390]: EXIT: shell status=0 pid=25732 duration=0(sec)
May 17 12:31:42 client-18 xinetd[3390]: START: shell pid=25767 from=192.168.4.5
May 17 12:31:42 client-18 rshd[25768]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark excepting tests: ");echo XXRETCODE:$?'
May 17 12:31:42 client-18 kernel: Lustre: DEBUG MARKER: excepting tests:
May 17 12:31:42 client-18 xinetd[3390]: EXIT: shell status=0 pid=25767 duration=0(sec)
May 17 12:31:42 client-18 xinetd[3390]: START: shell pid=25802 from=192.168.4.5
May 17 12:31:42 client-18 rshd[25803]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/tests:/usr/lib64/openmpi/1.4-gcc/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/usr/lib64/lustre/utils:/usr/lib64/lustre/utils/gss:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey::/usr/lib64/lustre/tests/mpi NAME=sarah sh rpc.sh check_config_client /mnt/lustre ");echo XXRETCODE:$?'
May 17 12:31:43 client-18 xinetd[3390]: EXIT: shell status=0 pid=25802 duration=1(sec)
May 17 12:31:43 client-18 xinetd[3390]: START: shell pid=25896 from=192.168.4.5
May 17 12:31:44 client-18 rshd[25897]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); mpts=\$(mount | grep -w -c /mnt/lustre); if [ \$running -ne \$mpts ]; then     echo \$(hostname) env are INSANE!;     exit 1; fi");echo XXRETCODE:$?'
May 17 12:31:44 client-18 xinetd[3390]: EXIT: shell status=0 pid=25896 duration=1(sec)
May 17 12:31:44 client-18 xinetd[3390]: START: shell pid=25936 from=192.168.4.5
May 17 12:31:44 client-18 rshd[25937]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); mpts=\$(mount | grep -w -c /mnt/lustre2); if [ \$running -ne \$mpts ]; then     echo \$(hostname) env are INSANE!;     exit 1; fi");echo XXRETCODE:$?'
May 17 12:31:44 client-18 xinetd[3390]: EXIT: shell status=0 pid=25936 duration=0(sec)
May 17 12:31:46 client-18 xinetd[3390]: START: shell pid=25976 from=192.168.4.5
May 17 12:31:46 client-18 rshd[25977]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark Using TIMEOUT=20");echo XXRETCODE:$?'
May 17 12:31:46 client-18 kernel: Lustre: DEBUG MARKER: Using TIMEOUT=20
May 17 12:31:46 client-18 xinetd[3390]: EXIT: shell status=0 pid=25976 duration=0(sec)
May 17 12:31:46 client-18 xinetd[3390]: START: shell pid=26011 from=192.168.4.5
May 17 12:31:46 client-18 rshd[26012]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param debug=\"0x33f0406\";             lctl set_param subsystem_debug=\"0xffb7e3ff\";             lctl set_param debug_mb=32;             sync");echo XXRETCODE:$?'
May 17 12:31:46 client-18 kernel: Lustre: 26046:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release.
May 17 12:31:46 client-18 kernel: Lustre: 26046:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message
May 17 12:31:46 client-18 xinetd[3390]: EXIT: shell status=0 pid=26011 duration=0(sec)
May 17 12:31:46 client-18 xinetd[3390]: START: shell pid=26049 from=192.168.4.5
May 17 12:31:46 client-18 rshd[26050]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "cat /proc/mounts");echo XXRETCODE:$?'
May 17 12:31:46 client-18 xinetd[3390]: EXIT: shell status=0 pid=26049 duration=0(sec)
May 17 12:31:47 client-18 xinetd[3390]: START: shell pid=26084 from=192.168.4.5
May 17 12:31:47 client-18 rshd[26085]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl get_param -n osc.*OST*-osc-[^M][^D][^T]*.srpc_info");echo XXRETCODE:$?'
May 17 12:31:47 client-18 xinetd[3390]: EXIT: shell status=0 pid=26084 duration=0(sec)
May 17 12:31:47 client-18 xinetd[3390]: START: shell pid=26119 from=192.168.4.5
May 17 12:31:47 client-18 rshd[26120]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl get_param -n mdc.*-MDT*-mdc-*.srpc_info");echo XXRETCODE:$?'
May 17 12:31:47 client-18 xinetd[3390]: EXIT: shell status=0 pid=26119 duration=0(sec)
May 17 12:31:47 client-18 xinetd[3390]: START: shell pid=26154 from=192.168.4.5
May 17 12:31:47 client-18 rshd[26155]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity-benchmark test dbench: dbench ============================================================== 12:31:47 \(1305660707\)");echo XXRETCODE:$?'
May 17 12:31:47 client-18 kernel: Lustre: DEBUG MARKER: == sanity-benchmark test dbench: dbench ============================================================== 12:31:47 (1305660707)
May 17 12:31:47 client-18 xinetd[3390]: EXIT: shell status=0 pid=26154 duration=0(sec)
May 17 12:55:52 client-18 xinetd[3390]: START: shell pid=26265 from=192.168.4.5
May 17 12:55:52 client-18 rshd[26266]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 12:55:52 client-18 xinetd[3390]: EXIT: shell status=0 pid=26265 duration=0(sec)
May 17 12:55:52 client-18 xinetd[3390]: START: shell pid=26301 from=192.168.4.5
May 17 12:55:52 client-18 rshd[26302]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 12:55:52 client-18 xinetd[3390]: EXIT: shell status=0 pid=26301 duration=0(sec)
May 17 12:55:52 client-18 xinetd[3390]: START: shell pid=26338 from=192.168.4.5
May 17 12:55:52 client-18 rshd[26339]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity-benchmark test bonnie: bonnie++ ============================================================ 12:55:52 \(1305662152\)");echo XXRETCODE:$?'
May 17 12:55:52 client-18 kernel: Lustre: DEBUG MARKER: == sanity-benchmark test bonnie: bonnie++ ============================================================ 12:55:52 (1305662152)
May 17 12:55:52 client-18 xinetd[3390]: EXIT: shell status=0 pid=26338 duration=0(sec)
May 17 12:55:52 client-18 xinetd[3390]: START: shell pid=26373 from=192.168.4.5
May 17 12:55:52 client-18 rshd[26374]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark min OST has 8907404kB available, using 24580944kB file size");echo XXRETCODE:$?'
May 17 12:55:52 client-18 kernel: Lustre: DEBUG MARKER: min OST has 8907404kB available, using 24580944kB file size
May 17 12:55:52 client-18 xinetd[3390]: EXIT: shell status=0 pid=26373 duration=0(sec)
May 17 13:08:07 client-18 xinetd[3390]: START: shell pid=26454 from=192.168.4.5
May 17 13:08:07 client-18 rshd[26455]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 13:08:07 client-18 xinetd[3390]: EXIT: shell status=0 pid=26454 duration=0(sec)
May 17 13:08:08 client-18 xinetd[3390]: START: shell pid=26490 from=192.168.4.5
May 17 13:08:08 client-18 rshd[26491]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 13:08:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=26490 duration=0(sec)
May 17 13:08:08 client-18 xinetd[3390]: START: shell pid=26527 from=192.168.4.5
May 17 13:08:08 client-18 rshd[26528]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity-benchmark test iozone: iozone ============================================================== 13:08:08 \(1305662888\)");echo XXRETCODE:$?'
May 17 13:08:08 client-18 kernel: Lustre: DEBUG MARKER: == sanity-benchmark test iozone: iozone ============================================================== 13:08:08 (1305662888)
May 17 13:08:09 client-18 xinetd[3390]: EXIT: shell status=0 pid=26527 duration=1(sec)
May 17 13:08:09 client-18 xinetd[3390]: START: shell pid=26562 from=192.168.4.5
May 17 13:08:09 client-18 rshd[26563]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark min OST has 8917752kB available, using 24580944kB file size");echo XXRETCODE:$?'
May 17 13:08:09 client-18 kernel: Lustre: DEBUG MARKER: min OST has 8917752kB available, using 24580944kB file size
May 17 13:08:09 client-18 xinetd[3390]: EXIT: shell status=0 pid=26562 duration=0(sec)
May 17 13:59:15 client-18 xinetd[3390]: START: shell pid=26764 from=192.168.4.5
May 17 13:59:15 client-18 rshd[26765]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 13:59:15 client-18 xinetd[3390]: EXIT: shell status=0 pid=26764 duration=0(sec)
May 17 13:59:16 client-18 xinetd[3390]: START: shell pid=26800 from=192.168.4.5
May 17 13:59:16 client-18 rshd[26801]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 13:59:16 client-18 xinetd[3390]: EXIT: shell status=0 pid=26800 duration=0(sec)
May 17 13:59:16 client-18 xinetd[3390]: START: shell pid=26837 from=192.168.4.5
May 17 13:59:16 client-18 rshd[26838]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity-benchmark test fsx: fsx ==================================================================== 13:59:16 \(1305665956\)");echo XXRETCODE:$?'
May 17 13:59:16 client-18 kernel: Lustre: DEBUG MARKER: == sanity-benchmark test fsx: fsx ==================================================================== 13:59:16 (1305665956)
May 17 13:59:16 client-18 xinetd[3390]: EXIT: shell status=0 pid=26837 duration=0(sec)
May 17 14:02:54 client-18 xinetd[3390]: START: shell pid=26890 from=192.168.4.5
May 17 14:02:54 client-18 rshd[26891]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:02:54 client-18 xinetd[3390]: EXIT: shell status=0 pid=26890 duration=0(sec)
May 17 14:02:54 client-18 xinetd[3390]: START: shell pid=26926 from=192.168.4.5
May 17 14:02:54 client-18 rshd[26927]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:02:55 client-18 xinetd[3390]: EXIT: shell status=0 pid=26926 duration=1(sec)
May 17 14:02:55 client-18 xinetd[3390]: START: shell pid=26963 from=192.168.4.5
May 17 14:02:55 client-18 rshd[26964]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity-benchmark test pios_ssf: pios shared single file =========================================== 14:02:55 \(1305666175\)");echo XXRETCODE:$?'
May 17 14:02:55 client-18 kernel: Lustre: DEBUG MARKER: == sanity-benchmark test pios_ssf: pios shared single file =========================================== 14:02:55 (1305666175)
May 17 14:02:55 client-18 xinetd[3390]: EXIT: shell status=0 pid=26963 duration=0(sec)
May 17 14:02:55 client-18 xinetd[3390]: START: shell pid=26998 from=192.168.4.5
May 17 14:02:55 client-18 rshd[26999]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark  SKIP: sanity-benchmark test_pios_ssf \/usr\/lib64\/lustre\/tests\/sanity-benchmark.sh : pios not found PIOSBIN=");echo XXRETCODE:$?'
May 17 14:02:55 client-18 kernel: Lustre: DEBUG MARKER: SKIP: sanity-benchmark test_pios_ssf /usr/lib64/lustre/tests/sanity-benchmark.sh : pios not found PIOSBIN=
May 17 14:02:55 client-18 xinetd[3390]: EXIT: shell status=0 pid=26998 duration=0(sec)
May 17 14:02:55 client-18 xinetd[3390]: START: shell pid=27033 from=192.168.4.5
May 17 14:02:55 client-18 rshd[27034]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:02:55 client-18 xinetd[3390]: EXIT: shell status=0 pid=27033 duration=0(sec)
May 17 14:02:55 client-18 xinetd[3390]: START: shell pid=27069 from=192.168.4.5
May 17 14:02:55 client-18 rshd[27070]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:02:55 client-18 xinetd[3390]: EXIT: shell status=0 pid=27069 duration=0(sec)
May 17 14:02:55 client-18 xinetd[3390]: START: shell pid=27106 from=192.168.4.5
May 17 14:02:55 client-18 rshd[27107]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity-benchmark test pios_fpp: pios file per process ============================================= 14:02:55 \(1305666175\)");echo XXRETCODE:$?'
May 17 14:02:55 client-18 kernel: Lustre: DEBUG MARKER: == sanity-benchmark test pios_fpp: pios file per process ============================================= 14:02:55 (1305666175)
May 17 14:02:55 client-18 xinetd[3390]: EXIT: shell status=0 pid=27106 duration=0(sec)
May 17 14:02:55 client-18 xinetd[3390]: START: shell pid=27141 from=192.168.4.5
May 17 14:02:55 client-18 rshd[27142]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark  SKIP: sanity-benchmark test_pios_fpp pios not found PIOSBIN=");echo XXRETCODE:$?'
May 17 14:02:55 client-18 kernel: Lustre: DEBUG MARKER: SKIP: sanity-benchmark test_pios_fpp pios not found PIOSBIN=
May 17 14:02:55 client-18 xinetd[3390]: EXIT: shell status=0 pid=27141 duration=0(sec)
May 17 14:02:56 client-18 xinetd[3390]: START: shell pid=27176 from=192.168.4.5
May 17 14:02:56 client-18 rshd[27177]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:02:56 client-18 xinetd[3390]: EXIT: shell status=0 pid=27176 duration=0(sec)
May 17 14:02:56 client-18 xinetd[3390]: START: shell pid=27212 from=192.168.4.5
May 17 14:02:56 client-18 rshd[27213]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:02:56 client-18 xinetd[3390]: EXIT: shell status=0 pid=27212 duration=0(sec)
May 17 14:02:56 client-18 xinetd[3390]: START: shell pid=27249 from=192.168.4.5
May 17 14:02:56 client-18 rshd[27250]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity-benchmark sanity-benchmark.sh test complete, duration 5474 sec ============================= 14:02:56 \(1305666176\)");echo XXRETCODE:$?'
May 17 14:02:56 client-18 kernel: Lustre: DEBUG MARKER: == sanity-benchmark sanity-benchmark.sh test complete, duration 5474 sec ============================= 14:02:56 (1305666176)
May 17 14:02:56 client-18 xinetd[3390]: EXIT: shell status=0 pid=27249 duration=0(sec)
May 17 14:02:56 client-18 xinetd[3390]: START: shell pid=27284 from=192.168.4.5
May 17 14:02:56 client-18 rshd[27285]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark -----============= acceptance-small: sanityn ============----- Tue May 17 14:02:56 PDT 2011");echo XXRETCODE:$?'
May 17 14:02:56 client-18 kernel: Lustre: DEBUG MARKER: -----============= acceptance-small: sanityn ============----- Tue May 17 14:02:56 PDT 2011
May 17 14:02:56 client-18 xinetd[3390]: EXIT: shell status=0 pid=27284 duration=0(sec)
May 17 14:02:56 client-18 xinetd[3390]: START: shell pid=27319 from=192.168.4.5
May 17 14:02:56 client-18 rshd[27320]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c " running=\$(mount | grep -c /mnt/lustre2' '); rc=0; if [ \$running -eq 0 ] ; then     mkdir -p /mnt/lustre2;     mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre2;     rc=\$?; fi; exit \$rc");echo XXRETCODE:$?'
May 17 14:02:57 client-18 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'.
May 17 14:02:57 client-18 kernel: Lustre: Increasing default stripe size to min 1048576
May 17 14:02:57 client-18 kernel: Lustre: 27359:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff810304e9a400->192.168.4.128@o2ib netid 50000: select flavor null
May 17 14:02:57 client-18 kernel: Lustre: 27359:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 7 previous similar messages
May 17 14:02:57 client-18 kernel: Lustre: Client lustre-client has started
May 17 14:02:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=27319 duration=1(sec)
May 17 14:02:57 client-18 xinetd[3390]: START: shell pid=27362 from=192.168.4.5
May 17 14:02:57 client-18 rshd[27363]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "mount | grep -w /mnt/lustre2");echo XXRETCODE:$?'
May 17 14:02:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=27362 duration=0(sec)
May 17 14:02:57 client-18 xinetd[3390]: START: shell pid=27399 from=192.168.4.5
May 17 14:02:57 client-18 rshd[27400]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param debug=\"0x33f0406\";         lctl set_param subsystem_debug=\"0xffb7e3ff\";         lctl set_param debug_mb=32;");echo XXRETCODE:$?'
May 17 14:02:57 client-18 kernel: Lustre: 27434:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release.
May 17 14:02:57 client-18 kernel: Lustre: 27434:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message
May 17 14:02:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=27399 duration=0(sec)
May 17 14:02:57 client-18 xinetd[3390]: START: shell pid=27436 from=192.168.4.5
May 17 14:02:57 client-18 rshd[27437]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/tests:/usr/lib64/openmpi/1.4-gcc/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/usr/lib64/lustre/utils:/usr/lib64/lustre/utils/gss:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey::/usr/lib64/lustre/tests/mpi NAME=sarah sh rpc.sh check_config_client /mnt/lustre ");echo XXRETCODE:$?'
May 17 14:02:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=27436 duration=0(sec)
May 17 14:02:58 client-18 xinetd[3390]: START: shell pid=27530 from=192.168.4.5
May 17 14:02:58 client-18 rshd[27531]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); mpts=\$(mount | grep -w -c /mnt/lustre); if [ \$running -ne \$mpts ]; then     echo \$(hostname) env are INSANE!;     exit 1; fi");echo XXRETCODE:$?'
May 17 14:02:58 client-18 xinetd[3390]: EXIT: shell status=0 pid=27530 duration=0(sec)
May 17 14:02:58 client-18 xinetd[3390]: START: shell pid=27570 from=192.168.4.5
May 17 14:02:58 client-18 rshd[27571]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); mpts=\$(mount | grep -w -c /mnt/lustre2); if [ \$running -ne \$mpts ]; then     echo \$(hostname) env are INSANE!;     exit 1; fi");echo XXRETCODE:$?'
May 17 14:02:58 client-18 xinetd[3390]: EXIT: shell status=0 pid=27570 duration=0(sec)
May 17 14:03:00 client-18 xinetd[3390]: START: shell pid=27610 from=192.168.4.5
May 17 14:03:00 client-18 rshd[27611]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark Using TIMEOUT=20");echo XXRETCODE:$?'
May 17 14:03:00 client-18 kernel: Lustre: DEBUG MARKER: Using TIMEOUT=20
May 17 14:03:00 client-18 xinetd[3390]: EXIT: shell status=0 pid=27610 duration=0(sec)
May 17 14:03:01 client-18 xinetd[3390]: START: shell pid=27645 from=192.168.4.5
May 17 14:03:01 client-18 rshd[27646]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param debug=\"0x33f0406\";             lctl set_param subsystem_debug=\"0xffb7e3ff\";             lctl set_param debug_mb=32;             sync");echo XXRETCODE:$?'
May 17 14:03:01 client-18 kernel: Lustre: 27680:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release.
May 17 14:03:01 client-18 kernel: Lustre: 27680:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message
May 17 14:03:01 client-18 xinetd[3390]: EXIT: shell status=0 pid=27645 duration=0(sec)
May 17 14:03:01 client-18 xinetd[3390]: START: shell pid=27683 from=192.168.4.5
May 17 14:03:01 client-18 rshd[27684]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "cat /proc/mounts");echo XXRETCODE:$?'
May 17 14:03:01 client-18 xinetd[3390]: EXIT: shell status=0 pid=27683 duration=0(sec)
May 17 14:03:02 client-18 xinetd[3390]: START: shell pid=27718 from=192.168.4.5
May 17 14:03:02 client-18 rshd[27719]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl get_param -n osc.*OST*-osc-[^M][^D][^T]*.srpc_info");echo XXRETCODE:$?'
May 17 14:03:02 client-18 xinetd[3390]: EXIT: shell status=0 pid=27718 duration=0(sec)
May 17 14:03:02 client-18 xinetd[3390]: START: shell pid=27753 from=192.168.4.5
May 17 14:03:02 client-18 rshd[27754]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl get_param -n mdc.*-MDT*-mdc-*.srpc_info");echo XXRETCODE:$?'
May 17 14:03:02 client-18 xinetd[3390]: EXIT: shell status=0 pid=27753 duration=0(sec)
May 17 14:03:02 client-18 xinetd[3390]: START: shell pid=27788 from=192.168.4.5
May 17 14:03:02 client-18 rshd[27789]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark excepting tests: 22 14b 19 22 28 29 35");echo XXRETCODE:$?'
May 17 14:03:02 client-18 kernel: Lustre: DEBUG MARKER: excepting tests: 22 14b 19 22 28 29 35
May 17 14:03:02 client-18 xinetd[3390]: EXIT: shell status=0 pid=27788 duration=0(sec)
May 17 14:03:02 client-18 xinetd[3390]: START: shell pid=27823 from=192.168.4.5
May 17 14:03:02 client-18 rshd[27824]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 1a: check create on 2 mtpt\'s ========================================================= 14:03:02 \(1305666182\)");echo XXRETCODE:$?'
May 17 14:03:02 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 1a: check create on 2 mtpt's ========================================================= 14:03:02 (1305666182)
May 17 14:03:02 client-18 xinetd[3390]: EXIT: shell status=0 pid=27823 duration=0(sec)
May 17 14:03:02 client-18 xinetd[3390]: START: shell pid=27858 from=192.168.4.5
May 17 14:03:02 client-18 rshd[27859]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:03:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=27858 duration=1(sec)
May 17 14:03:03 client-18 xinetd[3390]: START: shell pid=27894 from=192.168.4.5
May 17 14:03:03 client-18 rshd[27895]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:03:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=27894 duration=0(sec)
May 17 14:03:03 client-18 xinetd[3390]: START: shell pid=27931 from=192.168.4.5
May 17 14:03:03 client-18 rshd[27932]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 1b: check attribute updates on 2 mtpt\'s ============================================== 14:03:03 \(1305666183\)");echo XXRETCODE:$?'
May 17 14:03:03 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 1b: check attribute updates on 2 mtpt's ============================================== 14:03:03 (1305666183)
May 17 14:03:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=27931 duration=0(sec)
May 17 14:03:03 client-18 xinetd[3390]: START: shell pid=27966 from=192.168.4.5
May 17 14:03:03 client-18 rshd[27967]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:03:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=27966 duration=0(sec)
May 17 14:03:03 client-18 xinetd[3390]: START: shell pid=28002 from=192.168.4.5
May 17 14:03:03 client-18 rshd[28003]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:03:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=28002 duration=0(sec)
May 17 14:03:03 client-18 xinetd[3390]: START: shell pid=28039 from=192.168.4.5
May 17 14:03:03 client-18 rshd[28040]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 1c: check after remount attribute updates on 2 mtpt\'s ================================ 14:03:03 \(1305666183\)");echo XXRETCODE:$?'
May 17 14:03:03 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 1c: check after remount attribute updates on 2 mtpt's ================================ 14:03:03 (1305666183)
May 17 14:03:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=28039 duration=0(sec)
May 17 14:03:03 client-18 xinetd[3390]: START: shell pid=28074 from=192.168.4.5
May 17 14:03:03 client-18 rshd[28075]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:03:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=28074 duration=1(sec)
May 17 14:03:04 client-18 xinetd[3390]: START: shell pid=28110 from=192.168.4.5
May 17 14:03:04 client-18 rshd[28111]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:03:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=28110 duration=0(sec)
May 17 14:03:04 client-18 xinetd[3390]: START: shell pid=28147 from=192.168.4.5
May 17 14:03:04 client-18 rshd[28148]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 1d: unlink on one mountpoint removes file on other =================================== 14:03:04 \(1305666184\)");echo XXRETCODE:$?'
May 17 14:03:04 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 1d: unlink on one mountpoint removes file on other =================================== 14:03:04 (1305666184)
May 17 14:03:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=28147 duration=0(sec)
May 17 14:03:04 client-18 xinetd[3390]: START: shell pid=28182 from=192.168.4.5
May 17 14:03:04 client-18 rshd[28183]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:03:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=28182 duration=0(sec)
May 17 14:03:04 client-18 xinetd[3390]: START: shell pid=28218 from=192.168.4.5
May 17 14:03:04 client-18 rshd[28219]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:03:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=28218 duration=0(sec)
May 17 14:03:04 client-18 xinetd[3390]: START: shell pid=28255 from=192.168.4.5
May 17 14:03:04 client-18 rshd[28256]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 2a: check cached attribute updates on 2 mtpt\'s ======================================= 14:03:04 \(1305666184\)");echo XXRETCODE:$?'
May 17 14:03:04 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 2a: check cached attribute updates on 2 mtpt's ======================================= 14:03:04 (1305666184)
May 17 14:03:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=28255 duration=0(sec)
May 17 14:03:04 client-18 xinetd[3390]: START: shell pid=28290 from=192.168.4.5
May 17 14:03:04 client-18 rshd[28291]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:03:05 client-18 xinetd[3390]: EXIT: shell status=0 pid=28290 duration=1(sec)
May 17 14:03:05 client-18 xinetd[3390]: START: shell pid=28326 from=192.168.4.5
May 17 14:03:05 client-18 rshd[28327]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:03:05 client-18 xinetd[3390]: EXIT: shell status=0 pid=28326 duration=0(sec)
May 17 14:03:05 client-18 xinetd[3390]: START: shell pid=28363 from=192.168.4.5
May 17 14:03:05 client-18 rshd[28364]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 2b: check cached attribute updates on 2 mtpt\'s ======================================= 14:03:05 \(1305666185\)");echo XXRETCODE:$?'
May 17 14:03:05 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 2b: check cached attribute updates on 2 mtpt's ======================================= 14:03:05 (1305666185)
May 17 14:03:05 client-18 xinetd[3390]: EXIT: shell status=0 pid=28363 duration=0(sec)
May 17 14:03:05 client-18 xinetd[3390]: START: shell pid=28398 from=192.168.4.5
May 17 14:03:05 client-18 rshd[28399]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:03:05 client-18 xinetd[3390]: EXIT: shell status=0 pid=28398 duration=0(sec)
May 17 14:03:05 client-18 xinetd[3390]: START: shell pid=28434 from=192.168.4.5
May 17 14:03:05 client-18 rshd[28435]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:03:05 client-18 xinetd[3390]: EXIT: shell status=0 pid=28434 duration=0(sec)
May 17 14:03:05 client-18 xinetd[3390]: START: shell pid=28471 from=192.168.4.5
May 17 14:03:05 client-18 rshd[28472]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 2c: check cached attribute updates on 2 mtpt\'s root ================================== 14:03:05 \(1305666185\)");echo XXRETCODE:$?'
May 17 14:03:05 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 2c: check cached attribute updates on 2 mtpt's root ================================== 14:03:05 (1305666185)
May 17 14:03:05 client-18 xinetd[3390]: EXIT: shell status=0 pid=28471 duration=0(sec)
May 17 14:03:05 client-18 xinetd[3390]: START: shell pid=28506 from=192.168.4.5
May 17 14:03:05 client-18 rshd[28507]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:03:06 client-18 xinetd[3390]: EXIT: shell status=0 pid=28506 duration=1(sec)
May 17 14:03:06 client-18 xinetd[3390]: START: shell pid=28542 from=192.168.4.5
May 17 14:03:06 client-18 rshd[28543]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:03:06 client-18 xinetd[3390]: EXIT: shell status=0 pid=28542 duration=0(sec)
May 17 14:03:06 client-18 xinetd[3390]: START: shell pid=28579 from=192.168.4.5
May 17 14:03:06 client-18 rshd[28580]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 2d: check cached attribute updates on 2 mtpt\'s root ================================== 14:03:06 \(1305666186\)");echo XXRETCODE:$?'
May 17 14:03:06 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 2d: check cached attribute updates on 2 mtpt's root ================================== 14:03:06 (1305666186)
May 17 14:03:06 client-18 xinetd[3390]: EXIT: shell status=0 pid=28579 duration=0(sec)
May 17 14:03:06 client-18 xinetd[3390]: START: shell pid=28614 from=192.168.4.5
May 17 14:03:06 client-18 rshd[28615]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:03:06 client-18 xinetd[3390]: EXIT: shell status=0 pid=28614 duration=0(sec)
May 17 14:03:06 client-18 xinetd[3390]: START: shell pid=28650 from=192.168.4.5
May 17 14:03:06 client-18 rshd[28651]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:03:06 client-18 xinetd[3390]: EXIT: shell status=0 pid=28650 duration=0(sec)
May 17 14:03:06 client-18 xinetd[3390]: START: shell pid=28687 from=192.168.4.5
May 17 14:03:06 client-18 rshd[28688]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 2e: check chmod on root is propagated to others ====================================== 14:03:06 \(1305666186\)");echo XXRETCODE:$?'
May 17 14:03:06 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 2e: check chmod on root is propagated to others ====================================== 14:03:06 (1305666186)
May 17 14:03:06 client-18 xinetd[3390]: EXIT: shell status=0 pid=28687 duration=0(sec)
May 17 14:03:06 client-18 xinetd[3390]: START: shell pid=28722 from=192.168.4.5
May 17 14:03:06 client-18 rshd[28723]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:03:07 client-18 xinetd[3390]: EXIT: shell status=0 pid=28722 duration=1(sec)
May 17 14:03:07 client-18 xinetd[3390]: START: shell pid=28758 from=192.168.4.5
May 17 14:03:07 client-18 rshd[28759]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:03:07 client-18 xinetd[3390]: EXIT: shell status=0 pid=28758 duration=0(sec)
May 17 14:03:07 client-18 xinetd[3390]: START: shell pid=28795 from=192.168.4.5
May 17 14:03:07 client-18 rshd[28796]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 3: symlink on one mtpt, readlink on another ========================================== 14:03:07 \(1305666187\)");echo XXRETCODE:$?'
May 17 14:03:07 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 3: symlink on one mtpt, readlink on another ========================================== 14:03:07 (1305666187)
May 17 14:03:07 client-18 xinetd[3390]: EXIT: shell status=0 pid=28795 duration=0(sec)
May 17 14:03:07 client-18 xinetd[3390]: START: shell pid=28830 from=192.168.4.5
May 17 14:03:07 client-18 rshd[28831]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:03:07 client-18 xinetd[3390]: EXIT: shell status=0 pid=28830 duration=0(sec)
May 17 14:03:07 client-18 xinetd[3390]: START: shell pid=28866 from=192.168.4.5
May 17 14:03:07 client-18 rshd[28867]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:03:07 client-18 xinetd[3390]: EXIT: shell status=0 pid=28866 duration=0(sec)
May 17 14:03:07 client-18 xinetd[3390]: START: shell pid=28903 from=192.168.4.5
May 17 14:03:07 client-18 rshd[28904]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 4: fstat validation on multiple mount points ========================================= 14:03:07 \(1305666187\)");echo XXRETCODE:$?'
May 17 14:03:07 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 4: fstat validation on multiple mount points ========================================= 14:03:07 (1305666187)
May 17 14:03:07 client-18 xinetd[3390]: EXIT: shell status=0 pid=28903 duration=0(sec)
May 17 14:03:08 client-18 xinetd[3390]: START: shell pid=28938 from=192.168.4.5
May 17 14:03:08 client-18 rshd[28939]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:03:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=28938 duration=0(sec)
May 17 14:03:09 client-18 xinetd[3390]: START: shell pid=28974 from=192.168.4.5
May 17 14:03:09 client-18 rshd[28975]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:03:09 client-18 xinetd[3390]: EXIT: shell status=0 pid=28974 duration=0(sec)
May 17 14:03:09 client-18 xinetd[3390]: START: shell pid=29011 from=192.168.4.5
May 17 14:03:09 client-18 rshd[29012]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 5: create a file on one mount, truncate it on the other ============================== 14:03:09 \(1305666189\)");echo XXRETCODE:$?'
May 17 14:03:09 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 5: create a file on one mount, truncate it on the other ============================== 14:03:09 (1305666189)
May 17 14:03:09 client-18 xinetd[3390]: EXIT: shell status=0 pid=29011 duration=0(sec)
May 17 14:03:09 client-18 xinetd[3390]: START: shell pid=29046 from=192.168.4.5
May 17 14:03:09 client-18 rshd[29047]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:03:09 client-18 xinetd[3390]: EXIT: shell status=0 pid=29046 duration=0(sec)
May 17 14:03:09 client-18 xinetd[3390]: START: shell pid=29082 from=192.168.4.5
May 17 14:03:09 client-18 rshd[29083]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:03:09 client-18 xinetd[3390]: EXIT: shell status=0 pid=29082 duration=0(sec)
May 17 14:03:09 client-18 xinetd[3390]: START: shell pid=29119 from=192.168.4.5
May 17 14:03:09 client-18 rshd[29120]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 6: remove of open file on other node ================================================= 14:03:09 \(1305666189\)");echo XXRETCODE:$?'
May 17 14:03:09 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 6: remove of open file on other node ================================================= 14:03:09 (1305666189)
May 17 14:03:09 client-18 xinetd[3390]: EXIT: shell status=0 pid=29119 duration=0(sec)
May 17 14:03:09 client-18 xinetd[3390]: START: shell pid=29154 from=192.168.4.5
May 17 14:03:09 client-18 rshd[29155]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:03:10 client-18 xinetd[3390]: EXIT: shell status=0 pid=29154 duration=1(sec)
May 17 14:03:10 client-18 xinetd[3390]: START: shell pid=29190 from=192.168.4.5
May 17 14:03:10 client-18 rshd[29191]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:03:10 client-18 xinetd[3390]: EXIT: shell status=0 pid=29190 duration=0(sec)
May 17 14:03:10 client-18 xinetd[3390]: START: shell pid=29227 from=192.168.4.5
May 17 14:03:10 client-18 rshd[29228]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 7: remove of open directory on other node ============================================ 14:03:10 \(1305666190\)");echo XXRETCODE:$?'
May 17 14:03:10 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 7: remove of open directory on other node ============================================ 14:03:10 (1305666190)
May 17 14:03:10 client-18 xinetd[3390]: EXIT: shell status=0 pid=29227 duration=0(sec)
May 17 14:03:10 client-18 xinetd[3390]: START: shell pid=29262 from=192.168.4.5
May 17 14:03:10 client-18 rshd[29263]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:03:10 client-18 xinetd[3390]: EXIT: shell status=0 pid=29262 duration=0(sec)
May 17 14:03:10 client-18 xinetd[3390]: START: shell pid=29298 from=192.168.4.5
May 17 14:03:10 client-18 rshd[29299]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:03:10 client-18 xinetd[3390]: EXIT: shell status=0 pid=29298 duration=0(sec)
May 17 14:03:10 client-18 xinetd[3390]: START: shell pid=29335 from=192.168.4.5
May 17 14:03:10 client-18 rshd[29336]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 8: remove of open special file on other node ========================================= 14:03:10 \(1305666190\)");echo XXRETCODE:$?'
May 17 14:03:10 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 8: remove of open special file on other node ========================================= 14:03:10 (1305666190)
May 17 14:03:10 client-18 xinetd[3390]: EXIT: shell status=0 pid=29335 duration=0(sec)
May 17 14:03:11 client-18 xinetd[3390]: START: shell pid=29370 from=192.168.4.5
May 17 14:03:11 client-18 rshd[29371]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:03:11 client-18 xinetd[3390]: EXIT: shell status=0 pid=29370 duration=0(sec)
May 17 14:03:11 client-18 xinetd[3390]: START: shell pid=29406 from=192.168.4.5
May 17 14:03:11 client-18 rshd[29407]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:03:11 client-18 xinetd[3390]: EXIT: shell status=0 pid=29406 duration=0(sec)
May 17 14:03:11 client-18 xinetd[3390]: START: shell pid=29443 from=192.168.4.5
May 17 14:03:11 client-18 rshd[29444]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 9: append of file with sub-page size on multiple mounts ============================== 14:03:11 \(1305666191\)");echo XXRETCODE:$?'
May 17 14:03:11 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 9: append of file with sub-page size on multiple mounts ============================== 14:03:11 (1305666191)
May 17 14:03:11 client-18 xinetd[3390]: EXIT: shell status=0 pid=29443 duration=0(sec)
May 17 14:03:11 client-18 xinetd[3390]: START: shell pid=29478 from=192.168.4.5
May 17 14:03:11 client-18 rshd[29479]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:03:11 client-18 xinetd[3390]: EXIT: shell status=0 pid=29478 duration=0(sec)
May 17 14:03:11 client-18 xinetd[3390]: START: shell pid=29514 from=192.168.4.5
May 17 14:03:11 client-18 rshd[29515]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:03:12 client-18 xinetd[3390]: EXIT: shell status=0 pid=29514 duration=1(sec)
May 17 14:03:12 client-18 xinetd[3390]: START: shell pid=29551 from=192.168.4.5
May 17 14:03:12 client-18 rshd[29552]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 10a: write of file with sub-page size on multiple mounts ============================= 14:03:12 \(1305666192\)");echo XXRETCODE:$?'
May 17 14:03:12 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 10a: write of file with sub-page size on multiple mounts ============================= 14:03:12 (1305666192)
May 17 14:03:12 client-18 xinetd[3390]: EXIT: shell status=0 pid=29551 duration=0(sec)
May 17 14:03:12 client-18 xinetd[3390]: START: shell pid=29586 from=192.168.4.5
May 17 14:03:12 client-18 rshd[29587]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:03:12 client-18 xinetd[3390]: EXIT: shell status=0 pid=29586 duration=0(sec)
May 17 14:03:12 client-18 xinetd[3390]: START: shell pid=29622 from=192.168.4.5
May 17 14:03:12 client-18 rshd[29623]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:03:12 client-18 xinetd[3390]: EXIT: shell status=0 pid=29622 duration=0(sec)
May 17 14:03:12 client-18 xinetd[3390]: START: shell pid=29659 from=192.168.4.5
May 17 14:03:12 client-18 rshd[29660]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 10b: write of file with sub-page size on multiple mounts ============================= 14:03:12 \(1305666192\)");echo XXRETCODE:$?'
May 17 14:03:12 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 10b: write of file with sub-page size on multiple mounts ============================= 14:03:12 (1305666192)
May 17 14:03:12 client-18 xinetd[3390]: EXIT: shell status=0 pid=29659 duration=0(sec)
May 17 14:03:13 client-18 xinetd[3390]: START: shell pid=29694 from=192.168.4.5
May 17 14:03:13 client-18 rshd[29695]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:03:13 client-18 xinetd[3390]: EXIT: shell status=0 pid=29694 duration=0(sec)
May 17 14:03:13 client-18 xinetd[3390]: START: shell pid=29730 from=192.168.4.5
May 17 14:03:13 client-18 rshd[29731]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:03:13 client-18 xinetd[3390]: EXIT: shell status=0 pid=29730 duration=0(sec)
May 17 14:03:13 client-18 xinetd[3390]: START: shell pid=29767 from=192.168.4.5
May 17 14:03:13 client-18 rshd[29768]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 11: execution of file opened for write should return error =========================== 14:03:13 \(1305666193\)");echo XXRETCODE:$?'
May 17 14:03:13 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 11: execution of file opened for write should return error =========================== 14:03:13 (1305666193)
May 17 14:03:13 client-18 xinetd[3390]: EXIT: shell status=0 pid=29767 duration=0(sec)
May 17 14:03:13 client-18 xinetd[3390]: START: shell pid=29802 from=192.168.4.5
May 17 14:03:13 client-18 rshd[29803]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:03:13 client-18 xinetd[3390]: EXIT: shell status=0 pid=29802 duration=0(sec)
May 17 14:03:13 client-18 xinetd[3390]: START: shell pid=29838 from=192.168.4.5
May 17 14:03:13 client-18 rshd[29839]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:03:13 client-18 xinetd[3390]: EXIT: shell status=0 pid=29838 duration=0(sec)
May 17 14:03:14 client-18 xinetd[3390]: START: shell pid=29875 from=192.168.4.5
May 17 14:03:14 client-18 rshd[29876]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 12: test lock ordering \(link, stat, unlink\) ========================================== 14:03:13 \(1305666193\)");echo XXRETCODE:$?'
May 17 14:03:14 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 12: test lock ordering (link, stat, unlink) ========================================== 14:03:13 (1305666193)
May 17 14:03:14 client-18 xinetd[3390]: EXIT: shell status=0 pid=29875 duration=0(sec)
May 17 14:05:36 client-18 xinetd[3390]: START: shell pid=29918 from=192.168.4.5
May 17 14:05:36 client-18 rshd[29919]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:05:36 client-18 xinetd[3390]: EXIT: shell status=0 pid=29918 duration=0(sec)
May 17 14:05:36 client-18 xinetd[3390]: START: shell pid=29954 from=192.168.4.5
May 17 14:05:36 client-18 rshd[29955]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:05:36 client-18 xinetd[3390]: EXIT: shell status=0 pid=29954 duration=0(sec)
May 17 14:05:36 client-18 xinetd[3390]: START: shell pid=29991 from=192.168.4.5
May 17 14:05:36 client-18 rshd[29992]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 13: test directory page revocation =================================================== 14:05:36 \(1305666336\)");echo XXRETCODE:$?'
May 17 14:05:36 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 13: test directory page revocation =================================================== 14:05:36 (1305666336)
May 17 14:05:36 client-18 xinetd[3390]: EXIT: shell status=0 pid=29991 duration=0(sec)
May 17 14:05:36 client-18 xinetd[3390]: START: shell pid=30026 from=192.168.4.5
May 17 14:05:36 client-18 rshd[30027]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:05:36 client-18 xinetd[3390]: EXIT: shell status=0 pid=30026 duration=0(sec)
May 17 14:05:36 client-18 xinetd[3390]: START: shell pid=30062 from=192.168.4.5
May 17 14:05:36 client-18 rshd[30063]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:05:36 client-18 xinetd[3390]: EXIT: shell status=0 pid=30062 duration=0(sec)
May 17 14:05:36 client-18 xinetd[3390]: START: shell pid=30099 from=192.168.4.5
May 17 14:05:36 client-18 rshd[30100]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 14: execution of file open for write returns -ETXTBSY ================================ 14:05:36 \(1305666336\)");echo XXRETCODE:$?'
May 17 14:05:37 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 14: execution of file open for write returns -ETXTBSY ================================ 14:05:36 (1305666336)
May 17 14:05:37 client-18 xinetd[3390]: EXIT: shell status=0 pid=30099 duration=1(sec)
May 17 14:05:37 client-18 xinetd[3390]: START: shell pid=30134 from=192.168.4.5
May 17 14:05:37 client-18 rshd[30135]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:05:37 client-18 xinetd[3390]: EXIT: shell status=0 pid=30134 duration=0(sec)
May 17 14:05:37 client-18 xinetd[3390]: START: shell pid=30170 from=192.168.4.5
May 17 14:05:37 client-18 rshd[30171]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:05:37 client-18 xinetd[3390]: EXIT: shell status=0 pid=30170 duration=0(sec)
May 17 14:05:37 client-18 xinetd[3390]: START: shell pid=30207 from=192.168.4.5
May 17 14:05:37 client-18 rshd[30208]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 14a: open\(RDWR\) of executing file returns -ETXTBSY =================================== 14:05:37 \(1305666337\)");echo XXRETCODE:$?'
May 17 14:05:37 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 14a: open(RDWR) of executing file returns -ETXTBSY =================================== 14:05:37 (1305666337)
May 17 14:05:37 client-18 xinetd[3390]: EXIT: shell status=0 pid=30207 duration=0(sec)
May 17 14:05:37 client-18 xinetd[3390]: START: shell pid=30242 from=192.168.4.5
May 17 14:05:37 client-18 rshd[30243]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:05:37 client-18 xinetd[3390]: EXIT: shell status=0 pid=30242 duration=0(sec)
May 17 14:05:37 client-18 xinetd[3390]: START: shell pid=30278 from=192.168.4.5
May 17 14:05:37 client-18 rshd[30279]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:05:37 client-18 xinetd[3390]: EXIT: shell status=0 pid=30278 duration=0(sec)
May 17 14:05:37 client-18 xinetd[3390]: START: shell pid=30315 from=192.168.4.5
May 17 14:05:38 client-18 rshd[30316]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark  SKIP: sanityn test_14b skipping ALWAYS excluded test 14b");echo XXRETCODE:$?'
May 17 14:05:38 client-18 kernel: Lustre: DEBUG MARKER: SKIP: sanityn test_14b skipping ALWAYS excluded test 14b
May 17 14:05:38 client-18 xinetd[3390]: EXIT: shell status=0 pid=30315 duration=1(sec)
May 17 14:05:38 client-18 xinetd[3390]: START: shell pid=30350 from=192.168.4.5
May 17 14:05:38 client-18 rshd[30351]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 14c: open\(O_TRUNC\) of executing file return -ETXTBSY ================================= 14:05:38 \(1305666338\)");echo XXRETCODE:$?'
May 17 14:05:38 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ================================= 14:05:38 (1305666338)
May 17 14:05:38 client-18 xinetd[3390]: EXIT: shell status=0 pid=30350 duration=0(sec)
May 17 14:05:38 client-18 xinetd[3390]: START: shell pid=30385 from=192.168.4.5
May 17 14:05:38 client-18 rshd[30386]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:05:38 client-18 xinetd[3390]: EXIT: shell status=0 pid=30385 duration=0(sec)
May 17 14:05:38 client-18 xinetd[3390]: START: shell pid=30421 from=192.168.4.5
May 17 14:05:38 client-18 rshd[30422]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:05:38 client-18 xinetd[3390]: EXIT: shell status=0 pid=30421 duration=0(sec)
May 17 14:05:38 client-18 xinetd[3390]: START: shell pid=30458 from=192.168.4.5
May 17 14:05:38 client-18 rshd[30459]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 14d: chmod of executing file is still possible ======================================= 14:05:38 \(1305666338\)");echo XXRETCODE:$?'
May 17 14:05:38 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 14d: chmod of executing file is still possible ======================================= 14:05:38 (1305666338)
May 17 14:05:38 client-18 xinetd[3390]: EXIT: shell status=0 pid=30458 duration=0(sec)
May 17 14:05:38 client-18 xinetd[3390]: START: shell pid=30493 from=192.168.4.5
May 17 14:05:38 client-18 rshd[30494]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark chmod");echo XXRETCODE:$?'
May 17 14:05:38 client-18 kernel: Lustre: DEBUG MARKER: chmod
May 17 14:05:38 client-18 xinetd[3390]: EXIT: shell status=0 pid=30493 duration=0(sec)
May 17 14:05:39 client-18 xinetd[3390]: START: shell pid=30528 from=192.168.4.5
May 17 14:05:39 client-18 rshd[30529]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:05:39 client-18 xinetd[3390]: EXIT: shell status=0 pid=30528 duration=0(sec)
May 17 14:05:39 client-18 xinetd[3390]: START: shell pid=30564 from=192.168.4.5
May 17 14:05:39 client-18 rshd[30565]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:05:39 client-18 xinetd[3390]: EXIT: shell status=0 pid=30564 duration=0(sec)
May 17 14:05:39 client-18 xinetd[3390]: START: shell pid=30601 from=192.168.4.5
May 17 14:05:39 client-18 rshd[30602]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 15: test out-of-space with multiple writers ========================================== 14:05:39 \(1305666339\)");echo XXRETCODE:$?'
May 17 14:05:39 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 15: test out-of-space with multiple writers ========================================== 14:05:39 (1305666339)
May 17 14:05:39 client-18 xinetd[3390]: EXIT: shell status=0 pid=30601 duration=0(sec)
May 17 14:05:40 client-18 xinetd[3390]: START: shell pid=30636 from=192.168.4.5
May 17 14:05:40 client-18 rshd[30637]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark  SKIP: sanityn test_15 oos2.sh: 53532968kB free gt MAXFREE 2400000kB, increase 2400000 \(or reduce test fs size\) to proceed");echo XXRETCODE:$?'
May 17 14:05:40 client-18 kernel: Lustre: DEBUG MARKER: SKIP: sanityn test_15 oos2.sh: 53532968kB free gt MAXFREE 2400000kB, increase 2400000 (or reduce test fs size) to proceed
May 17 14:05:40 client-18 xinetd[3390]: EXIT: shell status=0 pid=30636 duration=0(sec)
May 17 14:05:40 client-18 xinetd[3390]: START: shell pid=30671 from=192.168.4.5
May 17 14:05:40 client-18 rshd[30672]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:05:40 client-18 xinetd[3390]: EXIT: shell status=0 pid=30671 duration=0(sec)
May 17 14:05:40 client-18 xinetd[3390]: START: shell pid=30707 from=192.168.4.5
May 17 14:05:40 client-18 rshd[30708]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:05:40 client-18 xinetd[3390]: EXIT: shell status=0 pid=30707 duration=0(sec)
May 17 14:05:41 client-18 xinetd[3390]: START: shell pid=30744 from=192.168.4.5
May 17 14:05:41 client-18 rshd[30745]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 16: 2500 iterations of dual-mount fsx ================================================ 14:05:41 \(1305666341\)");echo XXRETCODE:$?'
May 17 14:05:41 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 16: 2500 iterations of dual-mount fsx ================================================ 14:05:41 (1305666341)
May 17 14:05:41 client-18 xinetd[3390]: EXIT: shell status=0 pid=30744 duration=0(sec)
May 17 14:06:46 client-18 xinetd[3390]: START: shell pid=30783 from=192.168.4.5
May 17 14:06:46 client-18 rshd[30784]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:06:46 client-18 xinetd[3390]: EXIT: shell status=0 pid=30783 duration=0(sec)
May 17 14:06:46 client-18 xinetd[3390]: START: shell pid=30819 from=192.168.4.5
May 17 14:06:46 client-18 rshd[30820]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:06:46 client-18 xinetd[3390]: EXIT: shell status=0 pid=30819 duration=0(sec)
May 17 14:06:46 client-18 xinetd[3390]: START: shell pid=30856 from=192.168.4.5
May 17 14:06:46 client-18 rshd[30857]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 17: resource creation\/LVB creation race ============================================== 14:06:46 \(1305666406\)");echo XXRETCODE:$?'
May 17 14:06:47 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 17: resource creation/LVB creation race ============================================== 14:06:46 (1305666406)
May 17 14:06:47 client-18 xinetd[3390]: EXIT: shell status=0 pid=30856 duration=1(sec)
May 17 14:06:49 client-18 xinetd[3390]: START: shell pid=30891 from=192.168.4.5
May 17 14:06:49 client-18 rshd[30892]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:06:49 client-18 xinetd[3390]: EXIT: shell status=0 pid=30891 duration=0(sec)
May 17 14:06:49 client-18 xinetd[3390]: START: shell pid=30927 from=192.168.4.5
May 17 14:06:49 client-18 rshd[30928]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:06:49 client-18 xinetd[3390]: EXIT: shell status=0 pid=30927 duration=0(sec)
May 17 14:06:49 client-18 xinetd[3390]: START: shell pid=30964 from=192.168.4.5
May 17 14:06:49 client-18 rshd[30965]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 18: mmap sanity check ================================================================ 14:06:49 \(1305666409\)");echo XXRETCODE:$?'
May 17 14:06:49 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 18: mmap sanity check ================================================================ 14:06:49 (1305666409)
May 17 14:06:49 client-18 xinetd[3390]: EXIT: shell status=0 pid=30964 duration=0(sec)
May 17 14:06:55 client-18 xinetd[3390]: START: shell pid=30999 from=192.168.4.5
May 17 14:06:55 client-18 rshd[31000]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:06:55 client-18 xinetd[3390]: EXIT: shell status=0 pid=30999 duration=0(sec)
May 17 14:06:55 client-18 xinetd[3390]: START: shell pid=31035 from=192.168.4.5
May 17 14:06:55 client-18 rshd[31036]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:06:55 client-18 xinetd[3390]: EXIT: shell status=0 pid=31035 duration=0(sec)
May 17 14:06:55 client-18 xinetd[3390]: START: shell pid=31072 from=192.168.4.5
May 17 14:06:55 client-18 rshd[31073]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark  SKIP: sanityn test_19 skipping ALWAYS excluded test 19");echo XXRETCODE:$?'
May 17 14:06:55 client-18 kernel: Lustre: DEBUG MARKER: SKIP: sanityn test_19 skipping ALWAYS excluded test 19
May 17 14:06:55 client-18 xinetd[3390]: EXIT: shell status=0 pid=31072 duration=0(sec)
May 17 14:06:56 client-18 xinetd[3390]: START: shell pid=31107 from=192.168.4.5
May 17 14:06:56 client-18 rshd[31108]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 20: test extra readahead page left in cache ========================================== 14:06:55 \(1305666415\)");echo XXRETCODE:$?'
May 17 14:06:56 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 20: test extra readahead page left in cache ========================================== 14:06:55 (1305666415)
May 17 14:06:56 client-18 xinetd[3390]: EXIT: shell status=0 pid=31107 duration=0(sec)
May 17 14:06:58 client-18 xinetd[3390]: START: shell pid=31142 from=192.168.4.5
May 17 14:06:58 client-18 rshd[31143]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:06:58 client-18 xinetd[3390]: EXIT: shell status=0 pid=31142 duration=0(sec)
May 17 14:06:58 client-18 xinetd[3390]: START: shell pid=31178 from=192.168.4.5
May 17 14:06:58 client-18 rshd[31179]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:06:58 client-18 xinetd[3390]: EXIT: shell status=0 pid=31178 duration=0(sec)
May 17 14:06:58 client-18 xinetd[3390]: START: shell pid=31215 from=192.168.4.5
May 17 14:06:58 client-18 rshd[31216]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 21:  Try to remove mountpoint on another dir ========================================= 14:06:58 \(1305666418\)");echo XXRETCODE:$?'
May 17 14:06:58 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 21: Try to remove mountpoint on another dir ========================================= 14:06:58 (1305666418)
May 17 14:06:58 client-18 xinetd[3390]: EXIT: shell status=0 pid=31215 duration=0(sec)
May 17 14:06:58 client-18 xinetd[3390]: START: shell pid=31250 from=192.168.4.5
May 17 14:06:58 client-18 rshd[31251]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:06:58 client-18 xinetd[3390]: EXIT: shell status=0 pid=31250 duration=0(sec)
May 17 14:06:58 client-18 xinetd[3390]: START: shell pid=31286 from=192.168.4.5
May 17 14:06:58 client-18 rshd[31287]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:06:58 client-18 xinetd[3390]: EXIT: shell status=0 pid=31286 duration=0(sec)
May 17 14:06:59 client-18 xinetd[3390]: START: shell pid=31323 from=192.168.4.5
May 17 14:06:59 client-18 rshd[31324]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 23:  others should see updated atime while another read=============================== 14:06:58 \(1305666418\)");echo XXRETCODE:$?'
May 17 14:06:59 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 23: others should see updated atime while another read=============================== 14:06:58 (1305666418)
May 17 14:06:59 client-18 xinetd[3390]: EXIT: shell status=0 pid=31323 duration=0(sec)
May 17 14:08:00 client-18 xinetd[3390]: START: shell pid=31362 from=192.168.4.5
May 17 14:08:00 client-18 rshd[31363]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:08:00 client-18 xinetd[3390]: EXIT: shell status=0 pid=31362 duration=0(sec)
May 17 14:08:00 client-18 xinetd[3390]: START: shell pid=31398 from=192.168.4.5
May 17 14:08:00 client-18 rshd[31399]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:08:00 client-18 xinetd[3390]: EXIT: shell status=0 pid=31398 duration=0(sec)
May 17 14:08:00 client-18 xinetd[3390]: START: shell pid=31435 from=192.168.4.5
May 17 14:08:00 client-18 rshd[31436]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 24a: lfs df [-ih] [path] test ======================================================== 14:08:00 \(1305666480\)");echo XXRETCODE:$?'
May 17 14:08:00 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 24a: lfs df [-ih] [path] test ======================================================== 14:08:00 (1305666480)
May 17 14:08:00 client-18 xinetd[3390]: EXIT: shell status=0 pid=31435 duration=0(sec)
May 17 14:08:00 client-18 xinetd[3390]: START: shell pid=31470 from=192.168.4.5
May 17 14:08:00 client-18 rshd[31471]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:08:01 client-18 xinetd[3390]: EXIT: shell status=0 pid=31470 duration=1(sec)
May 17 14:08:01 client-18 xinetd[3390]: START: shell pid=31506 from=192.168.4.5
May 17 14:08:01 client-18 rshd[31507]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:08:01 client-18 xinetd[3390]: EXIT: shell status=0 pid=31506 duration=0(sec)
May 17 14:08:01 client-18 xinetd[3390]: START: shell pid=31543 from=192.168.4.5
May 17 14:08:01 client-18 rshd[31544]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 24b: lfs df should show both filesystems ============================================= 14:08:01 \(1305666481\)");echo XXRETCODE:$?'
May 17 14:08:01 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 24b: lfs df should show both filesystems ============================================= 14:08:01 (1305666481)
May 17 14:08:01 client-18 xinetd[3390]: EXIT: shell status=0 pid=31543 duration=0(sec)
May 17 14:08:01 client-18 xinetd[3390]: START: shell pid=31578 from=192.168.4.5
May 17 14:08:01 client-18 rshd[31579]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:08:01 client-18 xinetd[3390]: EXIT: shell status=0 pid=31578 duration=0(sec)
May 17 14:08:01 client-18 xinetd[3390]: START: shell pid=31614 from=192.168.4.5
May 17 14:08:01 client-18 rshd[31615]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:08:01 client-18 xinetd[3390]: EXIT: shell status=0 pid=31614 duration=0(sec)
May 17 14:08:01 client-18 xinetd[3390]: START: shell pid=31651 from=192.168.4.5
May 17 14:08:01 client-18 rshd[31652]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 25: change ACL on one mountpoint be seen on another ================================== 14:08:01 \(1305666481\)");echo XXRETCODE:$?'
May 17 14:08:01 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 25: change ACL on one mountpoint be seen on another ================================== 14:08:01 (1305666481)
May 17 14:08:01 client-18 xinetd[3390]: EXIT: shell status=0 pid=31651 duration=0(sec)
May 17 14:08:02 client-18 xinetd[3390]: START: shell pid=31686 from=192.168.4.5
May 17 14:08:02 client-18 rshd[31687]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:08:02 client-18 xinetd[3390]: EXIT: shell status=0 pid=31686 duration=0(sec)
May 17 14:08:02 client-18 xinetd[3390]: START: shell pid=31722 from=192.168.4.5
May 17 14:08:02 client-18 rshd[31723]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:08:02 client-18 xinetd[3390]: EXIT: shell status=0 pid=31722 duration=0(sec)
May 17 14:08:02 client-18 xinetd[3390]: START: shell pid=31759 from=192.168.4.5
May 17 14:08:02 client-18 rshd[31760]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 26a: allow mtime to get older ======================================================== 14:08:02 \(1305666482\)");echo XXRETCODE:$?'
May 17 14:08:02 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 26a: allow mtime to get older ======================================================== 14:08:02 (1305666482)
May 17 14:08:02 client-18 xinetd[3390]: EXIT: shell status=0 pid=31759 duration=0(sec)
May 17 14:08:03 client-18 xinetd[3390]: START: shell pid=31794 from=192.168.4.5
May 17 14:08:03 client-18 rshd[31795]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:08:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=31794 duration=0(sec)
May 17 14:08:03 client-18 xinetd[3390]: START: shell pid=31830 from=192.168.4.5
May 17 14:08:03 client-18 rshd[31831]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:08:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=31830 duration=0(sec)
May 17 14:08:03 client-18 xinetd[3390]: START: shell pid=31867 from=192.168.4.5
May 17 14:08:03 client-18 rshd[31868]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 26b: sync mtime between ost and mds ================================================== 14:08:03 \(1305666483\)");echo XXRETCODE:$?'
May 17 14:08:03 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 26b: sync mtime between ost and mds ================================================== 14:08:03 (1305666483)
May 17 14:08:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=31867 duration=1(sec)
May 17 14:08:06 client-18 xinetd[3390]: START: shell pid=31902 from=192.168.4.5
May 17 14:08:06 client-18 rshd[31903]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:08:06 client-18 xinetd[3390]: EXIT: shell status=0 pid=31902 duration=0(sec)
May 17 14:08:06 client-18 xinetd[3390]: START: shell pid=31938 from=192.168.4.5
May 17 14:08:06 client-18 rshd[31939]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:08:06 client-18 xinetd[3390]: EXIT: shell status=0 pid=31938 duration=0(sec)
May 17 14:08:06 client-18 xinetd[3390]: START: shell pid=31975 from=192.168.4.5
May 17 14:08:06 client-18 rshd[31976]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 27: align non-overlapping extent locks from request ================================== 14:08:06 \(1305666486\)");echo XXRETCODE:$?'
May 17 14:08:06 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 27: align non-overlapping extent locks from request ================================== 14:08:06 (1305666486)
May 17 14:08:06 client-18 xinetd[3390]: EXIT: shell status=0 pid=31975 duration=0(sec)
May 17 14:08:06 client-18 xinetd[3390]: START: shell pid=32010 from=192.168.4.5
May 17 14:08:06 client-18 rshd[32011]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark dd 1 started");echo XXRETCODE:$?'
May 17 14:08:06 client-18 kernel: Lustre: DEBUG MARKER: dd 1 started
May 17 14:08:06 client-18 xinetd[3390]: EXIT: shell status=0 pid=32010 duration=0(sec)
May 17 14:08:06 client-18 xinetd[3390]: START: shell pid=32045 from=192.168.4.5
May 17 14:08:06 client-18 rshd[32046]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark dd 2 started");echo XXRETCODE:$?'
May 17 14:08:06 client-18 kernel: Lustre: DEBUG MARKER: dd 2 started
May 17 14:08:06 client-18 xinetd[3390]: EXIT: shell status=0 pid=32045 duration=0(sec)
May 17 14:08:07 client-18 xinetd[3390]: START: shell pid=32080 from=192.168.4.5
May 17 14:08:07 client-18 rshd[32081]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark dd 3 finished");echo XXRETCODE:$?'
May 17 14:08:08 client-18 kernel: Lustre: DEBUG MARKER: dd 3 finished
May 17 14:08:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=32080 duration=1(sec)
May 17 14:08:08 client-18 xinetd[3390]: START: shell pid=32115 from=192.168.4.5
May 17 14:08:08 client-18 rshd[32116]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:08:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=32115 duration=0(sec)
May 17 14:08:08 client-18 xinetd[3390]: START: shell pid=32151 from=192.168.4.5
May 17 14:08:08 client-18 rshd[32152]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:08:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=32151 duration=0(sec)
May 17 14:08:08 client-18 xinetd[3390]: START: shell pid=32188 from=192.168.4.5
May 17 14:08:08 client-18 rshd[32189]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark  SKIP: sanityn test_28 skipping ALWAYS excluded test 28");echo XXRETCODE:$?'
May 17 14:08:08 client-18 kernel: Lustre: DEBUG MARKER: SKIP: sanityn test_28 skipping ALWAYS excluded test 28
May 17 14:08:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=32188 duration=0(sec)
May 17 14:08:08 client-18 xinetd[3390]: START: shell pid=32223 from=192.168.4.5
May 17 14:08:08 client-18 rshd[32224]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark  SKIP: sanityn test_29 skipping ALWAYS excluded test 29");echo XXRETCODE:$?'
May 17 14:08:08 client-18 kernel: Lustre: DEBUG MARKER: SKIP: sanityn test_29 skipping ALWAYS excluded test 29
May 17 14:08:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=32223 duration=0(sec)
May 17 14:08:08 client-18 xinetd[3390]: START: shell pid=32258 from=192.168.4.5
May 17 14:08:08 client-18 rshd[32259]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 30: recreate file race =============================================================== 14:08:08 \(1305666488\)");echo XXRETCODE:$?'
May 17 14:08:08 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 30: recreate file race =============================================================== 14:08:08 (1305666488)
May 17 14:08:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=32258 duration=0(sec)
May 17 14:08:10 client-18 xinetd[3390]: START: shell pid=32293 from=192.168.4.5
May 17 14:08:10 client-18 rshd[32294]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:08:11 client-18 xinetd[3390]: EXIT: shell status=0 pid=32293 duration=1(sec)
May 17 14:08:11 client-18 xinetd[3390]: START: shell pid=32329 from=192.168.4.5
May 17 14:08:11 client-18 rshd[32330]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:08:11 client-18 xinetd[3390]: EXIT: shell status=0 pid=32329 duration=0(sec)
May 17 14:08:11 client-18 xinetd[3390]: START: shell pid=32366 from=192.168.4.5
May 17 14:08:11 client-18 rshd[32367]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 31a: voluntary cancel \/ blocking ast race============================================= 14:08:11 \(1305666491\)");echo XXRETCODE:$?'
May 17 14:08:11 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 31a: voluntary cancel / blocking ast race============================================= 14:08:11 (1305666491)
May 17 14:08:11 client-18 xinetd[3390]: EXIT: shell status=0 pid=32366 duration=0(sec)
May 17 14:08:11 client-18 xinetd[3390]: START: shell pid=32401 from=192.168.4.5
May 17 14:08:11 client-18 rshd[32402]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:08:11 client-18 xinetd[3390]: EXIT: shell status=0 pid=32401 duration=0(sec)
May 17 14:08:11 client-18 xinetd[3390]: START: shell pid=32437 from=192.168.4.5
May 17 14:08:11 client-18 rshd[32438]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:08:11 client-18 xinetd[3390]: EXIT: shell status=0 pid=32437 duration=0(sec)
May 17 14:08:11 client-18 xinetd[3390]: START: shell pid=32474 from=192.168.4.5
May 17 14:08:11 client-18 rshd[32475]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 31b: voluntary OST cancel \/ blocking ast race========================================= 14:08:11 \(1305666491\)");echo XXRETCODE:$?'
May 17 14:08:11 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 31b: voluntary OST cancel / blocking ast race========================================= 14:08:11 (1305666491)
May 17 14:08:11 client-18 xinetd[3390]: EXIT: shell status=0 pid=32474 duration=0(sec)
May 17 14:08:12 client-18 xinetd[3390]: START: shell pid=32509 from=192.168.4.5
May 17 14:08:12 client-18 rshd[32510]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:08:12 client-18 xinetd[3390]: EXIT: shell status=0 pid=32509 duration=0(sec)
May 17 14:08:12 client-18 xinetd[3390]: START: shell pid=32545 from=192.168.4.5
May 17 14:08:12 client-18 rshd[32546]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:08:12 client-18 xinetd[3390]: EXIT: shell status=0 pid=32545 duration=0(sec)
May 17 14:08:12 client-18 xinetd[3390]: START: shell pid=32582 from=192.168.4.5
May 17 14:08:12 client-18 rshd[32583]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 32a: lockless truncate =============================================================== 14:08:12 \(1305666492\)");echo XXRETCODE:$?'
May 17 14:08:12 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 32a: lockless truncate =============================================================== 14:08:12 (1305666492)
May 17 14:08:12 client-18 xinetd[3390]: EXIT: shell status=0 pid=32582 duration=0(sec)
May 17 14:08:13 client-18 xinetd[3390]: START: shell pid=32617 from=192.168.4.5
May 17 14:08:13 client-18 rshd[32618]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark checking cached lockless truncate");echo XXRETCODE:$?'
May 17 14:08:13 client-18 kernel: Lustre: DEBUG MARKER: checking cached lockless truncate
May 17 14:08:13 client-18 xinetd[3390]: EXIT: shell status=0 pid=32617 duration=0(sec)
May 17 14:08:13 client-18 xinetd[3390]: START: shell pid=32652 from=192.168.4.5
May 17 14:08:13 client-18 rshd[32653]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark checking not cached lockless truncate");echo XXRETCODE:$?'
May 17 14:08:13 client-18 kernel: Lustre: DEBUG MARKER: checking not cached lockless truncate
May 17 14:08:13 client-18 xinetd[3390]: EXIT: shell status=0 pid=32652 duration=0(sec)
May 17 14:08:13 client-18 xinetd[3390]: START: shell pid=32687 from=192.168.4.5
May 17 14:08:13 client-18 rshd[32688]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark disabled lockless truncate");echo XXRETCODE:$?'
May 17 14:08:13 client-18 kernel: Lustre: DEBUG MARKER: disabled lockless truncate
May 17 14:08:13 client-18 xinetd[3390]: EXIT: shell status=0 pid=32687 duration=0(sec)
May 17 14:08:13 client-18 xinetd[3390]: START: shell pid=32722 from=192.168.4.5
May 17 14:08:13 client-18 rshd[32723]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:08:13 client-18 xinetd[3390]: EXIT: shell status=0 pid=32722 duration=0(sec)
May 17 14:08:13 client-18 xinetd[3390]: START: shell pid=32758 from=192.168.4.5
May 17 14:08:14 client-18 rshd[32759]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:08:14 client-18 xinetd[3390]: EXIT: shell status=0 pid=32758 duration=1(sec)
May 17 14:08:14 client-18 xinetd[3390]: START: shell pid=329 from=192.168.4.5
May 17 14:08:14 client-18 rshd[330]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 32b: lockless i\/o ==================================================================== 14:08:14 \(1305666494\)");echo XXRETCODE:$?'
May 17 14:08:14 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 32b: lockless i/o ==================================================================== 14:08:14 (1305666494)
May 17 14:08:14 client-18 xinetd[3390]: EXIT: shell status=0 pid=329 duration=0(sec)
May 17 14:08:17 client-18 xinetd[3390]: START: shell pid=364 from=192.168.4.5
May 17 14:08:17 client-18 rshd[365]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:08:17 client-18 xinetd[3390]: EXIT: shell status=0 pid=364 duration=0(sec)
May 17 14:08:17 client-18 xinetd[3390]: START: shell pid=408 from=192.168.4.5
May 17 14:08:17 client-18 rshd[409]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:08:17 client-18 xinetd[3390]: EXIT: shell status=0 pid=408 duration=0(sec)
May 17 14:08:17 client-18 xinetd[3390]: START: shell pid=445 from=192.168.4.5
May 17 14:08:17 client-18 rshd[446]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 33a: commit on sharing, cross crete\/delete, 2 clients, benchmark ===================== 14:08:17 \(1305666497\)");echo XXRETCODE:$?'
May 17 14:08:17 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 33a: commit on sharing, cross crete/delete, 2 clients, benchmark ===================== 14:08:17 (1305666497)
May 17 14:08:17 client-18 xinetd[3390]: EXIT: shell status=0 pid=445 duration=0(sec)
May 17 14:08:17 client-18 xinetd[3390]: START: shell pid=480 from=192.168.4.5
May 17 14:08:17 client-18 rshd[481]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "mkdir -p /mnt/lustre/d0.sanityn/d33-\$(hostname)-1");echo XXRETCODE:$?'
May 17 14:08:17 client-18 xinetd[3390]: EXIT: shell status=0 pid=480 duration=0(sec)
May 17 14:08:18 client-18 xinetd[3390]: START: shell pid=517 from=192.168.4.5
May 17 14:08:18 client-18 rshd[518]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "createmany -o /mnt/lustre/d0.sanityn/d33-\$(hostname)-1/f- -r /mnt/lustre2/d0.sanityn/d33-\$(hostname)-1/f- 10000 > /dev/null 2>&1");echo XXRETCODE:$?'
May 17 14:08:37 client-18 xinetd[3390]: EXIT: shell status=0 pid=517 duration=19(sec)
May 17 14:08:37 client-18 xinetd[3390]: START: shell pid=558 from=192.168.4.5
May 17 14:08:38 client-18 rshd[559]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "mkdir -p /mnt/lustre/d0.sanityn/d33-\$(hostname)-2");echo XXRETCODE:$?'
May 17 14:08:38 client-18 xinetd[3390]: EXIT: shell status=0 pid=558 duration=1(sec)
May 17 14:08:38 client-18 xinetd[3390]: START: shell pid=595 from=192.168.4.5
May 17 14:08:38 client-18 rshd[596]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "createmany -o /mnt/lustre/d0.sanityn/d33-\$(hostname)-2/f- -r /mnt/lustre2/d0.sanityn/d33-\$(hostname)-2/f- 10000 > /dev/null 2>&1");echo XXRETCODE:$?'
May 17 14:08:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=595 duration=19(sec)
May 17 14:08:57 client-18 xinetd[3390]: START: shell pid=652 from=192.168.4.5
May 17 14:08:57 client-18 rshd[653]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "mkdir -p /mnt/lustre/d0.sanityn/d33-\$(hostname)-3");echo XXRETCODE:$?'
May 17 14:08:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=652 duration=0(sec)
May 17 14:08:58 client-18 xinetd[3390]: START: shell pid=690 from=192.168.4.5
May 17 14:08:58 client-18 rshd[691]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "createmany -o /mnt/lustre/d0.sanityn/d33-\$(hostname)-3/f- -r /mnt/lustre2/d0.sanityn/d33-\$(hostname)-3/f- 10000 > /dev/null 2>&1");echo XXRETCODE:$?'
May 17 14:09:17 client-18 xinetd[3390]: EXIT: shell status=0 pid=690 duration=19(sec)
May 17 14:09:17 client-18 xinetd[3390]: START: shell pid=731 from=192.168.4.5
May 17 14:09:17 client-18 rshd[732]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "mkdir -p /mnt/lustre/d0.sanityn/d33-\$(hostname)-1");echo XXRETCODE:$?'
May 17 14:09:17 client-18 xinetd[3390]: EXIT: shell status=0 pid=731 duration=0(sec)
May 17 14:09:17 client-18 xinetd[3390]: START: shell pid=768 from=192.168.4.5
May 17 14:09:17 client-18 rshd[769]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "createmany -o /mnt/lustre/d0.sanityn/d33-\$(hostname)-1/f- -r /mnt/lustre2/d0.sanityn/d33-\$(hostname)-1/f- 10000 > /dev/null 2>&1");echo XXRETCODE:$?'
May 17 14:09:54 client-18 xinetd[3390]: EXIT: shell status=0 pid=768 duration=37(sec)
May 17 14:09:55 client-18 xinetd[3390]: START: shell pid=812 from=192.168.4.5
May 17 14:09:55 client-18 rshd[813]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "mkdir -p /mnt/lustre/d0.sanityn/d33-\$(hostname)-2");echo XXRETCODE:$?'
May 17 14:09:55 client-18 xinetd[3390]: EXIT: shell status=0 pid=812 duration=0(sec)
May 17 14:09:55 client-18 xinetd[3390]: START: shell pid=849 from=192.168.4.5
May 17 14:09:55 client-18 rshd[850]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "createmany -o /mnt/lustre/d0.sanityn/d33-\$(hostname)-2/f- -r /mnt/lustre2/d0.sanityn/d33-\$(hostname)-2/f- 10000 > /dev/null 2>&1");echo XXRETCODE:$?'
May 17 14:10:32 client-18 xinetd[3390]: EXIT: shell status=0 pid=849 duration=37(sec)
May 17 14:10:32 client-18 xinetd[3390]: START: shell pid=889 from=192.168.4.5
May 17 14:10:32 client-18 rshd[890]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "mkdir -p /mnt/lustre/d0.sanityn/d33-\$(hostname)-3");echo XXRETCODE:$?'
May 17 14:10:32 client-18 xinetd[3390]: EXIT: shell status=0 pid=889 duration=0(sec)
May 17 14:10:32 client-18 xinetd[3390]: START: shell pid=926 from=192.168.4.5
May 17 14:10:32 client-18 rshd[927]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "createmany -o /mnt/lustre/d0.sanityn/d33-\$(hostname)-3/f- -r /mnt/lustre2/d0.sanityn/d33-\$(hostname)-3/f- 10000 > /dev/null 2>&1");echo XXRETCODE:$?'
May 17 14:11:07 client-18 xinetd[3390]: EXIT: shell status=0 pid=926 duration=35(sec)
May 17 14:11:07 client-18 xinetd[3390]: START: shell pid=966 from=192.168.4.5
May 17 14:11:07 client-18 rshd[967]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:11:07 client-18 xinetd[3390]: EXIT: shell status=0 pid=966 duration=0(sec)
May 17 14:11:08 client-18 xinetd[3390]: START: shell pid=1002 from=192.168.4.5
May 17 14:11:08 client-18 rshd[1003]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:11:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=1002 duration=0(sec)
May 17 14:11:08 client-18 xinetd[3390]: START: shell pid=1039 from=192.168.4.5
May 17 14:11:08 client-18 rshd[1040]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 34: no lock timeout under IO ========================================================= 14:11:08 \(1305666668\)");echo XXRETCODE:$?'
May 17 14:11:08 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 34: no lock timeout under IO ========================================================= 14:11:08 (1305666668)
May 17 14:11:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=1039 duration=0(sec)
May 17 14:11:30 client-18 xinetd[3390]: START: shell pid=1081 from=192.168.4.5
May 17 14:11:30 client-18 rshd[1082]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:11:30 client-18 xinetd[3390]: EXIT: shell status=0 pid=1081 duration=0(sec)
May 17 14:11:30 client-18 xinetd[3390]: START: shell pid=1117 from=192.168.4.5
May 17 14:11:30 client-18 rshd[1118]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:11:30 client-18 xinetd[3390]: EXIT: shell status=0 pid=1117 duration=0(sec)
May 17 14:11:30 client-18 xinetd[3390]: START: shell pid=1154 from=192.168.4.5
May 17 14:11:30 client-18 rshd[1155]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark  SKIP: sanityn test_35 skipping ALWAYS excluded test 35");echo XXRETCODE:$?'
May 17 14:11:30 client-18 kernel: Lustre: DEBUG MARKER: SKIP: sanityn test_35 skipping ALWAYS excluded test 35
May 17 14:11:30 client-18 xinetd[3390]: EXIT: shell status=0 pid=1154 duration=0(sec)
May 17 14:11:30 client-18 xinetd[3390]: START: shell pid=1189 from=192.168.4.5
May 17 14:11:30 client-18 rshd[1190]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 36: handle ESTALE\/open-unlink corectly =============================================== 14:11:30 \(1305666690\)");echo XXRETCODE:$?'
May 17 14:11:30 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 36: handle ESTALE/open-unlink corectly =============================================== 14:11:30 (1305666690)
May 17 14:11:30 client-18 xinetd[3390]: EXIT: shell status=0 pid=1189 duration=0(sec)
May 17 14:12:03 client-18 xinetd[3390]: START: shell pid=1224 from=192.168.4.5
May 17 14:12:03 client-18 rshd[1225]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:12:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=1224 duration=0(sec)
May 17 14:12:03 client-18 xinetd[3390]: START: shell pid=1260 from=192.168.4.5
May 17 14:12:03 client-18 rshd[1261]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:12:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=1260 duration=0(sec)
May 17 14:12:03 client-18 xinetd[3390]: START: shell pid=1297 from=192.168.4.5
May 17 14:12:03 client-18 rshd[1298]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 37: check i_size is not updated for directory on close \(bug 18695\) =================== 14:12:03 \(1305666723\)");echo XXRETCODE:$?'
May 17 14:12:03 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 37: check i_size is not updated for directory on close (bug 18695) =================== 14:12:03 (1305666723)
May 17 14:12:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=1297 duration=0(sec)
May 17 14:12:12 client-18 xinetd[3390]: START: shell pid=1332 from=192.168.4.5
May 17 14:12:12 client-18 rshd[1333]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:12:12 client-18 xinetd[3390]: EXIT: shell status=0 pid=1332 duration=0(sec)
May 17 14:12:12 client-18 xinetd[3390]: START: shell pid=1368 from=192.168.4.5
May 17 14:12:12 client-18 rshd[1369]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:12:12 client-18 xinetd[3390]: EXIT: shell status=0 pid=1368 duration=0(sec)
May 17 14:12:12 client-18 xinetd[3390]: START: shell pid=1405 from=192.168.4.5
May 17 14:12:12 client-18 rshd[1406]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 39a: test from 11063 ================================================================= 14:12:12 \(1305666732\)");echo XXRETCODE:$?'
May 17 14:12:12 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 39a: test from 11063 ================================================================= 14:12:12 (1305666732)
May 17 14:12:12 client-18 xinetd[3390]: EXIT: shell status=0 pid=1405 duration=0(sec)
May 17 14:12:12 client-18 xinetd[3390]: START: shell pid=1441 from=192.168.4.5
May 17 14:12:12 client-18 rshd[1442]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "stat -c %Y /mnt/lustre/f39a");echo XXRETCODE:$?'
May 17 14:12:12 client-18 xinetd[3390]: EXIT: shell status=0 pid=1441 duration=0(sec)
May 17 14:12:12 client-18 xinetd[3390]: START: shell pid=1476 from=192.168.4.5
May 17 14:12:12 client-18 rshd[1477]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "stat -c %Y /mnt/lustre/f39a");echo XXRETCODE:$?'
May 17 14:12:12 client-18 xinetd[3390]: EXIT: shell status=0 pid=1476 duration=0(sec)
May 17 14:12:12 client-18 xinetd[3390]: START: shell pid=1511 from=192.168.4.5
May 17 14:12:12 client-18 rshd[1512]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "stat -c %Y /mnt/lustre/f39a-1");echo XXRETCODE:$?'
May 17 14:12:13 client-18 xinetd[3390]: EXIT: shell status=0 pid=1511 duration=1(sec)
May 17 14:12:13 client-18 xinetd[3390]: START: shell pid=1546 from=192.168.4.5
May 17 14:12:13 client-18 rshd[1547]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "stat -c %Y /mnt/lustre/f39a-1");echo XXRETCODE:$?'
May 17 14:12:13 client-18 xinetd[3390]: EXIT: shell status=0 pid=1546 duration=0(sec)
May 17 14:12:13 client-18 xinetd[3390]: START: shell pid=1581 from=192.168.4.5
May 17 14:12:13 client-18 rshd[1582]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:12:13 client-18 xinetd[3390]: EXIT: shell status=0 pid=1581 duration=0(sec)
May 17 14:12:13 client-18 xinetd[3390]: START: shell pid=1617 from=192.168.4.5
May 17 14:12:13 client-18 rshd[1618]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:12:13 client-18 xinetd[3390]: EXIT: shell status=0 pid=1617 duration=0(sec)
May 17 14:12:13 client-18 xinetd[3390]: START: shell pid=1655 from=192.168.4.5
May 17 14:12:13 client-18 rshd[1656]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 39b: 11063 problem 1 ================================================================= 14:12:13 \(1305666733\)");echo XXRETCODE:$?'
May 17 14:12:13 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 39b: 11063 problem 1 ================================================================= 14:12:13 (1305666733)
May 17 14:12:13 client-18 xinetd[3390]: EXIT: shell status=0 pid=1655 duration=0(sec)
May 17 14:12:13 client-18 xinetd[3390]: START: shell pid=1692 from=192.168.4.5
May 17 14:12:13 client-18 rshd[1693]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "stat -c %Y /mnt/lustre/f39b");echo XXRETCODE:$?'
May 17 14:12:13 client-18 xinetd[3390]: EXIT: shell status=0 pid=1692 duration=0(sec)
May 17 14:12:14 client-18 xinetd[3390]: START: shell pid=1727 from=192.168.4.5
May 17 14:12:14 client-18 rshd[1728]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "stat -c %Y /mnt/lustre/f39b");echo XXRETCODE:$?'
May 17 14:12:14 client-18 xinetd[3390]: EXIT: shell status=0 pid=1727 duration=0(sec)
May 17 14:12:14 client-18 xinetd[3390]: START: shell pid=1762 from=192.168.4.5
May 17 14:12:15 client-18 rshd[1763]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "stat -c %Y /mnt/lustre/f39b");echo XXRETCODE:$?'
May 17 14:12:15 client-18 xinetd[3390]: EXIT: shell status=0 pid=1762 duration=1(sec)
May 17 14:12:15 client-18 xinetd[3390]: START: shell pid=1797 from=192.168.4.5
May 17 14:12:15 client-18 rshd[1798]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:12:15 client-18 xinetd[3390]: EXIT: shell status=0 pid=1797 duration=0(sec)
May 17 14:12:15 client-18 xinetd[3390]: START: shell pid=1833 from=192.168.4.5
May 17 14:12:15 client-18 rshd[1834]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:12:15 client-18 xinetd[3390]: EXIT: shell status=0 pid=1833 duration=0(sec)
May 17 14:12:15 client-18 xinetd[3390]: START: shell pid=1870 from=192.168.4.5
May 17 14:12:15 client-18 rshd[1871]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 39c: check truncate mtime update ===================================================== 14:12:15 \(1305666735\)");echo XXRETCODE:$?'
May 17 14:12:15 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 39c: check truncate mtime update ===================================================== 14:12:15 (1305666735)
May 17 14:12:15 client-18 xinetd[3390]: EXIT: shell status=0 pid=1870 duration=0(sec)
May 17 14:12:15 client-18 xinetd[3390]: START: shell pid=1910 from=192.168.4.5
May 17 14:12:15 client-18 rshd[1911]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "stat -c %Y /mnt/lustre/f39c");echo XXRETCODE:$?'
May 17 14:12:15 client-18 xinetd[3390]: EXIT: shell status=0 pid=1910 duration=0(sec)
May 17 14:12:16 client-18 xinetd[3390]: START: shell pid=1945 from=192.168.4.5
May 17 14:12:16 client-18 rshd[1946]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "stat -c %Y /mnt/lustre/f39c");echo XXRETCODE:$?'
May 17 14:12:16 client-18 xinetd[3390]: EXIT: shell status=0 pid=1945 duration=0(sec)
May 17 14:12:16 client-18 xinetd[3390]: START: shell pid=1980 from=192.168.4.5
May 17 14:12:16 client-18 rshd[1981]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "stat -c %Y /mnt/lustre/f39c");echo XXRETCODE:$?'
May 17 14:12:16 client-18 xinetd[3390]: EXIT: shell status=0 pid=1980 duration=0(sec)
May 17 14:12:17 client-18 xinetd[3390]: START: shell pid=2015 from=192.168.4.5
May 17 14:12:17 client-18 rshd[2016]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:12:17 client-18 xinetd[3390]: EXIT: shell status=0 pid=2015 duration=0(sec)
May 17 14:12:17 client-18 xinetd[3390]: START: shell pid=2052 from=192.168.4.5
May 17 14:12:17 client-18 rshd[2053]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:12:17 client-18 xinetd[3390]: EXIT: shell status=0 pid=2052 duration=0(sec)
May 17 14:12:17 client-18 xinetd[3390]: START: shell pid=2089 from=192.168.4.5
May 17 14:12:17 client-18 rshd[2090]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 40a: pdirops: create vs others ======================================================= 14:12:17 \(1305666737\)");echo XXRETCODE:$?'
May 17 14:12:17 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 40a: pdirops: create vs others ======================================================= 14:12:17 (1305666737)
May 17 14:12:17 client-18 xinetd[3390]: EXIT: shell status=0 pid=2089 duration=0(sec)
May 17 14:12:34 client-18 xinetd[3390]: START: shell pid=2129 from=192.168.4.5
May 17 14:12:34 client-18 rshd[2130]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:12:34 client-18 xinetd[3390]: EXIT: shell status=0 pid=2129 duration=0(sec)
May 17 14:12:34 client-18 xinetd[3390]: START: shell pid=2166 from=192.168.4.5
May 17 14:12:34 client-18 rshd[2167]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:12:34 client-18 xinetd[3390]: EXIT: shell status=0 pid=2166 duration=0(sec)
May 17 14:12:35 client-18 xinetd[3390]: START: shell pid=2207 from=192.168.4.5
May 17 14:12:35 client-18 rshd[2208]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 40b: pdirops: open\|create and others ================================================= 14:12:34 \(1305666754\)");echo XXRETCODE:$?'
May 17 14:12:35 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 40b: pdirops: open|create and others ================================================= 14:12:34 (1305666754)
May 17 14:12:35 client-18 xinetd[3390]: EXIT: shell status=0 pid=2207 duration=0(sec)
May 17 14:12:45 client-18 xinetd[3390]: START: shell pid=2242 from=192.168.4.5
May 17 14:12:45 client-18 rshd[2246]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:12:45 client-18 xinetd[3390]: EXIT: shell status=0 pid=2242 duration=0(sec)
May 17 14:12:45 client-18 xinetd[3390]: START: shell pid=2286 from=192.168.4.5
May 17 14:12:45 client-18 rshd[2287]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:12:45 client-18 xinetd[3390]: EXIT: shell status=0 pid=2286 duration=0(sec)
May 17 14:12:45 client-18 xinetd[3390]: START: shell pid=2323 from=192.168.4.5
May 17 14:12:45 client-18 rshd[2324]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 40c: pdirops: link and others ======================================================== 14:12:45 \(1305666765\)");echo XXRETCODE:$?'
May 17 14:12:45 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 40c: pdirops: link and others ======================================================== 14:12:45 (1305666765)
May 17 14:12:45 client-18 xinetd[3390]: EXIT: shell status=0 pid=2323 duration=0(sec)
May 17 14:12:55 client-18 xinetd[3390]: START: shell pid=2358 from=192.168.4.5
May 17 14:12:55 client-18 rshd[2359]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:12:56 client-18 xinetd[3390]: EXIT: shell status=0 pid=2358 duration=1(sec)
May 17 14:12:56 client-18 xinetd[3390]: START: shell pid=2397 from=192.168.4.5
May 17 14:12:56 client-18 rshd[2398]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:12:56 client-18 xinetd[3390]: EXIT: shell status=0 pid=2397 duration=0(sec)
May 17 14:12:56 client-18 xinetd[3390]: START: shell pid=2442 from=192.168.4.5
May 17 14:12:56 client-18 rshd[2443]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 40d: pdirops: unlink and others ====================================================== 14:12:56 \(1305666776\)");echo XXRETCODE:$?'
May 17 14:12:56 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 40d: pdirops: unlink and others ====================================================== 14:12:56 (1305666776)
May 17 14:12:56 client-18 xinetd[3390]: EXIT: shell status=0 pid=2442 duration=0(sec)
May 17 14:13:06 client-18 xinetd[3390]: START: shell pid=2478 from=192.168.4.5
May 17 14:13:06 client-18 rshd[2479]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:13:06 client-18 xinetd[3390]: EXIT: shell status=0 pid=2478 duration=0(sec)
May 17 14:13:06 client-18 xinetd[3390]: START: shell pid=2514 from=192.168.4.5
May 17 14:13:06 client-18 rshd[2515]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:13:06 client-18 xinetd[3390]: EXIT: shell status=0 pid=2514 duration=0(sec)
May 17 14:13:06 client-18 xinetd[3390]: START: shell pid=2556 from=192.168.4.5
May 17 14:13:06 client-18 rshd[2557]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 40e: pdirops: rename and others ====================================================== 14:13:06 \(1305666786\)");echo XXRETCODE:$?'
May 17 14:13:06 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 40e: pdirops: rename and others ====================================================== 14:13:06 (1305666786)
May 17 14:13:06 client-18 xinetd[3390]: EXIT: shell status=0 pid=2556 duration=0(sec)
May 17 14:13:17 client-18 xinetd[3390]: START: shell pid=2591 from=192.168.4.5
May 17 14:13:17 client-18 rshd[2592]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:13:17 client-18 xinetd[3390]: EXIT: shell status=0 pid=2591 duration=0(sec)
May 17 14:13:17 client-18 xinetd[3390]: START: shell pid=2627 from=192.168.4.5
May 17 14:13:17 client-18 rshd[2628]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:13:17 client-18 xinetd[3390]: EXIT: shell status=0 pid=2627 duration=0(sec)
May 17 14:13:17 client-18 xinetd[3390]: START: shell pid=2664 from=192.168.4.5
May 17 14:13:17 client-18 rshd[2665]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 41a: pdirops: create vs mkdir ======================================================== 14:13:17 \(1305666797\)");echo XXRETCODE:$?'
May 17 14:13:17 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 41a: pdirops: create vs mkdir ======================================================== 14:13:17 (1305666797)
May 17 14:13:17 client-18 xinetd[3390]: EXIT: shell status=0 pid=2664 duration=0(sec)
May 17 14:13:27 client-18 xinetd[3390]: START: shell pid=2699 from=192.168.4.5
May 17 14:13:27 client-18 rshd[2700]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:13:27 client-18 xinetd[3390]: EXIT: shell status=0 pid=2699 duration=0(sec)
May 17 14:13:27 client-18 xinetd[3390]: START: shell pid=2735 from=192.168.4.5
May 17 14:13:27 client-18 rshd[2736]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:13:27 client-18 xinetd[3390]: EXIT: shell status=0 pid=2735 duration=0(sec)
May 17 14:13:28 client-18 xinetd[3390]: START: shell pid=2772 from=192.168.4.5
May 17 14:13:28 client-18 rshd[2773]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 41b: pdirops: create vs create ======================================================= 14:13:28 \(1305666808\)");echo XXRETCODE:$?'
May 17 14:13:28 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 41b: pdirops: create vs create ======================================================= 14:13:28 (1305666808)
May 17 14:13:28 client-18 xinetd[3390]: EXIT: shell status=0 pid=2772 duration=0(sec)
May 17 14:13:38 client-18 xinetd[3390]: START: shell pid=2807 from=192.168.4.5
May 17 14:13:38 client-18 rshd[2808]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:13:38 client-18 xinetd[3390]: EXIT: shell status=0 pid=2807 duration=0(sec)
May 17 14:13:38 client-18 xinetd[3390]: START: shell pid=2843 from=192.168.4.5
May 17 14:13:38 client-18 rshd[2844]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:13:38 client-18 xinetd[3390]: EXIT: shell status=0 pid=2843 duration=0(sec)
May 17 14:13:38 client-18 xinetd[3390]: START: shell pid=2880 from=192.168.4.5
May 17 14:13:38 client-18 rshd[2881]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 41c: pdirops: create vs link ========================================================= 14:13:38 \(1305666818\)");echo XXRETCODE:$?'
May 17 14:13:38 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 41c: pdirops: create vs link ========================================================= 14:13:38 (1305666818)
May 17 14:13:38 client-18 xinetd[3390]: EXIT: shell status=0 pid=2880 duration=0(sec)
May 17 14:13:48 client-18 xinetd[3390]: START: shell pid=2919 from=192.168.4.5
May 17 14:13:48 client-18 rshd[2920]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:13:49 client-18 xinetd[3390]: EXIT: shell status=0 pid=2919 duration=1(sec)
May 17 14:13:49 client-18 xinetd[3390]: START: shell pid=2955 from=192.168.4.5
May 17 14:13:49 client-18 rshd[2956]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:13:49 client-18 xinetd[3390]: EXIT: shell status=0 pid=2955 duration=0(sec)
May 17 14:13:49 client-18 xinetd[3390]: START: shell pid=2992 from=192.168.4.5
May 17 14:13:49 client-18 rshd[2993]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 41d: pdirops: create vs unlink ======================================================= 14:13:49 \(1305666829\)");echo XXRETCODE:$?'
May 17 14:13:49 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 41d: pdirops: create vs unlink ======================================================= 14:13:49 (1305666829)
May 17 14:13:49 client-18 xinetd[3390]: EXIT: shell status=0 pid=2992 duration=0(sec)
May 17 14:13:59 client-18 xinetd[3390]: START: shell pid=3033 from=192.168.4.5
May 17 14:13:59 client-18 rshd[3034]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:13:59 client-18 xinetd[3390]: EXIT: shell status=0 pid=3033 duration=0(sec)
May 17 14:13:59 client-18 xinetd[3390]: START: shell pid=3069 from=192.168.4.5
May 17 14:13:59 client-18 rshd[3070]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:13:59 client-18 xinetd[3390]: EXIT: shell status=0 pid=3069 duration=0(sec)
May 17 14:13:59 client-18 xinetd[3390]: START: shell pid=3112 from=192.168.4.5
May 17 14:13:59 client-18 rshd[3113]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 41e: pdirops: create and rename \(tgt\) ================================================ 14:13:59 \(1305666839\)");echo XXRETCODE:$?'
May 17 14:14:00 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 41e: pdirops: create and rename (tgt) ================================================ 14:13:59 (1305666839)
May 17 14:14:00 client-18 xinetd[3390]: EXIT: shell status=0 pid=3112 duration=1(sec)
May 17 14:14:10 client-18 xinetd[3390]: START: shell pid=3152 from=192.168.4.5
May 17 14:14:10 client-18 rshd[3153]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:14:10 client-18 xinetd[3390]: EXIT: shell status=0 pid=3152 duration=0(sec)
May 17 14:14:10 client-18 xinetd[3390]: START: shell pid=3192 from=192.168.4.5
May 17 14:14:10 client-18 rshd[3193]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:14:10 client-18 xinetd[3390]: EXIT: shell status=0 pid=3192 duration=0(sec)
May 17 14:14:10 client-18 xinetd[3390]: START: shell pid=3230 from=192.168.4.5
May 17 14:14:10 client-18 rshd[3231]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 41f: pdirops: create and rename \(src\) ================================================ 14:14:10 \(1305666850\)");echo XXRETCODE:$?'
May 17 14:14:10 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 41f: pdirops: create and rename (src) ================================================ 14:14:10 (1305666850)
May 17 14:14:10 client-18 xinetd[3390]: EXIT: shell status=0 pid=3230 duration=0(sec)
May 17 14:14:20 client-18 xinetd[3390]: START: shell pid=3267 from=192.168.4.5
May 17 14:14:20 client-18 rshd[3268]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:14:20 client-18 xinetd[3390]: EXIT: shell status=0 pid=3267 duration=0(sec)
May 17 14:14:20 client-18 xinetd[3390]: START: shell pid=3308 from=192.168.4.5
May 17 14:14:20 client-18 rshd[3309]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:14:21 client-18 xinetd[3390]: EXIT: shell status=0 pid=3308 duration=1(sec)
May 17 14:14:21 client-18 xinetd[3390]: START: shell pid=3350 from=192.168.4.5
May 17 14:14:21 client-18 rshd[3354]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 41g: pdirops: create vs getattr ====================================================== 14:14:21 \(1305666861\)");echo XXRETCODE:$?'
May 17 14:14:21 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 41g: pdirops: create vs getattr ====================================================== 14:14:21 (1305666861)
May 17 14:14:21 client-18 xinetd[3390]: EXIT: shell status=0 pid=3350 duration=0(sec)
May 17 14:14:31 client-18 xinetd[3390]: START: shell pid=3394 from=192.168.4.5
May 17 14:14:31 client-18 rshd[3395]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:14:31 client-18 xinetd[3390]: EXIT: shell status=0 pid=3394 duration=0(sec)
May 17 14:14:31 client-18 xinetd[3390]: START: shell pid=3433 from=192.168.4.5
May 17 14:14:31 client-18 rshd[3434]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:14:31 client-18 xinetd[3390]: EXIT: shell status=0 pid=3433 duration=0(sec)
May 17 14:14:31 client-18 xinetd[3390]: START: shell pid=3474 from=192.168.4.5
May 17 14:14:31 client-18 rshd[3475]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 41h: pdirops: create vs readdir ====================================================== 14:14:31 \(1305666871\)");echo XXRETCODE:$?'
May 17 14:14:31 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 41h: pdirops: create vs readdir ====================================================== 14:14:31 (1305666871)
May 17 14:14:31 client-18 xinetd[3390]: EXIT: shell status=0 pid=3474 duration=0(sec)
May 17 14:14:42 client-18 xinetd[3390]: START: shell pid=3514 from=192.168.4.5
May 17 14:14:42 client-18 rshd[3515]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:14:42 client-18 xinetd[3390]: EXIT: shell status=0 pid=3514 duration=0(sec)
May 17 14:14:42 client-18 xinetd[3390]: START: shell pid=3554 from=192.168.4.5
May 17 14:14:42 client-18 rshd[3555]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:14:42 client-18 xinetd[3390]: EXIT: shell status=0 pid=3554 duration=0(sec)
May 17 14:14:42 client-18 xinetd[3390]: START: shell pid=3594 from=192.168.4.5
May 17 14:14:42 client-18 rshd[3595]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 42a: pdirops: mkdir vs mkdir ========================================================= 14:14:42 \(1305666882\)");echo XXRETCODE:$?'
May 17 14:14:42 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 42a: pdirops: mkdir vs mkdir ========================================================= 14:14:42 (1305666882)
May 17 14:14:42 client-18 xinetd[3390]: EXIT: shell status=0 pid=3594 duration=0(sec)
May 17 14:14:52 client-18 xinetd[3390]: START: shell pid=3629 from=192.168.4.5
May 17 14:14:52 client-18 rshd[3630]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:14:52 client-18 xinetd[3390]: EXIT: shell status=0 pid=3629 duration=0(sec)
May 17 14:14:52 client-18 xinetd[3390]: START: shell pid=3665 from=192.168.4.5
May 17 14:14:52 client-18 rshd[3666]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:14:52 client-18 xinetd[3390]: EXIT: shell status=0 pid=3665 duration=0(sec)
May 17 14:14:53 client-18 xinetd[3390]: START: shell pid=3702 from=192.168.4.5
May 17 14:14:53 client-18 rshd[3703]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 42b: pdirops: mkdir vs create ======================================================== 14:14:52 \(1305666892\)");echo XXRETCODE:$?'
May 17 14:14:53 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 42b: pdirops: mkdir vs create ======================================================== 14:14:52 (1305666892)
May 17 14:14:53 client-18 xinetd[3390]: EXIT: shell status=0 pid=3702 duration=0(sec)
May 17 14:15:03 client-18 xinetd[3390]: START: shell pid=3741 from=192.168.4.5
May 17 14:15:03 client-18 rshd[3742]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:15:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=3741 duration=0(sec)
May 17 14:15:03 client-18 xinetd[3390]: START: shell pid=3777 from=192.168.4.5
May 17 14:15:03 client-18 rshd[3778]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:15:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=3777 duration=0(sec)
May 17 14:15:03 client-18 xinetd[3390]: START: shell pid=3820 from=192.168.4.5
May 17 14:15:03 client-18 rshd[3821]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 42c: pdirops: mkdir vs link ========================================================== 14:15:03 \(1305666903\)");echo XXRETCODE:$?'
May 17 14:15:03 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 42c: pdirops: mkdir vs link ========================================================== 14:15:03 (1305666903)
May 17 14:15:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=3820 duration=0(sec)
May 17 14:15:13 client-18 xinetd[3390]: START: shell pid=3880 from=192.168.4.5
May 17 14:15:13 client-18 rshd[3881]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:15:13 client-18 xinetd[3390]: EXIT: shell status=0 pid=3880 duration=0(sec)
May 17 14:15:14 client-18 xinetd[3390]: START: shell pid=3917 from=192.168.4.5
May 17 14:15:14 client-18 rshd[3918]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:15:14 client-18 xinetd[3390]: EXIT: shell status=0 pid=3917 duration=0(sec)
May 17 14:15:14 client-18 xinetd[3390]: START: shell pid=3954 from=192.168.4.5
May 17 14:15:14 client-18 rshd[3955]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 42d: pdirops: mkdir vs unlink ======================================================== 14:15:14 \(1305666914\)");echo XXRETCODE:$?'
May 17 14:15:14 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 42d: pdirops: mkdir vs unlink ======================================================== 14:15:14 (1305666914)
May 17 14:15:14 client-18 xinetd[3390]: EXIT: shell status=0 pid=3954 duration=0(sec)
May 17 14:15:24 client-18 xinetd[3390]: START: shell pid=3989 from=192.168.4.5
May 17 14:15:24 client-18 rshd[3990]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:15:24 client-18 xinetd[3390]: EXIT: shell status=0 pid=3989 duration=0(sec)
May 17 14:15:24 client-18 xinetd[3390]: START: shell pid=4025 from=192.168.4.5
May 17 14:15:24 client-18 rshd[4026]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:15:24 client-18 xinetd[3390]: EXIT: shell status=0 pid=4025 duration=0(sec)
May 17 14:15:24 client-18 xinetd[3390]: START: shell pid=4062 from=192.168.4.5
May 17 14:15:24 client-18 rshd[4063]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 42e: pdirops: mkdir and rename \(tgt\) ================================================= 14:15:24 \(1305666924\)");echo XXRETCODE:$?'
May 17 14:15:24 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 42e: pdirops: mkdir and rename (tgt) ================================================= 14:15:24 (1305666924)
May 17 14:15:24 client-18 xinetd[3390]: EXIT: shell status=0 pid=4062 duration=0(sec)
May 17 14:15:35 client-18 xinetd[3390]: START: shell pid=4097 from=192.168.4.5
May 17 14:15:35 client-18 rshd[4098]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:15:35 client-18 xinetd[3390]: EXIT: shell status=0 pid=4097 duration=0(sec)
May 17 14:15:35 client-18 xinetd[3390]: START: shell pid=4133 from=192.168.4.5
May 17 14:15:35 client-18 rshd[4134]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:15:35 client-18 xinetd[3390]: EXIT: shell status=0 pid=4133 duration=0(sec)
May 17 14:15:35 client-18 xinetd[3390]: START: shell pid=4170 from=192.168.4.5
May 17 14:15:35 client-18 rshd[4171]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 42f: pdirops: mkdir and rename \(src\) ================================================= 14:15:35 \(1305666935\)");echo XXRETCODE:$?'
May 17 14:15:35 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 42f: pdirops: mkdir and rename (src) ================================================= 14:15:35 (1305666935)
May 17 14:15:35 client-18 xinetd[3390]: EXIT: shell status=0 pid=4170 duration=0(sec)
May 17 14:15:45 client-18 xinetd[3390]: START: shell pid=4205 from=192.168.4.5
May 17 14:15:45 client-18 rshd[4206]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:15:45 client-18 xinetd[3390]: EXIT: shell status=0 pid=4205 duration=0(sec)
May 17 14:15:45 client-18 xinetd[3390]: START: shell pid=4241 from=192.168.4.5
May 17 14:15:45 client-18 rshd[4242]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:15:45 client-18 xinetd[3390]: EXIT: shell status=0 pid=4241 duration=0(sec)
May 17 14:15:46 client-18 xinetd[3390]: START: shell pid=4278 from=192.168.4.5
May 17 14:15:46 client-18 rshd[4279]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 42g: pdirops: mkdir vs getattr ======================================================= 14:15:45 \(1305666945\)");echo XXRETCODE:$?'
May 17 14:15:46 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 42g: pdirops: mkdir vs getattr ======================================================= 14:15:45 (1305666945)
May 17 14:15:46 client-18 xinetd[3390]: EXIT: shell status=0 pid=4278 duration=0(sec)
May 17 14:15:56 client-18 xinetd[3390]: START: shell pid=4313 from=192.168.4.5
May 17 14:15:56 client-18 rshd[4314]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:15:56 client-18 xinetd[3390]: EXIT: shell status=0 pid=4313 duration=0(sec)
May 17 14:15:56 client-18 xinetd[3390]: START: shell pid=4349 from=192.168.4.5
May 17 14:15:56 client-18 rshd[4350]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:15:56 client-18 xinetd[3390]: EXIT: shell status=0 pid=4349 duration=0(sec)
May 17 14:15:56 client-18 xinetd[3390]: START: shell pid=4386 from=192.168.4.5
May 17 14:15:56 client-18 rshd[4387]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 42h: pdirops: mkdir vs readdir ======================================================= 14:15:56 \(1305666956\)");echo XXRETCODE:$?'
May 17 14:15:56 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 42h: pdirops: mkdir vs readdir ======================================================= 14:15:56 (1305666956)
May 17 14:15:56 client-18 xinetd[3390]: EXIT: shell status=0 pid=4386 duration=0(sec)
May 17 14:16:06 client-18 xinetd[3390]: START: shell pid=4421 from=192.168.4.5
May 17 14:16:06 client-18 rshd[4422]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:16:07 client-18 xinetd[3390]: EXIT: shell status=0 pid=4421 duration=1(sec)
May 17 14:16:07 client-18 xinetd[3390]: START: shell pid=4457 from=192.168.4.5
May 17 14:16:07 client-18 rshd[4458]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:16:07 client-18 xinetd[3390]: EXIT: shell status=0 pid=4457 duration=0(sec)
May 17 14:16:07 client-18 xinetd[3390]: START: shell pid=4494 from=192.168.4.5
May 17 14:16:07 client-18 rshd[4495]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 43a: pdirops: unlink vs mkdir ======================================================== 14:16:07 \(1305666967\)");echo XXRETCODE:$?'
May 17 14:16:07 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 43a: pdirops: unlink vs mkdir ======================================================== 14:16:07 (1305666967)
May 17 14:16:07 client-18 xinetd[3390]: EXIT: shell status=0 pid=4494 duration=0(sec)
May 17 14:16:17 client-18 xinetd[3390]: START: shell pid=4533 from=192.168.4.5
May 17 14:16:17 client-18 rshd[4534]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:16:17 client-18 xinetd[3390]: EXIT: shell status=0 pid=4533 duration=0(sec)
May 17 14:16:17 client-18 xinetd[3390]: START: shell pid=4569 from=192.168.4.5
May 17 14:16:17 client-18 rshd[4570]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:16:17 client-18 xinetd[3390]: EXIT: shell status=0 pid=4569 duration=0(sec)
May 17 14:16:17 client-18 xinetd[3390]: START: shell pid=4606 from=192.168.4.5
May 17 14:16:17 client-18 rshd[4607]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 43b: pdirops: unlink vs create ======================================================= 14:16:17 \(1305666977\)");echo XXRETCODE:$?'
May 17 14:16:17 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 43b: pdirops: unlink vs create ======================================================= 14:16:17 (1305666977)
May 17 14:16:17 client-18 xinetd[3390]: EXIT: shell status=0 pid=4606 duration=0(sec)
May 17 14:16:28 client-18 xinetd[3390]: START: shell pid=4641 from=192.168.4.5
May 17 14:16:28 client-18 rshd[4642]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:16:28 client-18 xinetd[3390]: EXIT: shell status=0 pid=4641 duration=0(sec)
May 17 14:16:28 client-18 xinetd[3390]: START: shell pid=4677 from=192.168.4.5
May 17 14:16:28 client-18 rshd[4678]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:16:28 client-18 xinetd[3390]: EXIT: shell status=0 pid=4677 duration=0(sec)
May 17 14:16:28 client-18 xinetd[3390]: START: shell pid=4714 from=192.168.4.5
May 17 14:16:28 client-18 rshd[4715]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 43c: pdirops: unlink vs link ========================================================= 14:16:28 \(1305666988\)");echo XXRETCODE:$?'
May 17 14:16:28 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 43c: pdirops: unlink vs link ========================================================= 14:16:28 (1305666988)
May 17 14:16:28 client-18 xinetd[3390]: EXIT: shell status=0 pid=4714 duration=0(sec)
May 17 14:16:38 client-18 xinetd[3390]: START: shell pid=4749 from=192.168.4.5
May 17 14:16:38 client-18 rshd[4750]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:16:38 client-18 xinetd[3390]: EXIT: shell status=0 pid=4749 duration=0(sec)
May 17 14:16:38 client-18 xinetd[3390]: START: shell pid=4785 from=192.168.4.5
May 17 14:16:38 client-18 rshd[4786]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:16:38 client-18 xinetd[3390]: EXIT: shell status=0 pid=4785 duration=0(sec)
May 17 14:16:39 client-18 xinetd[3390]: START: shell pid=4822 from=192.168.4.5
May 17 14:16:39 client-18 rshd[4823]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 43d: pdirops: unlink vs unlink ======================================================= 14:16:39 \(1305666999\)");echo XXRETCODE:$?'
May 17 14:16:39 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 43d: pdirops: unlink vs unlink ======================================================= 14:16:39 (1305666999)
May 17 14:16:39 client-18 xinetd[3390]: EXIT: shell status=0 pid=4822 duration=0(sec)
May 17 14:16:49 client-18 xinetd[3390]: START: shell pid=4857 from=192.168.4.5
May 17 14:16:49 client-18 rshd[4858]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:16:49 client-18 xinetd[3390]: EXIT: shell status=0 pid=4857 duration=0(sec)
May 17 14:16:49 client-18 xinetd[3390]: START: shell pid=4893 from=192.168.4.5
May 17 14:16:49 client-18 rshd[4894]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:16:49 client-18 xinetd[3390]: EXIT: shell status=0 pid=4893 duration=0(sec)
May 17 14:16:49 client-18 xinetd[3390]: START: shell pid=4930 from=192.168.4.5
May 17 14:16:49 client-18 rshd[4931]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 43e: pdirops: unlink and rename \(tgt\) ================================================ 14:16:49 \(1305667009\)");echo XXRETCODE:$?'
May 17 14:16:49 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 43e: pdirops: unlink and rename (tgt) ================================================ 14:16:49 (1305667009)
May 17 14:16:49 client-18 xinetd[3390]: EXIT: shell status=0 pid=4930 duration=0(sec)
May 17 14:16:59 client-18 xinetd[3390]: START: shell pid=4965 from=192.168.4.5
May 17 14:16:59 client-18 rshd[4966]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:17:00 client-18 xinetd[3390]: EXIT: shell status=0 pid=4965 duration=1(sec)
May 17 14:17:00 client-18 xinetd[3390]: START: shell pid=5001 from=192.168.4.5
May 17 14:17:00 client-18 rshd[5002]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:17:00 client-18 xinetd[3390]: EXIT: shell status=0 pid=5001 duration=0(sec)
May 17 14:17:00 client-18 xinetd[3390]: START: shell pid=5038 from=192.168.4.5
May 17 14:17:00 client-18 rshd[5039]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 43f: pdirops: unlink and rename \(src\) ================================================ 14:17:00 \(1305667020\)");echo XXRETCODE:$?'
May 17 14:17:00 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 43f: pdirops: unlink and rename (src) ================================================ 14:17:00 (1305667020)
May 17 14:17:00 client-18 xinetd[3390]: EXIT: shell status=0 pid=5038 duration=0(sec)
May 17 14:17:10 client-18 xinetd[3390]: START: shell pid=5073 from=192.168.4.5
May 17 14:17:10 client-18 rshd[5074]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:17:10 client-18 xinetd[3390]: EXIT: shell status=0 pid=5073 duration=0(sec)
May 17 14:17:10 client-18 xinetd[3390]: START: shell pid=5109 from=192.168.4.5
May 17 14:17:10 client-18 rshd[5110]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:17:10 client-18 xinetd[3390]: EXIT: shell status=0 pid=5109 duration=0(sec)
May 17 14:17:10 client-18 xinetd[3390]: START: shell pid=5146 from=192.168.4.5
May 17 14:17:10 client-18 rshd[5147]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 43g: pdirops: unlink vs getattr ====================================================== 14:17:10 \(1305667030\)");echo XXRETCODE:$?'
May 17 14:17:11 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 43g: pdirops: unlink vs getattr ====================================================== 14:17:10 (1305667030)
May 17 14:17:11 client-18 xinetd[3390]: EXIT: shell status=0 pid=5146 duration=1(sec)
May 17 14:17:21 client-18 xinetd[3390]: START: shell pid=5181 from=192.168.4.5
May 17 14:17:21 client-18 rshd[5182]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:17:21 client-18 xinetd[3390]: EXIT: shell status=0 pid=5181 duration=0(sec)
May 17 14:17:21 client-18 xinetd[3390]: START: shell pid=5217 from=192.168.4.5
May 17 14:17:21 client-18 rshd[5218]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:17:21 client-18 xinetd[3390]: EXIT: shell status=0 pid=5217 duration=0(sec)
May 17 14:17:21 client-18 xinetd[3390]: START: shell pid=5254 from=192.168.4.5
May 17 14:17:21 client-18 rshd[5255]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 43h: pdirops: unlink vs readdir ====================================================== 14:17:21 \(1305667041\)");echo XXRETCODE:$?'
May 17 14:17:21 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 43h: pdirops: unlink vs readdir ====================================================== 14:17:21 (1305667041)
May 17 14:17:21 client-18 xinetd[3390]: EXIT: shell status=0 pid=5254 duration=0(sec)
May 17 14:17:31 client-18 xinetd[3390]: START: shell pid=5293 from=192.168.4.5
May 17 14:17:31 client-18 rshd[5294]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:17:31 client-18 xinetd[3390]: EXIT: shell status=0 pid=5293 duration=0(sec)
May 17 14:17:31 client-18 xinetd[3390]: START: shell pid=5329 from=192.168.4.5
May 17 14:17:31 client-18 rshd[5330]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:17:32 client-18 xinetd[3390]: EXIT: shell status=0 pid=5329 duration=1(sec)
May 17 14:17:32 client-18 xinetd[3390]: START: shell pid=5366 from=192.168.4.5
May 17 14:17:32 client-18 rshd[5367]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 44a: pdirops: rename tgt vs mkdir ==================================================== 14:17:32 \(1305667052\)");echo XXRETCODE:$?'
May 17 14:17:32 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 44a: pdirops: rename tgt vs mkdir ==================================================== 14:17:32 (1305667052)
May 17 14:17:32 client-18 xinetd[3390]: EXIT: shell status=0 pid=5366 duration=0(sec)
May 17 14:17:42 client-18 xinetd[3390]: START: shell pid=5401 from=192.168.4.5
May 17 14:17:42 client-18 rshd[5402]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:17:42 client-18 xinetd[3390]: EXIT: shell status=0 pid=5401 duration=0(sec)
May 17 14:17:42 client-18 xinetd[3390]: START: shell pid=5437 from=192.168.4.5
May 17 14:17:42 client-18 rshd[5438]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:17:42 client-18 xinetd[3390]: EXIT: shell status=0 pid=5437 duration=0(sec)
May 17 14:17:42 client-18 xinetd[3390]: START: shell pid=5474 from=192.168.4.5
May 17 14:17:42 client-18 rshd[5475]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 44b: pdirops: rename tgt vs create =================================================== 14:17:42 \(1305667062\)");echo XXRETCODE:$?'
May 17 14:17:42 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 44b: pdirops: rename tgt vs create =================================================== 14:17:42 (1305667062)
May 17 14:17:42 client-18 xinetd[3390]: EXIT: shell status=0 pid=5474 duration=0(sec)
May 17 14:17:52 client-18 xinetd[3390]: START: shell pid=5509 from=192.168.4.5
May 17 14:17:53 client-18 rshd[5510]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:17:53 client-18 xinetd[3390]: EXIT: shell status=0 pid=5509 duration=1(sec)
May 17 14:17:53 client-18 xinetd[3390]: START: shell pid=5545 from=192.168.4.5
May 17 14:17:53 client-18 rshd[5546]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:17:53 client-18 xinetd[3390]: EXIT: shell status=0 pid=5545 duration=0(sec)
May 17 14:17:53 client-18 xinetd[3390]: START: shell pid=5582 from=192.168.4.5
May 17 14:17:53 client-18 rshd[5583]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 44c: pdirops: rename tgt vs link ===================================================== 14:17:53 \(1305667073\)");echo XXRETCODE:$?'
May 17 14:17:53 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 44c: pdirops: rename tgt vs link ===================================================== 14:17:53 (1305667073)
May 17 14:17:53 client-18 xinetd[3390]: EXIT: shell status=0 pid=5582 duration=0(sec)
May 17 14:18:03 client-18 xinetd[3390]: START: shell pid=5617 from=192.168.4.5
May 17 14:18:03 client-18 rshd[5618]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:18:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=5617 duration=0(sec)
May 17 14:18:03 client-18 xinetd[3390]: START: shell pid=5653 from=192.168.4.5
May 17 14:18:03 client-18 rshd[5654]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:18:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=5653 duration=0(sec)
May 17 14:18:03 client-18 xinetd[3390]: START: shell pid=5690 from=192.168.4.5
May 17 14:18:03 client-18 rshd[5691]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 44d: pdirops: rename tgt vs unlink =================================================== 14:18:03 \(1305667083\)");echo XXRETCODE:$?'
May 17 14:18:04 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 44d: pdirops: rename tgt vs unlink =================================================== 14:18:03 (1305667083)
May 17 14:18:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=5690 duration=1(sec)
May 17 14:18:14 client-18 xinetd[3390]: START: shell pid=5725 from=192.168.4.5
May 17 14:18:14 client-18 rshd[5726]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:18:14 client-18 xinetd[3390]: EXIT: shell status=0 pid=5725 duration=0(sec)
May 17 14:18:14 client-18 xinetd[3390]: START: shell pid=5761 from=192.168.4.5
May 17 14:18:14 client-18 rshd[5762]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:18:14 client-18 xinetd[3390]: EXIT: shell status=0 pid=5761 duration=0(sec)
May 17 14:18:14 client-18 xinetd[3390]: START: shell pid=5798 from=192.168.4.5
May 17 14:18:14 client-18 rshd[5799]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 44e: pdirops: rename tgt and rename \(tgt\) ============================================ 14:18:14 \(1305667094\)");echo XXRETCODE:$?'
May 17 14:18:14 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 44e: pdirops: rename tgt and rename (tgt) ============================================ 14:18:14 (1305667094)
May 17 14:18:14 client-18 xinetd[3390]: EXIT: shell status=0 pid=5798 duration=0(sec)
May 17 14:18:24 client-18 xinetd[3390]: START: shell pid=5833 from=192.168.4.5
May 17 14:18:24 client-18 rshd[5834]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:18:24 client-18 xinetd[3390]: EXIT: shell status=0 pid=5833 duration=0(sec)
May 17 14:18:25 client-18 xinetd[3390]: START: shell pid=5869 from=192.168.4.5
May 17 14:18:25 client-18 rshd[5870]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:18:25 client-18 xinetd[3390]: EXIT: shell status=0 pid=5869 duration=0(sec)
May 17 14:18:25 client-18 xinetd[3390]: START: shell pid=5906 from=192.168.4.5
May 17 14:18:25 client-18 rshd[5907]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 44f: pdirops: rename tgt and rename \(src\) ============================================ 14:18:25 \(1305667105\)");echo XXRETCODE:$?'
May 17 14:18:25 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 44f: pdirops: rename tgt and rename (src) ============================================ 14:18:25 (1305667105)
May 17 14:18:25 client-18 xinetd[3390]: EXIT: shell status=0 pid=5906 duration=0(sec)
May 17 14:18:35 client-18 xinetd[3390]: START: shell pid=5941 from=192.168.4.5
May 17 14:18:35 client-18 rshd[5942]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:18:35 client-18 xinetd[3390]: EXIT: shell status=0 pid=5941 duration=0(sec)
May 17 14:18:35 client-18 xinetd[3390]: START: shell pid=5977 from=192.168.4.5
May 17 14:18:35 client-18 rshd[5978]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:18:35 client-18 xinetd[3390]: EXIT: shell status=0 pid=5977 duration=0(sec)
May 17 14:18:35 client-18 xinetd[3390]: START: shell pid=6014 from=192.168.4.5
May 17 14:18:35 client-18 rshd[6015]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 44g: pdirops: rename tgt vs getattr ================================================== 14:18:35 \(1305667115\)");echo XXRETCODE:$?'
May 17 14:18:35 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 44g: pdirops: rename tgt vs getattr ================================================== 14:18:35 (1305667115)
May 17 14:18:35 client-18 xinetd[3390]: EXIT: shell status=0 pid=6014 duration=0(sec)
May 17 14:18:46 client-18 xinetd[3390]: START: shell pid=6053 from=192.168.4.5
May 17 14:18:46 client-18 rshd[6054]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:18:46 client-18 xinetd[3390]: EXIT: shell status=0 pid=6053 duration=0(sec)
May 17 14:18:46 client-18 xinetd[3390]: START: shell pid=6089 from=192.168.4.5
May 17 14:18:46 client-18 rshd[6090]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:18:46 client-18 xinetd[3390]: EXIT: shell status=0 pid=6089 duration=0(sec)
May 17 14:18:46 client-18 xinetd[3390]: START: shell pid=6126 from=192.168.4.5
May 17 14:18:46 client-18 rshd[6127]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 44h: pdirops: rename tgt vs readdir ================================================== 14:18:46 \(1305667126\)");echo XXRETCODE:$?'
May 17 14:18:46 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 44h: pdirops: rename tgt vs readdir ================================================== 14:18:46 (1305667126)
May 17 14:18:46 client-18 xinetd[3390]: EXIT: shell status=0 pid=6126 duration=0(sec)
May 17 14:18:56 client-18 xinetd[3390]: START: shell pid=6161 from=192.168.4.5
May 17 14:18:56 client-18 rshd[6162]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:18:56 client-18 xinetd[3390]: EXIT: shell status=0 pid=6161 duration=0(sec)
May 17 14:18:56 client-18 xinetd[3390]: START: shell pid=6197 from=192.168.4.5
May 17 14:18:56 client-18 rshd[6198]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:18:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=6197 duration=1(sec)
May 17 14:18:57 client-18 xinetd[3390]: START: shell pid=6234 from=192.168.4.5
May 17 14:18:57 client-18 rshd[6235]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 45a: pdirops: rename src vs mkdir ==================================================== 14:18:57 \(1305667137\)");echo XXRETCODE:$?'
May 17 14:18:57 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 45a: pdirops: rename src vs mkdir ==================================================== 14:18:57 (1305667137)
May 17 14:18:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=6234 duration=0(sec)
May 17 14:19:07 client-18 xinetd[3390]: START: shell pid=6269 from=192.168.4.5
May 17 14:19:07 client-18 rshd[6270]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:19:07 client-18 xinetd[3390]: EXIT: shell status=0 pid=6269 duration=0(sec)
May 17 14:19:07 client-18 xinetd[3390]: START: shell pid=6305 from=192.168.4.5
May 17 14:19:07 client-18 rshd[6306]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:19:07 client-18 xinetd[3390]: EXIT: shell status=0 pid=6305 duration=0(sec)
May 17 14:19:07 client-18 xinetd[3390]: START: shell pid=6342 from=192.168.4.5
May 17 14:19:07 client-18 rshd[6343]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 45b: pdirops: rename src vs create =================================================== 14:19:07 \(1305667147\)");echo XXRETCODE:$?'
May 17 14:19:07 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 45b: pdirops: rename src vs create =================================================== 14:19:07 (1305667147)
May 17 14:19:07 client-18 xinetd[3390]: EXIT: shell status=0 pid=6342 duration=0(sec)
May 17 14:19:17 client-18 xinetd[3390]: START: shell pid=6377 from=192.168.4.5
May 17 14:19:18 client-18 rshd[6378]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:19:18 client-18 xinetd[3390]: EXIT: shell status=0 pid=6377 duration=1(sec)
May 17 14:19:18 client-18 xinetd[3390]: START: shell pid=6413 from=192.168.4.5
May 17 14:19:18 client-18 rshd[6414]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:19:18 client-18 xinetd[3390]: EXIT: shell status=0 pid=6413 duration=0(sec)
May 17 14:19:18 client-18 xinetd[3390]: START: shell pid=6450 from=192.168.4.5
May 17 14:19:18 client-18 rshd[6451]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 45c: pdirops: rename src vs link ===================================================== 14:19:18 \(1305667158\)");echo XXRETCODE:$?'
May 17 14:19:18 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 45c: pdirops: rename src vs link ===================================================== 14:19:18 (1305667158)
May 17 14:19:18 client-18 xinetd[3390]: EXIT: shell status=0 pid=6450 duration=0(sec)
May 17 14:19:28 client-18 xinetd[3390]: START: shell pid=6485 from=192.168.4.5
May 17 14:19:28 client-18 rshd[6486]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:19:28 client-18 xinetd[3390]: EXIT: shell status=0 pid=6485 duration=0(sec)
May 17 14:19:28 client-18 xinetd[3390]: START: shell pid=6521 from=192.168.4.5
May 17 14:19:28 client-18 rshd[6522]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:19:28 client-18 xinetd[3390]: EXIT: shell status=0 pid=6521 duration=0(sec)
May 17 14:19:28 client-18 xinetd[3390]: START: shell pid=6558 from=192.168.4.5
May 17 14:19:28 client-18 rshd[6559]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 45d: pdirops: rename src vs unlink =================================================== 14:19:28 \(1305667168\)");echo XXRETCODE:$?'
May 17 14:19:29 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 45d: pdirops: rename src vs unlink =================================================== 14:19:28 (1305667168)
May 17 14:19:29 client-18 xinetd[3390]: EXIT: shell status=0 pid=6558 duration=1(sec)
May 17 14:19:39 client-18 xinetd[3390]: START: shell pid=6593 from=192.168.4.5
May 17 14:19:39 client-18 rshd[6594]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:19:39 client-18 xinetd[3390]: EXIT: shell status=0 pid=6593 duration=0(sec)
May 17 14:19:39 client-18 xinetd[3390]: START: shell pid=6629 from=192.168.4.5
May 17 14:19:39 client-18 rshd[6630]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:19:39 client-18 xinetd[3390]: EXIT: shell status=0 pid=6629 duration=0(sec)
May 17 14:19:39 client-18 xinetd[3390]: START: shell pid=6666 from=192.168.4.5
May 17 14:19:39 client-18 rshd[6667]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 45e: pdirops: rename src and rename \(tgt\) ============================================ 14:19:39 \(1305667179\)");echo XXRETCODE:$?'
May 17 14:19:39 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 45e: pdirops: rename src and rename (tgt) ============================================ 14:19:39 (1305667179)
May 17 14:19:39 client-18 xinetd[3390]: EXIT: shell status=0 pid=6666 duration=0(sec)
May 17 14:19:49 client-18 xinetd[3390]: START: shell pid=6701 from=192.168.4.5
May 17 14:19:49 client-18 rshd[6702]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:19:49 client-18 xinetd[3390]: EXIT: shell status=0 pid=6701 duration=0(sec)
May 17 14:19:49 client-18 xinetd[3390]: START: shell pid=6737 from=192.168.4.5
May 17 14:19:50 client-18 rshd[6738]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:19:50 client-18 xinetd[3390]: EXIT: shell status=0 pid=6737 duration=1(sec)
May 17 14:19:50 client-18 xinetd[3390]: START: shell pid=6774 from=192.168.4.5
May 17 14:19:50 client-18 rshd[6775]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 45f: pdirops: rename src and rename \(src\) ============================================ 14:19:50 \(1305667190\)");echo XXRETCODE:$?'
May 17 14:19:50 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 45f: pdirops: rename src and rename (src) ============================================ 14:19:50 (1305667190)
May 17 14:19:50 client-18 xinetd[3390]: EXIT: shell status=0 pid=6774 duration=0(sec)
May 17 14:20:00 client-18 xinetd[3390]: START: shell pid=6813 from=192.168.4.5
May 17 14:20:00 client-18 rshd[6814]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:20:00 client-18 xinetd[3390]: EXIT: shell status=0 pid=6813 duration=0(sec)
May 17 14:20:00 client-18 xinetd[3390]: START: shell pid=6849 from=192.168.4.5
May 17 14:20:00 client-18 rshd[6850]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:20:00 client-18 xinetd[3390]: EXIT: shell status=0 pid=6849 duration=0(sec)
May 17 14:20:00 client-18 xinetd[3390]: START: shell pid=6886 from=192.168.4.5
May 17 14:20:00 client-18 rshd[6887]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 45g: pdirops: rename src vs getattr ================================================== 14:20:00 \(1305667200\)");echo XXRETCODE:$?'
May 17 14:20:00 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 45g: pdirops: rename src vs getattr ================================================== 14:20:00 (1305667200)
May 17 14:20:00 client-18 xinetd[3390]: EXIT: shell status=0 pid=6886 duration=0(sec)
May 17 14:20:11 client-18 xinetd[3390]: START: shell pid=6921 from=192.168.4.5
May 17 14:20:11 client-18 rshd[6922]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:20:11 client-18 xinetd[3390]: EXIT: shell status=0 pid=6921 duration=0(sec)
May 17 14:20:11 client-18 xinetd[3390]: START: shell pid=6957 from=192.168.4.5
May 17 14:20:11 client-18 rshd[6958]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:20:11 client-18 xinetd[3390]: EXIT: shell status=0 pid=6957 duration=0(sec)
May 17 14:20:11 client-18 xinetd[3390]: START: shell pid=6994 from=192.168.4.5
May 17 14:20:11 client-18 rshd[6995]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 45h: pdirops: unlink vs readdir ====================================================== 14:20:11 \(1305667211\)");echo XXRETCODE:$?'
May 17 14:20:11 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 45h: pdirops: unlink vs readdir ====================================================== 14:20:11 (1305667211)
May 17 14:20:11 client-18 xinetd[3390]: EXIT: shell status=0 pid=6994 duration=0(sec)
May 17 14:20:21 client-18 xinetd[3390]: START: shell pid=7029 from=192.168.4.5
May 17 14:20:21 client-18 rshd[7030]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:20:21 client-18 xinetd[3390]: EXIT: shell status=0 pid=7029 duration=0(sec)
May 17 14:20:21 client-18 xinetd[3390]: START: shell pid=7065 from=192.168.4.5
May 17 14:20:21 client-18 rshd[7066]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:20:21 client-18 xinetd[3390]: EXIT: shell status=0 pid=7065 duration=0(sec)
May 17 14:20:22 client-18 xinetd[3390]: START: shell pid=7102 from=192.168.4.5
May 17 14:20:22 client-18 rshd[7103]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 46a: pdirops: link vs mkdir ========================================================== 14:20:21 \(1305667221\)");echo XXRETCODE:$?'
May 17 14:20:22 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 46a: pdirops: link vs mkdir ========================================================== 14:20:21 (1305667221)
May 17 14:20:22 client-18 xinetd[3390]: EXIT: shell status=0 pid=7102 duration=0(sec)
May 17 14:20:32 client-18 xinetd[3390]: START: shell pid=7137 from=192.168.4.5
May 17 14:20:32 client-18 rshd[7138]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:20:32 client-18 xinetd[3390]: EXIT: shell status=0 pid=7137 duration=0(sec)
May 17 14:20:32 client-18 xinetd[3390]: START: shell pid=7173 from=192.168.4.5
May 17 14:20:32 client-18 rshd[7174]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:20:32 client-18 xinetd[3390]: EXIT: shell status=0 pid=7173 duration=0(sec)
May 17 14:20:32 client-18 xinetd[3390]: START: shell pid=7210 from=192.168.4.5
May 17 14:20:32 client-18 rshd[7211]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 46b: pdirops: link vs create ========================================================= 14:20:32 \(1305667232\)");echo XXRETCODE:$?'
May 17 14:20:32 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 46b: pdirops: link vs create ========================================================= 14:20:32 (1305667232)
May 17 14:20:32 client-18 xinetd[3390]: EXIT: shell status=0 pid=7210 duration=0(sec)
May 17 14:20:42 client-18 xinetd[3390]: START: shell pid=7245 from=192.168.4.5
May 17 14:20:42 client-18 rshd[7246]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:20:43 client-18 xinetd[3390]: EXIT: shell status=0 pid=7245 duration=1(sec)
May 17 14:20:43 client-18 xinetd[3390]: START: shell pid=7281 from=192.168.4.5
May 17 14:20:43 client-18 rshd[7282]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:20:43 client-18 xinetd[3390]: EXIT: shell status=0 pid=7281 duration=0(sec)
May 17 14:20:43 client-18 xinetd[3390]: START: shell pid=7318 from=192.168.4.5
May 17 14:20:43 client-18 rshd[7319]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 46c: pdirops: link vs link =========================================================== 14:20:43 \(1305667243\)");echo XXRETCODE:$?'
May 17 14:20:43 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 46c: pdirops: link vs link =========================================================== 14:20:43 (1305667243)
May 17 14:20:43 client-18 xinetd[3390]: EXIT: shell status=0 pid=7318 duration=0(sec)
May 17 14:20:53 client-18 xinetd[3390]: START: shell pid=7353 from=192.168.4.5
May 17 14:20:53 client-18 rshd[7354]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:20:53 client-18 xinetd[3390]: EXIT: shell status=0 pid=7353 duration=0(sec)
May 17 14:20:53 client-18 xinetd[3390]: START: shell pid=7389 from=192.168.4.5
May 17 14:20:53 client-18 rshd[7390]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:20:53 client-18 xinetd[3390]: EXIT: shell status=0 pid=7389 duration=0(sec)
May 17 14:20:53 client-18 xinetd[3390]: START: shell pid=7426 from=192.168.4.5
May 17 14:20:53 client-18 rshd[7427]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 46d: pdirops: link vs unlink ========================================================= 14:20:53 \(1305667253\)");echo XXRETCODE:$?'
May 17 14:20:53 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 46d: pdirops: link vs unlink ========================================================= 14:20:53 (1305667253)
May 17 14:20:53 client-18 xinetd[3390]: EXIT: shell status=0 pid=7426 duration=0(sec)
May 17 14:21:04 client-18 xinetd[3390]: START: shell pid=7461 from=192.168.4.5
May 17 14:21:04 client-18 rshd[7462]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:21:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=7461 duration=0(sec)
May 17 14:21:04 client-18 xinetd[3390]: START: shell pid=7497 from=192.168.4.5
May 17 14:21:04 client-18 rshd[7498]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:21:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=7497 duration=0(sec)
May 17 14:21:04 client-18 xinetd[3390]: START: shell pid=7534 from=192.168.4.5
May 17 14:21:04 client-18 rshd[7535]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 46e: pdirops: link and rename \(tgt\) ================================================== 14:21:04 \(1305667264\)");echo XXRETCODE:$?'
May 17 14:21:04 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 46e: pdirops: link and rename (tgt) ================================================== 14:21:04 (1305667264)
May 17 14:21:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=7534 duration=0(sec)
May 17 14:21:14 client-18 xinetd[3390]: START: shell pid=7573 from=192.168.4.5
May 17 14:21:14 client-18 rshd[7574]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:21:14 client-18 xinetd[3390]: EXIT: shell status=0 pid=7573 duration=0(sec)
May 17 14:21:14 client-18 xinetd[3390]: START: shell pid=7609 from=192.168.4.5
May 17 14:21:14 client-18 rshd[7610]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:21:15 client-18 xinetd[3390]: EXIT: shell status=0 pid=7609 duration=1(sec)
May 17 14:21:15 client-18 xinetd[3390]: START: shell pid=7646 from=192.168.4.5
May 17 14:21:15 client-18 rshd[7647]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 46f: pdirops: link and rename \(src\) ================================================== 14:21:15 \(1305667275\)");echo XXRETCODE:$?'
May 17 14:21:15 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 46f: pdirops: link and rename (src) ================================================== 14:21:15 (1305667275)
May 17 14:21:15 client-18 xinetd[3390]: EXIT: shell status=0 pid=7646 duration=0(sec)
May 17 14:21:25 client-18 xinetd[3390]: START: shell pid=7681 from=192.168.4.5
May 17 14:21:25 client-18 rshd[7682]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:21:25 client-18 xinetd[3390]: EXIT: shell status=0 pid=7681 duration=0(sec)
May 17 14:21:25 client-18 xinetd[3390]: START: shell pid=7717 from=192.168.4.5
May 17 14:21:25 client-18 rshd[7718]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:21:25 client-18 xinetd[3390]: EXIT: shell status=0 pid=7717 duration=0(sec)
May 17 14:21:25 client-18 xinetd[3390]: START: shell pid=7754 from=192.168.4.5
May 17 14:21:25 client-18 rshd[7755]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 46g: pdirops: link vs getattr ======================================================== 14:21:25 \(1305667285\)");echo XXRETCODE:$?'
May 17 14:21:25 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 46g: pdirops: link vs getattr ======================================================== 14:21:25 (1305667285)
May 17 14:21:25 client-18 xinetd[3390]: EXIT: shell status=0 pid=7754 duration=0(sec)
May 17 14:21:36 client-18 xinetd[3390]: START: shell pid=7789 from=192.168.4.5
May 17 14:21:36 client-18 rshd[7790]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:21:36 client-18 xinetd[3390]: EXIT: shell status=0 pid=7789 duration=0(sec)
May 17 14:21:36 client-18 xinetd[3390]: START: shell pid=7825 from=192.168.4.5
May 17 14:21:36 client-18 rshd[7826]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:21:36 client-18 xinetd[3390]: EXIT: shell status=0 pid=7825 duration=0(sec)
May 17 14:21:36 client-18 xinetd[3390]: START: shell pid=7862 from=192.168.4.5
May 17 14:21:36 client-18 rshd[7863]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 46h: pdirops: link vs readdir ======================================================== 14:21:36 \(1305667296\)");echo XXRETCODE:$?'
May 17 14:21:36 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 46h: pdirops: link vs readdir ======================================================== 14:21:36 (1305667296)
May 17 14:21:36 client-18 xinetd[3390]: EXIT: shell status=0 pid=7862 duration=0(sec)
May 17 14:21:46 client-18 xinetd[3390]: START: shell pid=7897 from=192.168.4.5
May 17 14:21:46 client-18 rshd[7898]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:21:46 client-18 xinetd[3390]: EXIT: shell status=0 pid=7897 duration=0(sec)
May 17 14:21:46 client-18 xinetd[3390]: START: shell pid=7933 from=192.168.4.5
May 17 14:21:46 client-18 rshd[7934]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:21:46 client-18 xinetd[3390]: EXIT: shell status=0 pid=7933 duration=0(sec)
May 17 14:21:46 client-18 xinetd[3390]: START: shell pid=7970 from=192.168.4.5
May 17 14:21:47 client-18 rshd[7971]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ================================================ 14:21:46 \(1305667306\)");echo XXRETCODE:$?'
May 17 14:21:47 client-18 kernel: Lustre: DEBUG MARKER: == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ================================================ 14:21:46 (1305667306)
May 17 14:21:47 client-18 xinetd[3390]: EXIT: shell status=0 pid=7970 duration=1(sec)
May 17 14:21:50 client-18 xinetd[3390]: START: shell pid=8005 from=192.168.4.5
May 17 14:21:50 client-18 rshd[8006]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:21:50 client-18 xinetd[3390]: EXIT: shell status=0 pid=8005 duration=0(sec)
May 17 14:21:50 client-18 xinetd[3390]: START: shell pid=8041 from=192.168.4.5
May 17 14:21:50 client-18 rshd[8042]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:21:50 client-18 xinetd[3390]: EXIT: shell status=0 pid=8041 duration=0(sec)
May 17 14:21:50 client-18 xinetd[3390]: START: shell pid=8078 from=192.168.4.5
May 17 14:21:50 client-18 rshd[8079]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark cleanup: ======================================================");echo XXRETCODE:$?'
May 17 14:21:50 client-18 kernel: Lustre: DEBUG MARKER: cleanup: ======================================================
May 17 14:21:50 client-18 xinetd[3390]: EXIT: shell status=0 pid=8078 duration=0(sec)
May 17 14:21:51 client-18 xinetd[3390]: START: shell pid=8113 from=192.168.4.5
May 17 14:21:51 client-18 rshd[8114]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanityn sanityn.sh test complete, duration 1135 sec =============================================== 14:21:51 \(1305667311\)");echo XXRETCODE:$?'
May 17 14:21:51 client-18 kernel: Lustre: DEBUG MARKER: == sanityn sanityn.sh test complete, duration 1135 sec =============================================== 14:21:51 (1305667311)
May 17 14:21:51 client-18 xinetd[3390]: EXIT: shell status=0 pid=8113 duration=0(sec)
May 17 14:21:51 client-18 xinetd[3390]: START: shell pid=8148 from=192.168.4.5
May 17 14:21:51 client-18 rshd[8149]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:; lsof -t /mnt/lustre2 || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre2 | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount  /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre2 is still busy, wait one second && sleep 1;     umount  /mnt/lustre2; fi fi");echo XXRETCODE:$?'
May 17 14:21:51 client-18 kernel: LustreError: 8190:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway
May 17 14:21:51 client-18 kernel: LustreError: 8190:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Skipped 1 previous similar message
May 17 14:21:51 client-18 kernel: LustreError: 8190:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108
May 17 14:21:51 client-18 kernel: LustreError: 8190:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) Skipped 1 previous similar message
May 17 14:21:51 client-18 kernel: Lustre: client ffff810304e9a400 umount complete
May 17 14:21:51 client-18 xinetd[3390]: EXIT: shell status=0 pid=8148 duration=0(sec)
May 17 14:21:51 client-18 xinetd[3390]: START: shell pid=8192 from=192.168.4.5
May 17 14:21:51 client-18 rshd[8193]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark -----============= acceptance-small: lfsck ============----- Tue May 17 14:21:51 PDT 2011");echo XXRETCODE:$?'
May 17 14:21:51 client-18 kernel: Lustre: DEBUG MARKER: -----============= acceptance-small: lfsck ============----- Tue May 17 14:21:51 PDT 2011
May 17 14:21:51 client-18 xinetd[3390]: EXIT: shell status=0 pid=8192 duration=0(sec)
May 17 14:21:51 client-18 xinetd[3390]: START: shell pid=8227 from=192.168.4.5
May 17 14:21:51 client-18 rshd[8228]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/tests:/usr/lib64/openmpi/1.4-gcc/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/usr/lib64/lustre/utils:/usr/lib64/lustre/utils/gss:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey::/usr/lib64/lustre/tests/mpi NAME=sarah sh rpc.sh check_config_client /mnt/lustre ");echo XXRETCODE:$?'
May 17 14:21:52 client-18 xinetd[3390]: EXIT: shell status=0 pid=8227 duration=1(sec)
May 17 14:21:53 client-18 xinetd[3390]: START: shell pid=8321 from=192.168.4.5
May 17 14:21:53 client-18 rshd[8322]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); mpts=\$(mount | grep -w -c /mnt/lustre); if [ \$running -ne \$mpts ]; then     echo \$(hostname) env are INSANE!;     exit 1; fi");echo XXRETCODE:$?'
May 17 14:21:53 client-18 xinetd[3390]: EXIT: shell status=0 pid=8321 duration=0(sec)
May 17 14:21:53 client-18 xinetd[3390]: START: shell pid=8361 from=192.168.4.5
May 17 14:21:53 client-18 rshd[8362]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); mpts=\$(mount | grep -w -c /mnt/lustre2); if [ \$running -ne \$mpts ]; then     echo \$(hostname) env are INSANE!;     exit 1; fi");echo XXRETCODE:$?'
May 17 14:21:53 client-18 xinetd[3390]: EXIT: shell status=0 pid=8361 duration=0(sec)
May 17 14:21:55 client-18 xinetd[3390]: START: shell pid=8401 from=192.168.4.5
May 17 14:21:55 client-18 rshd[8402]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark Using TIMEOUT=20");echo XXRETCODE:$?'
May 17 14:21:55 client-18 kernel: Lustre: DEBUG MARKER: Using TIMEOUT=20
May 17 14:21:55 client-18 xinetd[3390]: EXIT: shell status=0 pid=8401 duration=0(sec)
May 17 14:21:55 client-18 xinetd[3390]: START: shell pid=8436 from=192.168.4.5
May 17 14:21:55 client-18 rshd[8437]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param debug=\"0x33f0406\";             lctl set_param subsystem_debug=\"0xffb7e3ff\";             lctl set_param debug_mb=32;             sync");echo XXRETCODE:$?'
May 17 14:21:55 client-18 kernel: Lustre: 8471:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release.
May 17 14:21:55 client-18 kernel: Lustre: 8471:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message
May 17 14:21:55 client-18 xinetd[3390]: EXIT: shell status=0 pid=8436 duration=0(sec)
May 17 14:21:55 client-18 xinetd[3390]: START: shell pid=8474 from=192.168.4.5
May 17 14:21:55 client-18 rshd[8475]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "cat /proc/mounts");echo XXRETCODE:$?'
May 17 14:21:55 client-18 xinetd[3390]: EXIT: shell status=0 pid=8474 duration=0(sec)
May 17 14:21:56 client-18 xinetd[3390]: START: shell pid=8509 from=192.168.4.5
May 17 14:21:56 client-18 rshd[8510]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl get_param -n osc.*OST*-osc-[^M][^D][^T]*.srpc_info");echo XXRETCODE:$?'
May 17 14:21:56 client-18 xinetd[3390]: EXIT: shell status=0 pid=8509 duration=0(sec)
May 17 14:21:56 client-18 xinetd[3390]: START: shell pid=8544 from=192.168.4.5
May 17 14:21:56 client-18 rshd[8545]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl get_param -n mdc.*-MDT*-mdc-*.srpc_info");echo XXRETCODE:$?'
May 17 14:21:56 client-18 xinetd[3390]: EXIT: shell status=0 pid=8544 duration=0(sec)
May 17 14:21:57 client-18 xinetd[3390]: START: shell pid=8579 from=192.168.4.5
May 17 14:21:57 client-18 rshd[8580]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "sync; sleep 2; sync");echo XXRETCODE:$?'
May 17 14:21:59 client-18 xinetd[3390]: EXIT: shell status=0 pid=8579 duration=2(sec)
May 17 14:21:59 client-18 xinetd[3390]: START: shell pid=8617 from=192.168.4.5
May 17 14:21:59 client-18 rshd[8618]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "ls /scratch/tmp.BAJFvFh912");echo XXRETCODE:$?'
May 17 14:21:59 client-18 kernel: FS-Cache: Loaded
May 17 14:21:59 client-18 xinetd[3390]: EXIT: shell status=0 pid=8617 duration=0(sec)
May 17 14:22:07 client-18 xinetd[3390]: START: shell pid=8674 from=192.168.4.5
May 17 14:22:07 client-18 rshd[8675]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == lfsck lfsck.sh test complete, duration 16 sec ===================================================== 14:22:07 \(1305667327\)");echo XXRETCODE:$?'
May 17 14:22:07 client-18 kernel: Lustre: DEBUG MARKER: == lfsck lfsck.sh test complete, duration 16 sec ===================================================== 14:22:07 (1305667327)
May 17 14:22:07 client-18 xinetd[3390]: EXIT: shell status=0 pid=8674 duration=0(sec)
May 17 14:22:07 client-18 xinetd[3390]: START: shell pid=8709 from=192.168.4.5
May 17 14:22:07 client-18 rshd[8710]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark -----============= acceptance-small: runracer ============----- Tue May 17 14:22:07 PDT 2011");echo XXRETCODE:$?'
May 17 14:22:08 client-18 kernel: Lustre: DEBUG MARKER: -----============= acceptance-small: runracer ============----- Tue May 17 14:22:07 PDT 2011
May 17 14:22:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=8709 duration=1(sec)
May 17 14:22:08 client-18 xinetd[3390]: START: shell pid=8744 from=192.168.4.5
May 17 14:22:08 client-18 rshd[8745]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark excepting tests: ");echo XXRETCODE:$?'
May 17 14:22:08 client-18 kernel: Lustre: DEBUG MARKER: excepting tests:
May 17 14:22:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=8744 duration=0(sec)
May 17 14:22:08 client-18 xinetd[3390]: START: shell pid=8779 from=192.168.4.5
May 17 14:22:08 client-18 rshd[8780]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/tests:/usr/lib64/openmpi/1.4-gcc/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/usr/lib64/lustre/utils:/usr/lib64/lustre/utils/gss:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey::/usr/lib64/lustre/tests/mpi NAME=sarah sh rpc.sh check_config_client /mnt/lustre ");echo XXRETCODE:$?'
May 17 14:22:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=8779 duration=0(sec)
May 17 14:22:09 client-18 xinetd[3390]: START: shell pid=8873 from=192.168.4.5
May 17 14:22:09 client-18 rshd[8874]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); mpts=\$(mount | grep -w -c /mnt/lustre); if [ \$running -ne \$mpts ]; then     echo \$(hostname) env are INSANE!;     exit 1; fi");echo XXRETCODE:$?'
May 17 14:22:09 client-18 xinetd[3390]: EXIT: shell status=0 pid=8873 duration=0(sec)
May 17 14:22:09 client-18 xinetd[3390]: START: shell pid=8913 from=192.168.4.5
May 17 14:22:09 client-18 rshd[8914]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); mpts=\$(mount | grep -w -c /mnt/lustre2); if [ \$running -ne \$mpts ]; then     echo \$(hostname) env are INSANE!;     exit 1; fi");echo XXRETCODE:$?'
May 17 14:22:09 client-18 xinetd[3390]: EXIT: shell status=0 pid=8913 duration=0(sec)
May 17 14:22:11 client-18 xinetd[3390]: START: shell pid=8953 from=192.168.4.5
May 17 14:22:11 client-18 rshd[8954]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark Using TIMEOUT=20");echo XXRETCODE:$?'
May 17 14:22:11 client-18 kernel: Lustre: DEBUG MARKER: Using TIMEOUT=20
May 17 14:22:11 client-18 xinetd[3390]: EXIT: shell status=0 pid=8953 duration=0(sec)
May 17 14:22:11 client-18 xinetd[3390]: START: shell pid=8988 from=192.168.4.5
May 17 14:22:11 client-18 rshd[8989]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param debug=\"0x33f0406\";             lctl set_param subsystem_debug=\"0xffb7e3ff\";             lctl set_param debug_mb=32;             sync");echo XXRETCODE:$?'
May 17 14:22:11 client-18 kernel: Lustre: 9023:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release.
May 17 14:22:11 client-18 kernel: Lustre: 9023:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message
May 17 14:22:11 client-18 xinetd[3390]: EXIT: shell status=0 pid=8988 duration=0(sec)
May 17 14:22:11 client-18 xinetd[3390]: START: shell pid=9026 from=192.168.4.5
May 17 14:22:11 client-18 rshd[9027]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "cat /proc/mounts");echo XXRETCODE:$?'
May 17 14:22:12 client-18 xinetd[3390]: EXIT: shell status=0 pid=9026 duration=1(sec)
May 17 14:22:12 client-18 xinetd[3390]: START: shell pid=9061 from=192.168.4.5
May 17 14:22:12 client-18 rshd[9062]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl get_param -n osc.*OST*-osc-[^M][^D][^T]*.srpc_info");echo XXRETCODE:$?'
May 17 14:22:12 client-18 xinetd[3390]: EXIT: shell status=0 pid=9061 duration=0(sec)
May 17 14:22:12 client-18 xinetd[3390]: START: shell pid=9096 from=192.168.4.5
May 17 14:22:12 client-18 rshd[9097]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl get_param -n mdc.*-MDT*-mdc-*.srpc_info");echo XXRETCODE:$?'
May 17 14:22:12 client-18 xinetd[3390]: EXIT: shell status=0 pid=9096 duration=0(sec)
May 17 14:22:12 client-18 xinetd[3390]: START: shell pid=9131 from=192.168.4.5
May 17 14:22:13 client-18 rshd[9132]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == runracer test 1: racer on clients: client-5-ib,client-18-ib DURATION=120 ========================== 14:22:12 \(1305667332\)");echo XXRETCODE:$?'
May 17 14:22:13 client-18 kernel: Lustre: DEBUG MARKER: == runracer test 1: racer on clients: client-5-ib,client-18-ib DURATION=120 ========================== 14:22:12 (1305667332)
May 17 14:22:13 client-18 xinetd[3390]: EXIT: shell status=0 pid=9131 duration=1(sec)
May 17 14:22:13 client-18 xinetd[3390]: START: shell pid=9166 from=192.168.4.5
May 17 14:22:13 client-18 rshd[9167]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "DURATION=120 /usr/lib64/lustre/tests/racer/racer.sh /mnt/lustre/racer ");echo XXRETCODE:$?'
May 17 14:22:31 client-18 kernel: LustreError: 2098:0:(file.c:2161:ll_inode_revalidate_fini()) failure -2 inode 144115305935863839
May 17 14:23:24 client-18 kernel: LustreError: 5540:0:(file.c:2161:ll_inode_revalidate_fini()) failure -2 inode 144115305935872847
May 17 14:23:55 client-18 kernel: LustreError: 10693:0:(file.c:2161:ll_inode_revalidate_fini()) failure -116 inode 144115305986174006
May 17 14:24:13 client-18 xinetd[3390]: EXIT: shell status=0 pid=9166 duration=120(sec)
May 17 14:24:14 client-18 xinetd[3390]: START: shell pid=4072 from=192.168.4.5
May 17 14:24:14 client-18 rshd[4073]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "ps uax | grep /mnt/lustre/racer ");echo XXRETCODE:$?'
May 17 14:24:14 client-18 xinetd[3390]: EXIT: shell status=0 pid=4072 duration=0(sec)
May 17 14:24:24 client-18 xinetd[3390]: START: shell pid=4109 from=192.168.4.5
May 17 14:24:24 client-18 rshd[4110]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "ps uax | grep /mnt/lustre/racer ");echo XXRETCODE:$?'
May 17 14:24:24 client-18 xinetd[3390]: EXIT: shell status=0 pid=4109 duration=0(sec)
May 17 14:24:44 client-18 xinetd[3390]: START: shell pid=4146 from=192.168.4.5
May 17 14:24:44 client-18 rshd[4147]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "ps uax | grep /mnt/lustre/racer ");echo XXRETCODE:$?'
May 17 14:24:44 client-18 xinetd[3390]: EXIT: shell status=0 pid=4146 duration=0(sec)
May 17 14:25:24 client-18 xinetd[3390]: START: shell pid=4187 from=192.168.4.5
May 17 14:25:24 client-18 rshd[4188]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "ps uax | grep /mnt/lustre/racer ");echo XXRETCODE:$?'
May 17 14:25:24 client-18 xinetd[3390]: EXIT: shell status=0 pid=4187 duration=0(sec)
May 17 14:26:04 client-18 xinetd[3390]: START: shell pid=4224 from=192.168.4.5
May 17 14:26:04 client-18 rshd[4225]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "ps uax | grep /mnt/lustre/racer ");echo XXRETCODE:$?'
May 17 14:26:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=4224 duration=0(sec)
May 17 14:26:44 client-18 xinetd[3390]: START: shell pid=4265 from=192.168.4.5
May 17 14:26:45 client-18 rshd[4266]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "ps uax | grep /mnt/lustre/racer ");echo XXRETCODE:$?'
May 17 14:26:45 client-18 xinetd[3390]: EXIT: shell status=0 pid=4265 duration=1(sec)
May 17 14:27:25 client-18 xinetd[3390]: START: shell pid=4309 from=192.168.4.5
May 17 14:27:25 client-18 rshd[4310]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "ps uax | grep /mnt/lustre/racer ");echo XXRETCODE:$?'
May 17 14:27:25 client-18 xinetd[3390]: EXIT: shell status=0 pid=4309 duration=0(sec)
May 17 14:28:05 client-18 xinetd[3390]: START: shell pid=4346 from=192.168.4.5
May 17 14:28:05 client-18 rshd[4347]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "ps uax | grep /mnt/lustre/racer ");echo XXRETCODE:$?'
May 17 14:28:05 client-18 xinetd[3390]: EXIT: shell status=0 pid=4346 duration=0(sec)
May 17 14:28:45 client-18 xinetd[3390]: START: shell pid=4387 from=192.168.4.5
May 17 14:28:45 client-18 rshd[4388]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "ps uax | grep /mnt/lustre/racer ");echo XXRETCODE:$?'
May 17 14:28:45 client-18 xinetd[3390]: EXIT: shell status=0 pid=4387 duration=0(sec)
May 17 14:29:25 client-18 xinetd[3390]: START: shell pid=4424 from=192.168.4.5
May 17 14:29:25 client-18 rshd[4425]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "ps uax | grep /mnt/lustre/racer ");echo XXRETCODE:$?'
May 17 14:29:25 client-18 xinetd[3390]: EXIT: shell status=0 pid=4424 duration=0(sec)
May 17 14:30:05 client-18 xinetd[3390]: START: shell pid=4465 from=192.168.4.5
May 17 14:30:05 client-18 rshd[4466]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "ps uax | grep /mnt/lustre/racer ");echo XXRETCODE:$?'
May 17 14:30:05 client-18 xinetd[3390]: EXIT: shell status=0 pid=4465 duration=0(sec)
May 17 14:30:45 client-18 xinetd[3390]: START: shell pid=4502 from=192.168.4.5
May 17 14:30:45 client-18 rshd[4503]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "ps uax | grep /mnt/lustre/racer ");echo XXRETCODE:$?'
May 17 14:30:45 client-18 xinetd[3390]: EXIT: shell status=0 pid=4502 duration=0(sec)
May 17 14:31:25 client-18 xinetd[3390]: START: shell pid=4543 from=192.168.4.5
May 17 14:31:25 client-18 rshd[4544]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "ps uax | grep /mnt/lustre/racer ");echo XXRETCODE:$?'
May 17 14:31:25 client-18 xinetd[3390]: EXIT: shell status=0 pid=4543 duration=0(sec)
May 17 14:32:05 client-18 xinetd[3390]: START: shell pid=4580 from=192.168.4.5
May 17 14:32:05 client-18 rshd[4581]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "ps uax | grep /mnt/lustre/racer ");echo XXRETCODE:$?'
May 17 14:32:06 client-18 xinetd[3390]: EXIT: shell status=0 pid=4580 duration=1(sec)
May 17 14:32:46 client-18 xinetd[3390]: START: shell pid=4621 from=192.168.4.5
May 17 14:32:46 client-18 rshd[4622]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "ps uax | grep /mnt/lustre/racer ");echo XXRETCODE:$?'
May 17 14:32:46 client-18 xinetd[3390]: EXIT: shell status=0 pid=4621 duration=0(sec)
May 17 14:33:26 client-18 xinetd[3390]: START: shell pid=4658 from=192.168.4.5
May 17 14:33:26 client-18 rshd[4659]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "ps uax | grep /mnt/lustre/racer ");echo XXRETCODE:$?'
May 17 14:33:26 client-18 xinetd[3390]: EXIT: shell status=0 pid=4658 duration=0(sec)
May 17 14:34:06 client-18 xinetd[3390]: START: shell pid=4699 from=192.168.4.5
May 17 14:34:06 client-18 rshd[4700]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "ps uax | grep /mnt/lustre/racer ");echo XXRETCODE:$?'
May 17 14:34:06 client-18 xinetd[3390]: EXIT: shell status=0 pid=4699 duration=0(sec)
May 17 14:35:06 client-18 xinetd[3390]: START: shell pid=4740 from=192.168.4.5
May 17 14:35:06 client-18 rshd[4741]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "ps uax | grep /mnt/lustre/racer ");echo XXRETCODE:$?'
May 17 14:35:06 client-18 xinetd[3390]: EXIT: shell status=0 pid=4740 duration=0(sec)
May 17 14:35:06 client-18 xinetd[3390]: START: shell pid=4777 from=192.168.4.5
May 17 14:35:06 client-18 rshd[4778]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:35:06 client-18 xinetd[3390]: EXIT: shell status=0 pid=4777 duration=0(sec)
May 17 14:35:06 client-18 xinetd[3390]: START: shell pid=4813 from=192.168.4.5
May 17 14:35:06 client-18 rshd[4814]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:35:06 client-18 xinetd[3390]: EXIT: shell status=0 pid=4813 duration=0(sec)
May 17 14:35:07 client-18 xinetd[3390]: START: shell pid=4850 from=192.168.4.5
May 17 14:35:07 client-18 rshd[4851]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == runracer runracer test complete, duration 778 sec ================================================= 14:35:06 \(1305668106\)");echo XXRETCODE:$?'
May 17 14:35:07 client-18 kernel: Lustre: DEBUG MARKER: == runracer runracer test complete, duration 778 sec ================================================= 14:35:06 (1305668106)
May 17 14:35:07 client-18 xinetd[3390]: EXIT: shell status=0 pid=4850 duration=0(sec)
May 17 14:35:07 client-18 xinetd[3390]: START: shell pid=4885 from=192.168.4.5
May 17 14:35:07 client-18 rshd[4886]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark -----============= acceptance-small: replay-single ============----- Tue May 17 14:35:07 PDT 2011");echo XXRETCODE:$?'
May 17 14:35:07 client-18 kernel: Lustre: DEBUG MARKER: -----============= acceptance-small: replay-single ============----- Tue May 17 14:35:07 PDT 2011
May 17 14:35:07 client-18 xinetd[3390]: EXIT: shell status=0 pid=4885 duration=0(sec)
May 17 14:35:07 client-18 xinetd[3390]: START: shell pid=4920 from=192.168.4.5
May 17 14:35:07 client-18 rshd[4921]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark excepting tests: 61d 33a 33b");echo XXRETCODE:$?'
May 17 14:35:07 client-18 kernel: Lustre: DEBUG MARKER: excepting tests: 61d 33a 33b
May 17 14:35:07 client-18 xinetd[3390]: EXIT: shell status=0 pid=4920 duration=0(sec)
May 17 14:35:07 client-18 xinetd[3390]: START: shell pid=4955 from=192.168.4.5
May 17 14:35:07 client-18 rshd[4956]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/tests:/usr/lib64/openmpi/1.4-gcc/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/usr/lib64/lustre/utils:/usr/lib64/lustre/utils/gss:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey::/usr/lib64/lustre/tests/mpi NAME=sarah sh rpc.sh check_config_client /mnt/lustre ");echo XXRETCODE:$?'
May 17 14:35:07 client-18 xinetd[3390]: EXIT: shell status=0 pid=4955 duration=0(sec)
May 17 14:35:08 client-18 xinetd[3390]: START: shell pid=5049 from=192.168.4.5
May 17 14:35:08 client-18 rshd[5050]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); mpts=\$(mount | grep -w -c /mnt/lustre); if [ \$running -ne \$mpts ]; then     echo \$(hostname) env are INSANE!;     exit 1; fi");echo XXRETCODE:$?'
May 17 14:35:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=5049 duration=0(sec)
May 17 14:35:08 client-18 xinetd[3390]: START: shell pid=5089 from=192.168.4.5
May 17 14:35:08 client-18 rshd[5090]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); mpts=\$(mount | grep -w -c /mnt/lustre2); if [ \$running -ne \$mpts ]; then     echo \$(hostname) env are INSANE!;     exit 1; fi");echo XXRETCODE:$?'
May 17 14:35:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=5089 duration=0(sec)
May 17 14:35:10 client-18 xinetd[3390]: START: shell pid=5129 from=192.168.4.5
May 17 14:35:10 client-18 rshd[5130]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark Using TIMEOUT=20");echo XXRETCODE:$?'
May 17 14:35:10 client-18 kernel: Lustre: DEBUG MARKER: Using TIMEOUT=20
May 17 14:35:10 client-18 xinetd[3390]: EXIT: shell status=0 pid=5129 duration=0(sec)
May 17 14:35:11 client-18 xinetd[3390]: START: shell pid=5164 from=192.168.4.5
May 17 14:35:11 client-18 rshd[5165]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param debug=\"0x33f0406\";             lctl set_param subsystem_debug=\"0xffb7e3ff\";             lctl set_param debug_mb=32;             sync");echo XXRETCODE:$?'
May 17 14:35:11 client-18 kernel: Lustre: 5199:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release.
May 17 14:35:11 client-18 kernel: Lustre: 5199:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message
May 17 14:35:11 client-18 xinetd[3390]: EXIT: shell status=0 pid=5164 duration=0(sec)
May 17 14:35:11 client-18 xinetd[3390]: START: shell pid=5202 from=192.168.4.5
May 17 14:35:11 client-18 rshd[5203]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "cat /proc/mounts");echo XXRETCODE:$?'
May 17 14:35:11 client-18 xinetd[3390]: EXIT: shell status=0 pid=5202 duration=0(sec)
May 17 14:35:11 client-18 xinetd[3390]: START: shell pid=5237 from=192.168.4.5
May 17 14:35:12 client-18 rshd[5238]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl get_param -n osc.*OST*-osc-[^M][^D][^T]*.srpc_info");echo XXRETCODE:$?'
May 17 14:35:12 client-18 xinetd[3390]: EXIT: shell status=0 pid=5237 duration=1(sec)
May 17 14:35:12 client-18 xinetd[3390]: START: shell pid=5272 from=192.168.4.5
May 17 14:35:12 client-18 rshd[5273]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl get_param -n mdc.*-MDT*-mdc-*.srpc_info");echo XXRETCODE:$?'
May 17 14:35:12 client-18 xinetd[3390]: EXIT: shell status=0 pid=5272 duration=0(sec)
May 17 14:35:12 client-18 xinetd[3390]: START: shell pid=5307 from=192.168.4.5
May 17 14:35:12 client-18 rshd[5308]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 0a: empty replay =============================================================== 14:35:12 \(1305668112\)");echo XXRETCODE:$?'
May 17 14:35:12 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 0a: empty replay =============================================================== 14:35:12 (1305668112)
May 17 14:35:12 client-18 xinetd[3390]: EXIT: shell status=0 pid=5307 duration=0(sec)
May 17 14:35:22 client-18 xinetd[3390]: START: shell pid=5342 from=192.168.4.5
May 17 14:35:22 client-18 rshd[5343]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 14:35:22 client-18 xinetd[3390]: EXIT: shell status=0 pid=5342 duration=0(sec)
May 17 14:35:34 client-18 kernel: Lustre: 3859:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1368947485951849 sent from MGC192.168.4.128@o2ib to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305668127] [real_sent 1305668127] [current 1305668134] [deadline 7s] [delay 0s]  req@ffff8102ee2bdc00 x1368947485951849/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 192/192 e 0 to 1 dl 1305668134 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
May 17 14:35:34 client-18 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail.
May 17 14:35:39 client-18 kernel: Lustre: 3859:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1368947485951857 sent from lustre-MDT0000-mdc-ffff810317273c00 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305668132] [real_sent 1305668132] [current 1305668139] [deadline 7s] [delay 0s]  req@ffff8102ef7f9000 x1368947485951857/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 192/192 e 0 to 1 dl 1305668139 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
May 17 14:35:39 client-18 kernel: Lustre: lustre-MDT0000-mdc-ffff810317273c00: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete.
May 17 14:35:39 client-18 kernel: Lustre: Skipped 1 previous similar message
May 17 14:35:40 client-18 kernel: Lustre: 3860:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1368947485951864 sent from MGC192.168.4.128@o2ib to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305668134] [real_sent 1305668134] [current 1305668140] [deadline 6s] [delay 0s]  req@ffff8102f1bc7400 x1368947485951864/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 368/392 e 0 to 1 dl 1305668140 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
May 17 14:35:41 client-18 xinetd[3390]: START: shell pid=5381 from=192.168.4.5
May 17 14:35:42 client-18 rshd[5382]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 14:35:44 client-18 kernel: Lustre: 3861:0:(import.c:529:import_select_connection()) MGC192.168.4.128@o2ib: tried all connections, increasing latency to 6s
May 17 14:35:44 client-18 kernel: Lustre: 3860:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d6cda9931 to 0x62e5a8d74c86ad1
May 17 14:35:44 client-18 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import
May 17 14:35:44 client-18 kernel: Lustre: MGC192.168.4.128@o2ib: Connection restored to service MGS using nid 192.168.4.128@o2ib.
May 17 14:35:44 client-18 kernel: Lustre: Skipped 1 previous similar message
May 17 14:35:45 client-18 kernel: Lustre: 3860:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1368947485951872 sent from lustre-MDT0000-mdc-ffff810317273c00 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305668139] [real_sent 1305668139] [current 1305668145] [deadline 6s] [delay 0s]  req@ffff8102ef7f5000 x1368947485951872/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 368/392 e 0 to 1 dl 1305668145 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
May 17 14:35:49 client-18 kernel: Lustre: 3861:0:(import.c:529:import_select_connection()) lustre-MDT0000-mdc-ffff810317273c00: tried all connections, increasing latency to 6s
May 17 14:35:49 client-18 kernel: LustreError: 3860:0:(client.c:2570:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff81031dabf000 x1368947485065797/t17179876760(17179876760) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 544/544 e 0 to 0 dl 1305668161 ref 2 fl Interpret:RP/ffffffff/ffffffff rc 301/-1
May 17 14:35:49 client-18 kernel: LustreError: 3860:0:(mdc_request.c:642:mdc_replay_open()) @@@ Can't properly replay without open data.  req@ffff8103021fcc00 x1368947485447692/t17185310206(17185310206) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 528/544 e 0 to 0 dl 1305668161 ref 2 fl Interpret:RP/ffffffff/ffffffff rc 301/-1
May 17 14:35:49 client-18 kernel: LustreError: 3860:0:(mdc_request.c:642:mdc_replay_open()) @@@ Can't properly replay without open data.  req@ffff8102fcd80c00 x1368947485452538/t17185314821(17185314821) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 528/544 e 0 to 0 dl 1305668161 ref 2 fl Interpret:RP/ffffffff/ffffffff rc 301/-1
May 17 14:35:49 client-18 kernel: LustreError: 3860:0:(client.c:2570:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff8102ee2bd400 x1368947485851640/t17185724665(17185724665) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 544/544 e 0 to 0 dl 1305668161 ref 2 fl Interpret:RP/ffffffff/ffffffff rc 301/-1
May 17 14:35:49 client-18 kernel: Lustre: lustre-MDT0000-mdc-ffff810317273c00: Connection restored to service lustre-MDT0000 using nid 192.168.4.128@o2ib.
May 17 14:35:49 client-18 xinetd[3390]: EXIT: shell status=0 pid=5381 duration=8(sec)
May 17 14:35:49 client-18 xinetd[3390]: START: shell pid=5419 from=192.168.4.5
May 17 14:35:49 client-18 rshd[5420]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:35:49 client-18 xinetd[3390]: EXIT: shell status=0 pid=5419 duration=0(sec)
May 17 14:35:49 client-18 xinetd[3390]: START: shell pid=5455 from=192.168.4.5
May 17 14:35:49 client-18 rshd[5456]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:35:50 client-18 xinetd[3390]: EXIT: shell status=0 pid=5455 duration=1(sec)
May 17 14:35:50 client-18 xinetd[3390]: START: shell pid=5492 from=192.168.4.5
May 17 14:35:50 client-18 rshd[5493]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 0b: ensure object created after recover exists. \(3284\) ========================= 14:35:50 \(1305668150\)");echo XXRETCODE:$?'
May 17 14:35:50 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 0b: ensure object created after recover exists. (3284) ========================= 14:35:50 (1305668150)
May 17 14:35:50 client-18 xinetd[3390]: EXIT: shell status=0 pid=5492 duration=0(sec)
May 17 14:35:59 client-18 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.129@o2ib. The obd_ping operation failed with -107
May 17 14:35:59 client-18 kernel: LustreError: Skipped 2 previous similar messages
May 17 14:35:59 client-18 kernel: Lustre: lustre-OST0000-osc-ffff810317273c00: Connection to service lustre-OST0000 via nid 192.168.4.129@o2ib was lost; in progress operations using this service will wait for recovery to complete.
May 17 14:36:04 client-18 xinetd[3390]: START: shell pid=5527 from=192.168.4.5
May 17 14:36:04 client-18 rshd[5528]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 14:36:09 client-18 kernel: Lustre: 3861:0:(import.c:529:import_select_connection()) lustre-OST0000-osc-ffff810317273c00: tried all connections, increasing latency to 6s
May 17 14:36:09 client-18 kernel: Lustre: lustre-OST0000-osc-ffff810317273c00: Connection restored to service lustre-OST0000 using nid 192.168.4.129@o2ib.
May 17 14:36:09 client-18 xinetd[3390]: EXIT: shell status=0 pid=5527 duration=5(sec)
May 17 14:36:09 client-18 xinetd[3390]: START: shell pid=5565 from=192.168.4.5
May 17 14:36:09 client-18 rshd[5566]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:36:09 client-18 xinetd[3390]: EXIT: shell status=0 pid=5565 duration=0(sec)
May 17 14:36:09 client-18 xinetd[3390]: START: shell pid=5601 from=192.168.4.5
May 17 14:36:09 client-18 rshd[5602]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:36:09 client-18 xinetd[3390]: EXIT: shell status=0 pid=5601 duration=0(sec)
May 17 14:36:09 client-18 xinetd[3390]: START: shell pid=5638 from=192.168.4.5
May 17 14:36:09 client-18 rshd[5639]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 0d: expired recovery with no clients =========================================== 14:36:09 \(1305668169\)");echo XXRETCODE:$?'
May 17 14:36:09 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 0d: expired recovery with no clients =========================================== 14:36:09 (1305668169)
May 17 14:36:09 client-18 xinetd[3390]: EXIT: shell status=0 pid=5638 duration=0(sec)
May 17 14:36:09 client-18 xinetd[3390]: START: shell pid=5673 from=192.168.4.5
May 17 14:36:10 client-18 rshd[5674]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 14:36:10 client-18 xinetd[3390]: EXIT: shell status=0 pid=5673 duration=1(sec)
May 17 14:36:26 client-18 kernel: Lustre: 3859:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1368947485951972 sent from MGC192.168.4.128@o2ib to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305668174] [real_sent 1305668174] [current 1305668186] [deadline 12s] [delay 0s]  req@ffff810317273800 x1368947485951972/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 192/192 e 0 to 1 dl 1305668186 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
May 17 14:36:26 client-18 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail.
May 17 14:36:31 client-18 kernel: Lustre: 3859:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1368947485951974 sent from lustre-MDT0000-mdc-ffff810317273c00 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305668179] [real_sent 1305668179] [current 1305668191] [deadline 12s] [delay 0s]  req@ffff8102f0742800 x1368947485951974/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 192/192 e 0 to 1 dl 1305668191 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
May 17 14:36:31 client-18 kernel: Lustre: lustre-MDT0000-mdc-ffff810317273c00: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete.
May 17 14:36:41 client-18 kernel: Lustre: 3861:0:(import.c:529:import_select_connection()) MGC192.168.4.128@o2ib: tried all connections, increasing latency to 11s
May 17 14:36:41 client-18 kernel: Lustre: 3860:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74c86ad1 to 0x62e5a8d74c87910
May 17 14:36:41 client-18 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import
May 17 14:36:41 client-18 kernel: Lustre: MGC192.168.4.128@o2ib: Connection restored to service MGS using nid 192.168.4.128@o2ib.
May 17 14:37:35 client-18 kernel: LustreError: 3860:0:(mdc_request.c:642:mdc_replay_open()) @@@ Can't properly replay without open data.  req@ffff8103021fcc00 x1368947485447692/t17185310206(17185310206) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 528/544 e 0 to 0 dl 1305668286 ref 2 fl Interpret:RP/ffffffff/ffffffff rc 301/-1
May 17 14:37:35 client-18 kernel: LustreError: 3860:0:(mdc_request.c:642:mdc_replay_open()) Skipped 41 previous similar messages
May 17 14:37:35 client-18 kernel: Lustre: lustre-MDT0000-mdc-ffff810317273c00: Connection restored to service lustre-MDT0000 using nid 192.168.4.128@o2ib.
May 17 14:37:40 client-18 xinetd[3390]: START: shell pid=5719 from=192.168.4.5
May 17 14:37:40 client-18 rshd[5720]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:37:40 client-18 xinetd[3390]: EXIT: shell status=0 pid=5719 duration=0(sec)
May 17 14:37:40 client-18 xinetd[3390]: START: shell pid=5755 from=192.168.4.5
May 17 14:37:40 client-18 rshd[5756]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:37:40 client-18 xinetd[3390]: EXIT: shell status=0 pid=5755 duration=0(sec)
May 17 14:37:40 client-18 xinetd[3390]: START: shell pid=5792 from=192.168.4.5
May 17 14:37:40 client-18 rshd[5793]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 1: simple create =============================================================== 14:37:40 \(1305668260\)");echo XXRETCODE:$?'
May 17 14:37:40 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 1: simple create =============================================================== 14:37:40 (1305668260)
May 17 14:37:40 client-18 xinetd[3390]: EXIT: shell status=0 pid=5792 duration=0(sec)
May 17 14:37:40 client-18 xinetd[3390]: START: shell pid=5827 from=192.168.4.5
May 17 14:37:40 client-18 rshd[5828]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 14:37:40 client-18 xinetd[3390]: EXIT: shell status=0 pid=5827 duration=0(sec)
May 17 14:38:00 client-18 xinetd[3390]: START: shell pid=5866 from=192.168.4.5
May 17 14:38:00 client-18 rshd[5867]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 14:38:00 client-18 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_statfs operation failed with -107
May 17 14:38:00 client-18 kernel: LustreError: Skipped 1 previous similar message
May 17 14:38:00 client-18 kernel: Lustre: lustre-MDT0000-mdc-ffff810317273c00: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete.
May 17 14:38:00 client-18 kernel: LustreError: 3860:0:(mdc_request.c:642:mdc_replay_open()) @@@ Can't properly replay without open data.  req@ffff8103021fcc00 x1368947485447692/t17185310206(17185310206) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 528/544 e 0 to 0 dl 1305668311 ref 2 fl Interpret:RP/ffffffff/ffffffff rc 301/-1
May 17 14:38:00 client-18 kernel: LustreError: 3860:0:(mdc_request.c:642:mdc_replay_open()) Skipped 42 previous similar messages
May 17 14:38:00 client-18 kernel: Lustre: lustre-MDT0000-mdc-ffff810317273c00: Connection restored to service lustre-MDT0000 using nid 192.168.4.128@o2ib.
May 17 14:38:00 client-18 xinetd[3390]: EXIT: shell status=0 pid=5866 duration=0(sec)
May 17 14:38:00 client-18 xinetd[3390]: START: shell pid=5900 from=192.168.4.5
May 17 14:38:00 client-18 rshd[5901]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:38:00 client-18 xinetd[3390]: EXIT: shell status=0 pid=5900 duration=0(sec)
May 17 14:38:00 client-18 xinetd[3390]: START: shell pid=5936 from=192.168.4.5
May 17 14:38:00 client-18 rshd[5937]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:38:00 client-18 xinetd[3390]: EXIT: shell status=0 pid=5936 duration=0(sec)
May 17 14:38:00 client-18 xinetd[3390]: START: shell pid=5973 from=192.168.4.5
May 17 14:38:00 client-18 rshd[5974]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 2a: touch ====================================================================== 14:38:00 \(1305668280\)");echo XXRETCODE:$?'
May 17 14:38:00 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 2a: touch ====================================================================== 14:38:00 (1305668280)
May 17 14:38:00 client-18 xinetd[3390]: EXIT: shell status=0 pid=5973 duration=0(sec)
May 17 14:38:00 client-18 xinetd[3390]: START: shell pid=6008 from=192.168.4.5
May 17 14:38:01 client-18 rshd[6009]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 14:38:01 client-18 xinetd[3390]: EXIT: shell status=0 pid=6008 duration=1(sec)
May 17 14:38:01 client-18 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The obd_ping operation failed with -107
May 17 14:38:01 client-18 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail.
May 17 14:38:01 client-18 kernel: Lustre: 3860:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74c87910 to 0x62e5a8d74c881d7
May 17 14:38:01 client-18 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import
May 17 14:38:17 client-18 kernel: Lustre: 3859:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1368947485952109 sent from lustre-MDT0000-mdc-ffff810317273c00 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305668266] [real_sent 1305668266] [current 1305668297] [deadline 31s] [delay 0s]  req@ffff8102fd7bcc00 x1368947485952109/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 192/192 e 0 to 1 dl 1305668297 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
May 17 14:38:17 client-18 kernel: Lustre: 3859:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
May 17 14:38:20 client-18 xinetd[3390]: START: shell pid=6049 from=192.168.4.5
May 17 14:38:20 client-18 rshd[6050]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 14:38:20 client-18 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_statfs operation failed with -107
May 17 14:38:20 client-18 kernel: Lustre: lustre-MDT0000-mdc-ffff810317273c00: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete.
May 17 14:38:20 client-18 kernel: LustreError: 3860:0:(mdc_request.c:642:mdc_replay_open()) @@@ Can't properly replay without open data.  req@ffff8103021fcc00 x1368947485447692/t17185310206(17185310206) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 528/544 e 0 to 0 dl 1305668331 ref 2 fl Interpret:RP/ffffffff/ffffffff rc 301/-1
May 17 14:38:20 client-18 kernel: LustreError: 3860:0:(mdc_request.c:642:mdc_replay_open()) Skipped 42 previous similar messages
May 17 14:38:20 client-18 kernel: Lustre: lustre-MDT0000-mdc-ffff810317273c00: Connection restored to service lustre-MDT0000 using nid 192.168.4.128@o2ib.
May 17 14:38:20 client-18 kernel: Lustre: Skipped 1 previous similar message
May 17 14:38:20 client-18 xinetd[3390]: EXIT: shell status=0 pid=6049 duration=0(sec)
May 17 14:38:20 client-18 xinetd[3390]: START: shell pid=6083 from=192.168.4.5
May 17 14:38:20 client-18 rshd[6084]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:38:20 client-18 xinetd[3390]: EXIT: shell status=0 pid=6083 duration=0(sec)
May 17 14:38:20 client-18 xinetd[3390]: START: shell pid=6119 from=192.168.4.5
May 17 14:38:20 client-18 rshd[6120]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:38:20 client-18 xinetd[3390]: EXIT: shell status=0 pid=6119 duration=0(sec)
May 17 14:38:21 client-18 xinetd[3390]: START: shell pid=6156 from=192.168.4.5
May 17 14:38:21 client-18 rshd[6157]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 2b: touch ====================================================================== 14:38:20 \(1305668300\)");echo XXRETCODE:$?'
May 17 14:38:21 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 2b: touch ====================================================================== 14:38:20 (1305668300)
May 17 14:38:21 client-18 xinetd[3390]: EXIT: shell status=0 pid=6156 duration=0(sec)
May 17 14:38:21 client-18 xinetd[3390]: START: shell pid=6191 from=192.168.4.5
May 17 14:38:21 client-18 rshd[6192]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 14:38:21 client-18 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail.
May 17 14:38:21 client-18 xinetd[3390]: EXIT: shell status=0 pid=6191 duration=0(sec)
May 17 14:38:21 client-18 kernel: LustreError: 6048:0:(client.c:1057:ptlrpc_import_delay_req()) @@@ IMP_INVALID  req@ffff810301ac0400 x1368947485952189/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:/ffffffff/ffffffff rc 0/-1
May 17 14:38:21 client-18 kernel: Lustre: 3860:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74c881d7 to 0x62e5a8d74c886ca
May 17 14:38:21 client-18 kernel: LustreError: 6048:0:(client.c:1057:ptlrpc_import_delay_req()) @@@ IMP_INVALID  req@ffff8102eaf97c00 x1368947485952193/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:/ffffffff/ffffffff rc 0/-1
May 17 14:38:21 client-18 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import
May 17 14:38:37 client-18 kernel: Lustre: 3859:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1368947485952151 sent from lustre-MDT0000-mdc-ffff810317273c00 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305668286] [real_sent 1305668286] [current 1305668317] [deadline 31s] [delay 0s]  req@ffff81030145ec00 x1368947485952151/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 192/192 e 0 to 1 dl 1305668317 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
May 17 14:38:37 client-18 kernel: Lustre: 3859:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
May 17 14:38:40 client-18 xinetd[3390]: START: shell pid=6235 from=192.168.4.5
May 17 14:38:40 client-18 rshd[6236]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 14:38:40 client-18 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_statfs operation failed with -107
May 17 14:38:40 client-18 kernel: LustreError: Skipped 1 previous similar message
May 17 14:38:40 client-18 kernel: Lustre: lustre-MDT0000-mdc-ffff810317273c00: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete.
May 17 14:38:40 client-18 kernel: LustreError: 3860:0:(mdc_request.c:642:mdc_replay_open()) @@@ Can't properly replay without open data.  req@ffff8103021fcc00 x1368947485447692/t17185310206(17185310206) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 528/544 e 0 to 0 dl 1305668351 ref 2 fl Interpret:RP/ffffffff/ffffffff rc 301/-1
May 17 14:38:40 client-18 kernel: LustreError: 3860:0:(mdc_request.c:642:mdc_replay_open()) Skipped 42 previous similar messages
May 17 14:38:40 client-18 xinetd[3390]: EXIT: shell status=0 pid=6235 duration=0(sec)
May 17 14:38:41 client-18 xinetd[3390]: START: shell pid=6269 from=192.168.4.5
May 17 14:38:41 client-18 rshd[6270]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:38:41 client-18 xinetd[3390]: EXIT: shell status=0 pid=6269 duration=0(sec)
May 17 14:38:41 client-18 xinetd[3390]: START: shell pid=6305 from=192.168.4.5
May 17 14:38:41 client-18 rshd[6306]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:38:41 client-18 xinetd[3390]: EXIT: shell status=0 pid=6305 duration=0(sec)
May 17 14:38:41 client-18 xinetd[3390]: START: shell pid=6342 from=192.168.4.5
May 17 14:38:41 client-18 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail.
May 17 14:38:41 client-18 kernel: LustreError: 6048:0:(client.c:1057:ptlrpc_import_delay_req()) @@@ IMP_INVALID  req@ffff8102eaf97c00 x1368947485952231/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:/ffffffff/ffffffff rc 0/-1
May 17 14:38:41 client-18 kernel: Lustre: 3860:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74c886ca to 0x62e5a8d74c88b77
May 17 14:38:41 client-18 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import
May 17 14:38:41 client-18 rshd[6344]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 3a: replay failed open\(O_DIRECTORY\) ============================================ 14:38:41 \(1305668321\)");echo XXRETCODE:$?'
May 17 14:38:41 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 3a: replay failed open(O_DIRECTORY) ============================================ 14:38:41 (1305668321)
May 17 14:38:41 client-18 xinetd[3390]: EXIT: shell status=0 pid=6342 duration=0(sec)
May 17 14:38:41 client-18 xinetd[3390]: START: shell pid=6378 from=192.168.4.5
May 17 14:38:41 client-18 rshd[6379]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 14:38:41 client-18 xinetd[3390]: EXIT: shell status=0 pid=6378 duration=0(sec)
May 17 14:39:00 client-18 xinetd[3390]: START: shell pid=6417 from=192.168.4.5
May 17 14:39:00 client-18 rshd[6418]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 14:39:01 client-18 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_statfs operation failed with -107
May 17 14:39:01 client-18 kernel: LustreError: Skipped 1 previous similar message
May 17 14:39:01 client-18 kernel: Lustre: lustre-MDT0000-mdc-ffff810317273c00: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete.
May 17 14:39:01 client-18 kernel: LustreError: 3860:0:(mdc_request.c:642:mdc_replay_open()) @@@ Can't properly replay without open data.  req@ffff8103021fcc00 x1368947485447692/t17185310206(17185310206) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 528/544 e 0 to 0 dl 1305668372 ref 2 fl Interpret:RP/ffffffff/ffffffff rc 301/-1
May 17 14:39:01 client-18 kernel: LustreError: 3860:0:(mdc_request.c:642:mdc_replay_open()) Skipped 42 previous similar messages
May 17 14:39:01 client-18 kernel: Lustre: lustre-MDT0000-mdc-ffff810317273c00: Connection restored to service lustre-MDT0000 using nid 192.168.4.128@o2ib.
May 17 14:39:01 client-18 kernel: Lustre: Skipped 3 previous similar messages
May 17 14:39:01 client-18 xinetd[3390]: EXIT: shell status=0 pid=6417 duration=1(sec)
May 17 14:39:01 client-18 xinetd[3390]: START: shell pid=6451 from=192.168.4.5
May 17 14:39:01 client-18 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail.
May 17 14:39:01 client-18 kernel: LustreError: 6048:0:(client.c:1057:ptlrpc_import_delay_req()) @@@ IMP_INVALID  req@ffff810301a38800 x1368947485952273/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:/ffffffff/ffffffff rc 0/-1
May 17 14:39:01 client-18 kernel: LustreError: 6048:0:(client.c:1057:ptlrpc_import_delay_req()) Skipped 1 previous similar message
May 17 14:39:01 client-18 kernel: Lustre: 3860:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74c88b77 to 0x62e5a8d74c89086
May 17 14:39:01 client-18 rshd[6452]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:39:01 client-18 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import
May 17 14:39:01 client-18 xinetd[3390]: EXIT: shell status=0 pid=6451 duration=0(sec)
May 17 14:39:01 client-18 xinetd[3390]: START: shell pid=6488 from=192.168.4.5
May 17 14:39:01 client-18 rshd[6489]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:39:01 client-18 xinetd[3390]: EXIT: shell status=0 pid=6488 duration=0(sec)
May 17 14:39:01 client-18 xinetd[3390]: START: shell pid=6525 from=192.168.4.5
May 17 14:39:01 client-18 rshd[6526]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 3b: replay failed open -ENOMEM ================================================= 14:39:01 \(1305668341\)");echo XXRETCODE:$?'
May 17 14:39:01 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 3b: replay failed open -ENOMEM ================================================= 14:39:01 (1305668341)
May 17 14:39:01 client-18 xinetd[3390]: EXIT: shell status=0 pid=6525 duration=0(sec)
May 17 14:39:01 client-18 xinetd[3390]: START: shell pid=6560 from=192.168.4.5
May 17 14:39:01 client-18 rshd[6561]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 14:39:02 client-18 xinetd[3390]: EXIT: shell status=0 pid=6560 duration=1(sec)
May 17 14:39:21 client-18 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The obd_ping operation failed with -107
May 17 14:39:21 client-18 kernel: LustreError: Skipped 1 previous similar message
May 17 14:39:21 client-18 kernel: Lustre: lustre-MDT0000-mdc-ffff810317273c00: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete.
May 17 14:39:21 client-18 xinetd[3390]: START: shell pid=6599 from=192.168.4.5
May 17 14:39:21 client-18 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail.
May 17 14:39:21 client-18 kernel: LustreError: 6048:0:(client.c:1057:ptlrpc_import_delay_req()) @@@ IMP_INVALID  req@ffff8102e44cd800 x1368947485952311/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:/ffffffff/ffffffff rc 0/-1
May 17 14:39:21 client-18 kernel: LustreError: 6048:0:(client.c:1057:ptlrpc_import_delay_req()) Skipped 1 previous similar message
May 17 14:39:21 client-18 rshd[6600]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 14:39:21 client-18 kernel: Lustre: 3860:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74c89086 to 0x62e5a8d74c89277
May 17 14:39:21 client-18 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import
May 17 14:39:21 client-18 kernel: LustreError: 3860:0:(mdc_request.c:642:mdc_replay_open()) @@@ Can't properly replay without open data.  req@ffff8103021fcc00 x1368947485447692/t17185310206(17185310206) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 528/544 e 0 to 0 dl 1305668392 ref 2 fl Interpret:RP/ffffffff/ffffffff rc 301/-1
May 17 14:39:21 client-18 kernel: LustreError: 3860:0:(mdc_request.c:642:mdc_replay_open()) Skipped 42 previous similar messages
May 17 14:39:21 client-18 xinetd[3390]: EXIT: shell status=0 pid=6599 duration=0(sec)
May 17 14:39:21 client-18 xinetd[3390]: START: shell pid=6634 from=192.168.4.5
May 17 14:39:21 client-18 rshd[6635]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:39:21 client-18 xinetd[3390]: EXIT: shell status=0 pid=6634 duration=0(sec)
May 17 14:39:21 client-18 xinetd[3390]: START: shell pid=6670 from=192.168.4.5
May 17 14:39:21 client-18 rshd[6671]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:39:22 client-18 xinetd[3390]: EXIT: shell status=0 pid=6670 duration=1(sec)
May 17 14:39:22 client-18 xinetd[3390]: START: shell pid=6707 from=192.168.4.5
May 17 14:39:22 client-18 rshd[6708]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 3c: replay failed open -ENOMEM ================================================= 14:39:22 \(1305668362\)");echo XXRETCODE:$?'
May 17 14:39:22 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 3c: replay failed open -ENOMEM ================================================= 14:39:22 (1305668362)
May 17 14:39:22 client-18 xinetd[3390]: EXIT: shell status=0 pid=6707 duration=0(sec)
May 17 14:39:22 client-18 xinetd[3390]: START: shell pid=6742 from=192.168.4.5
May 17 14:39:22 client-18 rshd[6743]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 14:39:22 client-18 kernel: Lustre: 3859:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1368947485952244 sent from lustre-MDT0000-mdc-ffff810317273c00 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305668331] [real_sent 1305668331] [current 1305668362] [deadline 31s] [delay 0s]  req@ffff810306f6dc00 x1368947485952244/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 192/192 e 0 to 1 dl 1305668362 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
May 17 14:39:22 client-18 kernel: Lustre: 3859:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 4 previous similar messages
May 17 14:39:22 client-18 xinetd[3390]: EXIT: shell status=0 pid=6742 duration=0(sec)
May 17 14:39:41 client-18 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail.
May 17 14:39:41 client-18 kernel: LustreError: 6048:0:(client.c:1057:ptlrpc_import_delay_req()) @@@ IMP_INVALID  req@ffff81030556f400 x1368947485952353/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:/ffffffff/ffffffff rc 0/-1
May 17 14:39:41 client-18 kernel: LustreError: 6048:0:(client.c:1057:ptlrpc_import_delay_req()) Skipped 1 previous similar message
May 17 14:39:41 client-18 kernel: Lustre: 3860:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74c89277 to 0x62e5a8d74c8974e
May 17 14:39:41 client-18 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import
May 17 14:39:41 client-18 xinetd[3390]: START: shell pid=6782 from=192.168.4.5
May 17 14:39:41 client-18 rshd[6783]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 14:39:42 client-18 xinetd[3390]: EXIT: shell status=0 pid=6782 duration=1(sec)
May 17 14:39:42 client-18 xinetd[3390]: START: shell pid=6816 from=192.168.4.5
May 17 14:39:42 client-18 rshd[6817]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:39:42 client-18 xinetd[3390]: EXIT: shell status=0 pid=6816 duration=0(sec)
May 17 14:39:42 client-18 xinetd[3390]: START: shell pid=6852 from=192.168.4.5
May 17 14:39:42 client-18 rshd[6853]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:39:42 client-18 xinetd[3390]: EXIT: shell status=0 pid=6852 duration=0(sec)
May 17 14:39:42 client-18 xinetd[3390]: START: shell pid=6889 from=192.168.4.5
May 17 14:39:42 client-18 rshd[6890]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 4a: \|x\| 10 open\(O_CREAT\)s ====================================================== 14:39:42 \(1305668382\)");echo XXRETCODE:$?'
May 17 14:39:42 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 4a: |x| 10 open(O_CREAT)s ====================================================== 14:39:42 (1305668382)
May 17 14:39:42 client-18 xinetd[3390]: EXIT: shell status=0 pid=6889 duration=0(sec)
May 17 14:39:42 client-18 xinetd[3390]: START: shell pid=6924 from=192.168.4.5
May 17 14:39:42 client-18 rshd[6925]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 14:39:42 client-18 xinetd[3390]: EXIT: shell status=0 pid=6924 duration=0(sec)
May 17 14:40:01 client-18 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The obd_ping operation failed with -107
May 17 14:40:01 client-18 kernel: LustreError: Skipped 3 previous similar messages
May 17 14:40:01 client-18 kernel: LustreError: 6048:0:(client.c:1057:ptlrpc_import_delay_req()) @@@ IMP_INVALID  req@ffff8102fbb17800 x1368947485952398/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:/ffffffff/ffffffff rc 0/-1
May 17 14:40:01 client-18 kernel: Lustre: lustre-MDT0000-mdc-ffff810317273c00: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete.
May 17 14:40:01 client-18 kernel: Lustre: Skipped 1 previous similar message
May 17 14:40:01 client-18 kernel: LustreError: 6048:0:(client.c:1057:ptlrpc_import_delay_req()) Skipped 1 previous similar message
May 17 14:40:01 client-18 kernel: Lustre: 3860:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74c8974e to 0x62e5a8d74c89d28
May 17 14:40:01 client-18 xinetd[3390]: START: shell pid=6968 from=192.168.4.5
May 17 14:40:01 client-18 rshd[6969]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 14:40:02 client-18 kernel: LustreError: 3860:0:(mdc_request.c:642:mdc_replay_open()) @@@ Can't properly replay without open data.  req@ffff8103021fcc00 x1368947485447692/t17185310206(17185310206) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 528/544 e 0 to 0 dl 1305668432 ref 2 fl Interpret:RP/ffffffff/ffffffff rc 301/-1
May 17 14:40:02 client-18 kernel: LustreError: 3860:0:(mdc_request.c:642:mdc_replay_open()) Skipped 85 previous similar messages
May 17 14:40:02 client-18 xinetd[3390]: EXIT: shell status=0 pid=6968 duration=1(sec)
May 17 14:40:02 client-18 xinetd[3390]: START: shell pid=7002 from=192.168.4.5
May 17 14:40:02 client-18 rshd[7003]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:40:02 client-18 xinetd[3390]: EXIT: shell status=0 pid=7002 duration=0(sec)
May 17 14:40:02 client-18 xinetd[3390]: START: shell pid=7038 from=192.168.4.5
May 17 14:40:02 client-18 rshd[7039]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:40:02 client-18 xinetd[3390]: EXIT: shell status=0 pid=7038 duration=0(sec)
May 17 14:40:02 client-18 xinetd[3390]: START: shell pid=7075 from=192.168.4.5
May 17 14:40:02 client-18 rshd[7076]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 4b: \|x\| rm 10 files ============================================================ 14:40:02 \(1305668402\)");echo XXRETCODE:$?'
May 17 14:40:02 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 4b: |x| rm 10 files ============================================================ 14:40:02 (1305668402)
May 17 14:40:02 client-18 xinetd[3390]: EXIT: shell status=0 pid=7075 duration=0(sec)
May 17 14:40:02 client-18 xinetd[3390]: START: shell pid=7110 from=192.168.4.5
May 17 14:40:02 client-18 rshd[7111]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 14:40:02 client-18 xinetd[3390]: EXIT: shell status=0 pid=7110 duration=0(sec)
May 17 14:40:21 client-18 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail.
May 17 14:40:21 client-18 kernel: LustreError: Skipped 1 previous similar message
May 17 14:40:21 client-18 kernel: LustreError: 6048:0:(client.c:1057:ptlrpc_import_delay_req()) @@@ IMP_INVALID  req@ffff8102eeead400 x1368947485952443/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:/ffffffff/ffffffff rc 0/-1
May 17 14:40:21 client-18 kernel: LustreError: 6048:0:(client.c:1057:ptlrpc_import_delay_req()) Skipped 1 previous similar message
May 17 14:40:21 client-18 kernel: Lustre: 3860:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74c89d28 to 0x62e5a8d74c8a52b
May 17 14:40:21 client-18 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import
May 17 14:40:21 client-18 kernel: Lustre: Skipped 1 previous similar message
May 17 14:40:21 client-18 kernel: Lustre: MGC192.168.4.128@o2ib: Connection restored to service MGS using nid 192.168.4.128@o2ib.
May 17 14:40:21 client-18 kernel: Lustre: Skipped 7 previous similar messages
May 17 14:40:22 client-18 xinetd[3390]: START: shell pid=7150 from=192.168.4.5
May 17 14:40:22 client-18 rshd[7151]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 14:40:22 client-18 xinetd[3390]: EXIT: shell status=0 pid=7150 duration=0(sec)
May 17 14:40:22 client-18 xinetd[3390]: START: shell pid=7184 from=192.168.4.5
May 17 14:40:22 client-18 rshd[7185]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:40:22 client-18 xinetd[3390]: EXIT: shell status=0 pid=7184 duration=0(sec)
May 17 14:40:22 client-18 xinetd[3390]: START: shell pid=7220 from=192.168.4.5
May 17 14:40:22 client-18 rshd[7221]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:40:22 client-18 xinetd[3390]: EXIT: shell status=0 pid=7220 duration=0(sec)
May 17 14:40:22 client-18 xinetd[3390]: START: shell pid=7257 from=192.168.4.5
May 17 14:40:22 client-18 rshd[7258]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 5: \|x\| 220 open\(O_CREAT\) ======================================================= 14:40:22 \(1305668422\)");echo XXRETCODE:$?'
May 17 14:40:22 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 5: |x| 220 open(O_CREAT) ======================================================= 14:40:22 (1305668422)
May 17 14:40:22 client-18 xinetd[3390]: EXIT: shell status=0 pid=7257 duration=0(sec)
May 17 14:40:23 client-18 xinetd[3390]: START: shell pid=7292 from=192.168.4.5
May 17 14:40:23 client-18 rshd[7293]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 14:40:23 client-18 xinetd[3390]: EXIT: shell status=0 pid=7292 duration=0(sec)
May 17 14:40:27 client-18 kernel: Lustre: 3859:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1368947485952382 sent from lustre-MDT0000-mdc-ffff810317273c00 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305668396] [real_sent 1305668396] [current 1305668427] [deadline 31s] [delay 0s]  req@ffff8102f05a6c00 x1368947485952382/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 192/192 e 0 to 1 dl 1305668427 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
May 17 14:40:27 client-18 kernel: Lustre: 3859:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 6 previous similar messages
May 17 14:40:41 client-18 kernel: Lustre: 3860:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74c8a52b to 0x62e5a8d74c8c297
May 17 14:40:42 client-18 xinetd[3390]: START: shell pid=7332 from=192.168.4.5
May 17 14:40:42 client-18 rshd[7333]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 14:40:42 client-18 xinetd[3390]: EXIT: shell status=0 pid=7332 duration=0(sec)
May 17 14:40:46 client-18 xinetd[3390]: START: shell pid=7368 from=192.168.4.5
May 17 14:40:46 client-18 rshd[7369]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:40:46 client-18 xinetd[3390]: EXIT: shell status=0 pid=7368 duration=0(sec)
May 17 14:40:46 client-18 xinetd[3390]: START: shell pid=7404 from=192.168.4.5
May 17 14:40:46 client-18 rshd[7405]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:40:47 client-18 xinetd[3390]: EXIT: shell status=0 pid=7404 duration=1(sec)
May 17 14:40:47 client-18 xinetd[3390]: START: shell pid=7441 from=192.168.4.5
May 17 14:40:47 client-18 rshd[7442]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 6a: mkdir + contained create =================================================== 14:40:47 \(1305668447\)");echo XXRETCODE:$?'
May 17 14:40:47 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 6a: mkdir + contained create =================================================== 14:40:47 (1305668447)
May 17 14:40:47 client-18 xinetd[3390]: EXIT: shell status=0 pid=7441 duration=0(sec)
May 17 14:40:47 client-18 xinetd[3390]: START: shell pid=7476 from=192.168.4.5
May 17 14:40:47 client-18 rshd[7477]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 14:40:47 client-18 xinetd[3390]: EXIT: shell status=0 pid=7476 duration=0(sec)
May 17 14:41:06 client-18 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The obd_ping operation failed with -107
May 17 14:41:06 client-18 kernel: LustreError: Skipped 5 previous similar messages
May 17 14:41:06 client-18 kernel: Lustre: 3860:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74c8c297 to 0x62e5a8d74c9226e
May 17 14:41:06 client-18 kernel: Lustre: lustre-MDT0000-mdc-ffff810317273c00: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete.
May 17 14:41:06 client-18 kernel: Lustre: Skipped 2 previous similar messages
May 17 14:41:06 client-18 xinetd[3390]: START: shell pid=7516 from=192.168.4.5
May 17 14:41:06 client-18 rshd[7517]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 14:41:06 client-18 kernel: LustreError: 3860:0:(mdc_request.c:642:mdc_replay_open()) @@@ Can't properly replay without open data.  req@ffff8103021fcc00 x1368947485447692/t17185310206(17185310206) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 528/544 e 0 to 0 dl 1305668497 ref 2 fl Interpret:RP/ffffffff/ffffffff rc 301/-1
May 17 14:41:06 client-18 kernel: LustreError: 3860:0:(mdc_request.c:642:mdc_replay_open()) Skipped 128 previous similar messages
May 17 14:41:06 client-18 xinetd[3390]: EXIT: shell status=0 pid=7516 duration=0(sec)
May 17 14:41:08 client-18 xinetd[3390]: START: shell pid=7550 from=192.168.4.5
May 17 14:41:08 client-18 rshd[7551]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:41:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=7550 duration=0(sec)
May 17 14:41:09 client-18 xinetd[3390]: START: shell pid=7586 from=192.168.4.5
May 17 14:41:09 client-18 rshd[7587]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:41:09 client-18 xinetd[3390]: EXIT: shell status=0 pid=7586 duration=0(sec)
May 17 14:41:09 client-18 xinetd[3390]: START: shell pid=7623 from=192.168.4.5
May 17 14:41:09 client-18 rshd[7624]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 6b: \|X\| rmdir ================================================================== 14:41:09 \(1305668469\)");echo XXRETCODE:$?'
May 17 14:41:09 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 6b: |X| rmdir ================================================================== 14:41:09 (1305668469)
May 17 14:41:09 client-18 xinetd[3390]: EXIT: shell status=0 pid=7623 duration=0(sec)
May 17 14:41:09 client-18 xinetd[3390]: START: shell pid=7662 from=192.168.4.5
May 17 14:41:09 client-18 rshd[7663]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 14:41:09 client-18 xinetd[3390]: EXIT: shell status=0 pid=7662 duration=0(sec)
May 17 14:41:27 client-18 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail.
May 17 14:41:27 client-18 kernel: LustreError: Skipped 2 previous similar messages
May 17 14:41:27 client-18 kernel: LustreError: 6048:0:(client.c:1057:ptlrpc_import_delay_req()) @@@ IMP_INVALID  req@ffff8102fbb17c00 x1368947485952584/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:/ffffffff/ffffffff rc 0/-1
May 17 14:41:27 client-18 kernel: LustreError: 6048:0:(client.c:1057:ptlrpc_import_delay_req()) Skipped 3 previous similar messages
May 17 14:41:27 client-18 kernel: Lustre: 3860:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74c9226e to 0x62e5a8d74c927c3
May 17 14:41:27 client-18 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import
May 17 14:41:27 client-18 kernel: Lustre: Skipped 2 previous similar messages
May 17 14:41:28 client-18 xinetd[3390]: START: shell pid=7702 from=192.168.4.5
May 17 14:41:28 client-18 rshd[7703]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 14:41:28 client-18 xinetd[3390]: EXIT: shell status=0 pid=7702 duration=0(sec)
May 17 14:41:29 client-18 xinetd[3390]: START: shell pid=7736 from=192.168.4.5
May 17 14:41:29 client-18 rshd[7737]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:41:29 client-18 xinetd[3390]: EXIT: shell status=0 pid=7736 duration=0(sec)
May 17 14:41:29 client-18 xinetd[3390]: START: shell pid=7772 from=192.168.4.5
May 17 14:41:29 client-18 rshd[7773]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:41:29 client-18 xinetd[3390]: EXIT: shell status=0 pid=7772 duration=0(sec)
May 17 14:41:29 client-18 xinetd[3390]: START: shell pid=7809 from=192.168.4.5
May 17 14:41:29 client-18 rshd[7810]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 7: mkdir \|X\| contained create ================================================== 14:41:29 \(1305668489\)");echo XXRETCODE:$?'
May 17 14:41:29 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 7: mkdir |X| contained create ================================================== 14:41:29 (1305668489)
May 17 14:41:29 client-18 xinetd[3390]: EXIT: shell status=0 pid=7809 duration=0(sec)
May 17 14:41:29 client-18 xinetd[3390]: START: shell pid=7844 from=192.168.4.5
May 17 14:41:29 client-18 rshd[7845]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 14:41:29 client-18 xinetd[3390]: EXIT: shell status=0 pid=7844 duration=0(sec)
May 17 14:41:47 client-18 kernel: Lustre: 3860:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74c927c3 to 0x62e5a8d74c92d42
May 17 14:41:48 client-18 xinetd[3390]: START: shell pid=7884 from=192.168.4.5
May 17 14:41:48 client-18 rshd[7885]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 14:41:48 client-18 xinetd[3390]: EXIT: shell status=0 pid=7884 duration=0(sec)
May 17 14:41:48 client-18 xinetd[3390]: START: shell pid=7918 from=192.168.4.5
May 17 14:41:48 client-18 rshd[7919]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:41:48 client-18 xinetd[3390]: EXIT: shell status=0 pid=7918 duration=0(sec)
May 17 14:41:49 client-18 xinetd[3390]: START: shell pid=7954 from=192.168.4.5
May 17 14:41:49 client-18 rshd[7955]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:41:49 client-18 xinetd[3390]: EXIT: shell status=0 pid=7954 duration=0(sec)
May 17 14:41:49 client-18 xinetd[3390]: START: shell pid=7991 from=192.168.4.5
May 17 14:41:49 client-18 rshd[7992]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 8: creat open \|X\| close ======================================================== 14:41:49 \(1305668509\)");echo XXRETCODE:$?'
May 17 14:41:49 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 8: creat open |X| close ======================================================== 14:41:49 (1305668509)
May 17 14:41:49 client-18 xinetd[3390]: EXIT: shell status=0 pid=7991 duration=0(sec)
May 17 14:41:49 client-18 xinetd[3390]: START: shell pid=8026 from=192.168.4.5
May 17 14:41:49 client-18 rshd[8027]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 14:41:49 client-18 xinetd[3390]: EXIT: shell status=0 pid=8026 duration=0(sec)
May 17 14:42:07 client-18 kernel: Lustre: 3860:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74c92d42 to 0x62e5a8d74c932f9
May 17 14:42:08 client-18 xinetd[3390]: START: shell pid=8066 from=192.168.4.5
May 17 14:42:08 client-18 rshd[8067]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 14:42:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=8066 duration=0(sec)
May 17 14:42:08 client-18 xinetd[3390]: START: shell pid=8100 from=192.168.4.5
May 17 14:42:08 client-18 rshd[8101]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:42:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=8100 duration=0(sec)
May 17 14:42:08 client-18 xinetd[3390]: START: shell pid=8136 from=192.168.4.5
May 17 14:42:08 client-18 rshd[8137]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:42:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=8136 duration=0(sec)
May 17 14:42:09 client-18 xinetd[3390]: START: shell pid=8173 from=192.168.4.5
May 17 14:42:09 client-18 rshd[8174]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 9: \|X\| create \(same inum\/gen\) ================================================== 14:42:08 \(1305668528\)");echo XXRETCODE:$?'
May 17 14:42:09 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 9: |X| create (same inum/gen) ================================================== 14:42:08 (1305668528)
May 17 14:42:09 client-18 xinetd[3390]: EXIT: shell status=0 pid=8173 duration=0(sec)
May 17 14:42:09 client-18 xinetd[3390]: START: shell pid=8208 from=192.168.4.5
May 17 14:42:09 client-18 rshd[8209]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 14:42:09 client-18 xinetd[3390]: EXIT: shell status=0 pid=8208 duration=0(sec)
May 17 14:42:27 client-18 kernel: Lustre: 3860:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74c932f9 to 0x62e5a8d74c9382b
May 17 14:42:28 client-18 xinetd[3390]: START: shell pid=8252 from=192.168.4.5
May 17 14:42:28 client-18 rshd[8253]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 14:42:28 client-18 xinetd[3390]: EXIT: shell status=0 pid=8252 duration=0(sec)
May 17 14:42:28 client-18 xinetd[3390]: START: shell pid=8286 from=192.168.4.5
May 17 14:42:28 client-18 rshd[8287]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:42:28 client-18 xinetd[3390]: EXIT: shell status=0 pid=8286 duration=0(sec)
May 17 14:42:28 client-18 xinetd[3390]: START: shell pid=8322 from=192.168.4.5
May 17 14:42:28 client-18 rshd[8323]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:42:28 client-18 xinetd[3390]: EXIT: shell status=0 pid=8322 duration=0(sec)
May 17 14:42:29 client-18 xinetd[3390]: START: shell pid=8359 from=192.168.4.5
May 17 14:42:29 client-18 rshd[8360]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 10: create \|X\| rename unlink =================================================== 14:42:28 \(1305668548\)");echo XXRETCODE:$?'
May 17 14:42:29 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 10: create |X| rename unlink =================================================== 14:42:28 (1305668548)
May 17 14:42:29 client-18 xinetd[3390]: EXIT: shell status=0 pid=8359 duration=0(sec)
May 17 14:42:29 client-18 xinetd[3390]: START: shell pid=8394 from=192.168.4.5
May 17 14:42:29 client-18 rshd[8395]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 14:42:29 client-18 xinetd[3390]: EXIT: shell status=0 pid=8394 duration=0(sec)
May 17 14:42:47 client-18 kernel: LustreError: 6048:0:(client.c:1057:ptlrpc_import_delay_req()) @@@ IMP_INVALID  req@ffff810301a38c00 x1368947485952761/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:/ffffffff/ffffffff rc 0/-1
May 17 14:42:47 client-18 kernel: LustreError: 6048:0:(client.c:1057:ptlrpc_import_delay_req()) Skipped 7 previous similar messages
May 17 14:42:47 client-18 kernel: Lustre: 3860:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74c9382b to 0x62e5a8d74c93dc6
May 17 14:42:47 client-18 kernel: Lustre: MGC192.168.4.128@o2ib: Connection restored to service MGS using nid 192.168.4.128@o2ib.
May 17 14:42:47 client-18 kernel: Lustre: Skipped 13 previous similar messages
May 17 14:42:48 client-18 xinetd[3390]: START: shell pid=8434 from=192.168.4.5
May 17 14:42:48 client-18 rshd[8435]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 14:42:48 client-18 xinetd[3390]: EXIT: shell status=0 pid=8434 duration=0(sec)
May 17 14:42:48 client-18 xinetd[3390]: START: shell pid=8468 from=192.168.4.5
May 17 14:42:48 client-18 rshd[8469]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:42:48 client-18 xinetd[3390]: EXIT: shell status=0 pid=8468 duration=0(sec)
May 17 14:42:48 client-18 xinetd[3390]: START: shell pid=8504 from=192.168.4.5
May 17 14:42:48 client-18 rshd[8505]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:42:48 client-18 xinetd[3390]: EXIT: shell status=0 pid=8504 duration=0(sec)
May 17 14:42:48 client-18 kernel: Lustre: 3859:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1368947485952693 sent from lustre-MDT0000-mdc-ffff810317273c00 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305668537] [real_sent 1305668537] [current 1305668568] [deadline 31s] [delay 0s]  req@ffff8102fdb75400 x1368947485952693/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 192/192 e 0 to 1 dl 1305668568 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
May 17 14:42:48 client-18 kernel: Lustre: 3859:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 14 previous similar messages
May 17 14:42:48 client-18 xinetd[3390]: START: shell pid=8541 from=192.168.4.5
May 17 14:42:48 client-18 rshd[8542]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 11: create open write rename \|X\| create-old-name read ========================== 14:42:48 \(1305668568\)");echo XXRETCODE:$?'
May 17 14:42:48 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 11: create open write rename |X| create-old-name read ========================== 14:42:48 (1305668568)
May 17 14:42:48 client-18 xinetd[3390]: EXIT: shell status=0 pid=8541 duration=0(sec)
May 17 14:42:49 client-18 xinetd[3390]: START: shell pid=8576 from=192.168.4.5
May 17 14:42:49 client-18 rshd[8577]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 14:42:49 client-18 xinetd[3390]: EXIT: shell status=0 pid=8576 duration=0(sec)
May 17 14:43:07 client-18 kernel: Lustre: 3860:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74c93dc6 to 0x62e5a8d74c94425
May 17 14:43:08 client-18 xinetd[3390]: START: shell pid=8616 from=192.168.4.5
May 17 14:43:08 client-18 rshd[8617]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 14:43:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=8616 duration=0(sec)
May 17 14:43:08 client-18 xinetd[3390]: START: shell pid=8650 from=192.168.4.5
May 17 14:43:08 client-18 rshd[8651]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:43:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=8650 duration=0(sec)
May 17 14:43:08 client-18 xinetd[3390]: START: shell pid=8687 from=192.168.4.5
May 17 14:43:08 client-18 rshd[8688]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:43:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=8687 duration=0(sec)
May 17 14:43:08 client-18 xinetd[3390]: START: shell pid=8724 from=192.168.4.5
May 17 14:43:08 client-18 rshd[8725]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 12: open, unlink \|X\| close ===================================================== 14:43:08 \(1305668588\)");echo XXRETCODE:$?'
May 17 14:43:08 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 12: open, unlink |X| close ===================================================== 14:43:08 (1305668588)
May 17 14:43:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=8724 duration=0(sec)
May 17 14:43:08 client-18 xinetd[3390]: START: shell pid=8759 from=192.168.4.5
May 17 14:43:08 client-18 rshd[8760]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 14:43:09 client-18 xinetd[3390]: EXIT: shell status=0 pid=8759 duration=1(sec)
May 17 14:43:27 client-18 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The obd_ping operation failed with -107
May 17 14:43:27 client-18 kernel: LustreError: Skipped 12 previous similar messages
May 17 14:43:27 client-18 kernel: Lustre: lustre-MDT0000-mdc-ffff810317273c00: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete.
May 17 14:43:27 client-18 kernel: Lustre: Skipped 6 previous similar messages
May 17 14:43:27 client-18 kernel: Lustre: 3860:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74c94425 to 0x62e5a8d74c949f8
May 17 14:43:27 client-18 kernel: LustreError: 3860:0:(mdc_request.c:642:mdc_replay_open()) @@@ Can't properly replay without open data.  req@ffff8103021fcc00 x1368947485447692/t17185310206(17185310206) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 528/544 e 0 to 0 dl 1305668638 ref 2 fl Interpret:RP/ffffffff/ffffffff rc 301/-1
May 17 14:43:27 client-18 kernel: LustreError: 3860:0:(mdc_request.c:642:mdc_replay_open()) Skipped 300 previous similar messages
May 17 14:43:28 client-18 xinetd[3390]: START: shell pid=8799 from=192.168.4.5
May 17 14:43:28 client-18 rshd[8800]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 14:43:28 client-18 xinetd[3390]: EXIT: shell status=0 pid=8799 duration=0(sec)
May 17 14:43:28 client-18 xinetd[3390]: START: shell pid=8833 from=192.168.4.5
May 17 14:43:28 client-18 rshd[8834]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:43:28 client-18 xinetd[3390]: EXIT: shell status=0 pid=8833 duration=0(sec)
May 17 14:43:28 client-18 xinetd[3390]: START: shell pid=8869 from=192.168.4.5
May 17 14:43:28 client-18 rshd[8870]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:43:28 client-18 xinetd[3390]: EXIT: shell status=0 pid=8869 duration=0(sec)
May 17 14:43:28 client-18 xinetd[3390]: START: shell pid=8906 from=192.168.4.5
May 17 14:43:28 client-18 rshd[8907]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 13: open chmod 0 \|x\| write close =============================================== 14:43:28 \(1305668608\)");echo XXRETCODE:$?'
May 17 14:43:28 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 13: open chmod 0 |x| write close =============================================== 14:43:28 (1305668608)
May 17 14:43:28 client-18 xinetd[3390]: EXIT: shell status=0 pid=8906 duration=0(sec)
May 17 14:43:28 client-18 xinetd[3390]: START: shell pid=8941 from=192.168.4.5
May 17 14:43:28 client-18 rshd[8942]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 14:43:29 client-18 xinetd[3390]: EXIT: shell status=0 pid=8941 duration=1(sec)
May 17 14:43:47 client-18 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail.
May 17 14:43:47 client-18 kernel: LustreError: Skipped 6 previous similar messages
May 17 14:43:47 client-18 kernel: Lustre: 3860:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74c949f8 to 0x62e5a8d74c94f38
May 17 14:43:47 client-18 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import
May 17 14:43:47 client-18 kernel: Lustre: Skipped 6 previous similar messages
May 17 14:43:48 client-18 xinetd[3390]: START: shell pid=8985 from=192.168.4.5
May 17 14:43:48 client-18 rshd[8986]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 14:43:48 client-18 xinetd[3390]: EXIT: shell status=0 pid=8985 duration=0(sec)
May 17 14:43:48 client-18 xinetd[3390]: START: shell pid=9019 from=192.168.4.5
May 17 14:43:48 client-18 rshd[9020]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:43:48 client-18 xinetd[3390]: EXIT: shell status=0 pid=9019 duration=0(sec)
May 17 14:43:48 client-18 xinetd[3390]: START: shell pid=9055 from=192.168.4.5
May 17 14:43:48 client-18 rshd[9056]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:43:48 client-18 xinetd[3390]: EXIT: shell status=0 pid=9055 duration=0(sec)
May 17 14:43:48 client-18 xinetd[3390]: START: shell pid=9092 from=192.168.4.5
May 17 14:43:48 client-18 rshd[9093]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 14: open\(O_CREAT\), unlink \|X\| close ============================================ 14:43:48 \(1305668628\)");echo XXRETCODE:$?'
May 17 14:43:48 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 14: open(O_CREAT), unlink |X| close ============================================ 14:43:48 (1305668628)
May 17 14:43:48 client-18 xinetd[3390]: EXIT: shell status=0 pid=9092 duration=0(sec)
May 17 14:43:48 client-18 xinetd[3390]: START: shell pid=9127 from=192.168.4.5
May 17 14:43:48 client-18 rshd[9128]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 14:43:49 client-18 xinetd[3390]: EXIT: shell status=0 pid=9127 duration=1(sec)
May 17 14:44:07 client-18 kernel: Lustre: 3860:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74c94f38 to 0x62e5a8d74c95494
May 17 14:44:08 client-18 xinetd[3390]: START: shell pid=9167 from=192.168.4.5
May 17 14:44:08 client-18 rshd[9168]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 14:44:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=9167 duration=0(sec)
May 17 14:44:08 client-18 xinetd[3390]: START: shell pid=9201 from=192.168.4.5
May 17 14:44:08 client-18 rshd[9202]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:44:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=9201 duration=0(sec)
May 17 14:44:08 client-18 xinetd[3390]: START: shell pid=9237 from=192.168.4.5
May 17 14:44:08 client-18 rshd[9238]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:44:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=9237 duration=0(sec)
May 17 14:44:08 client-18 xinetd[3390]: START: shell pid=9274 from=192.168.4.5
May 17 14:44:08 client-18 rshd[9275]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 15: open\(O_CREAT\), unlink \|X\|  touch new, close ================================ 14:44:08 \(1305668648\)");echo XXRETCODE:$?'
May 17 14:44:08 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 15: open(O_CREAT), unlink |X| touch new, close ================================ 14:44:08 (1305668648)
May 17 14:44:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=9274 duration=0(sec)
May 17 14:44:08 client-18 xinetd[3390]: START: shell pid=9310 from=192.168.4.5
May 17 14:44:08 client-18 rshd[9311]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 14:44:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=9310 duration=0(sec)
May 17 14:44:27 client-18 kernel: Lustre: 3860:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74c95494 to 0x62e5a8d74c959f0
May 17 14:44:27 client-18 xinetd[3390]: START: shell pid=9350 from=192.168.4.5
May 17 14:44:27 client-18 rshd[9351]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 14:44:28 client-18 xinetd[3390]: EXIT: shell status=0 pid=9350 duration=1(sec)
May 17 14:44:28 client-18 xinetd[3390]: START: shell pid=9384 from=192.168.4.5
May 17 14:44:28 client-18 rshd[9385]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:44:28 client-18 xinetd[3390]: EXIT: shell status=0 pid=9384 duration=0(sec)
May 17 14:44:28 client-18 xinetd[3390]: START: shell pid=9420 from=192.168.4.5
May 17 14:44:28 client-18 rshd[9421]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:44:28 client-18 xinetd[3390]: EXIT: shell status=0 pid=9420 duration=0(sec)
May 17 14:44:28 client-18 xinetd[3390]: START: shell pid=9457 from=192.168.4.5
May 17 14:44:28 client-18 rshd[9458]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 16: \|X\| open\(O_CREAT\), unlink, touch new,  unlink new ========================== 14:44:28 \(1305668668\)");echo XXRETCODE:$?'
May 17 14:44:28 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 16: |X| open(O_CREAT), unlink, touch new, unlink new ========================== 14:44:28 (1305668668)
May 17 14:44:28 client-18 xinetd[3390]: EXIT: shell status=0 pid=9457 duration=0(sec)
May 17 14:44:28 client-18 xinetd[3390]: START: shell pid=9492 from=192.168.4.5
May 17 14:44:28 client-18 rshd[9493]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 14:44:28 client-18 xinetd[3390]: EXIT: shell status=0 pid=9492 duration=0(sec)
May 17 14:44:47 client-18 xinetd[3390]: START: shell pid=9531 from=192.168.4.5
May 17 14:44:47 client-18 rshd[9532]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 14:44:48 client-18 xinetd[3390]: EXIT: shell status=0 pid=9531 duration=1(sec)
May 17 14:44:48 client-18 xinetd[3390]: START: shell pid=9566 from=192.168.4.5
May 17 14:44:48 client-18 rshd[9567]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:44:48 client-18 xinetd[3390]: EXIT: shell status=0 pid=9566 duration=0(sec)
May 17 14:44:48 client-18 xinetd[3390]: START: shell pid=9604 from=192.168.4.5
May 17 14:44:48 client-18 rshd[9605]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:44:48 client-18 kernel: Lustre: 3861:0:(import.c:529:import_select_connection()) MGC192.168.4.128@o2ib: tried all connections, increasing latency to 6s
May 17 14:44:48 client-18 kernel: Lustre: 3860:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74c959f0 to 0x62e5a8d74c962c5
May 17 14:44:48 client-18 xinetd[3390]: EXIT: shell status=0 pid=9604 duration=0(sec)
May 17 14:44:48 client-18 xinetd[3390]: START: shell pid=9642 from=192.168.4.5
May 17 14:44:48 client-18 rshd[9643]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 17: \|X\| open\(O_CREAT\), \|replay\| close ========================================== 14:44:48 \(1305668688\)");echo XXRETCODE:$?'
May 17 14:44:48 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 17: |X| open(O_CREAT), |replay| close ========================================== 14:44:48 (1305668688)
May 17 14:44:48 client-18 xinetd[3390]: EXIT: shell status=0 pid=9642 duration=0(sec)
May 17 14:44:48 client-18 xinetd[3390]: START: shell pid=9677 from=192.168.4.5
May 17 14:44:48 client-18 rshd[9678]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 14:44:48 client-18 xinetd[3390]: EXIT: shell status=0 pid=9677 duration=0(sec)
May 17 14:45:03 client-18 kernel: LustreError: 6048:0:(client.c:1057:ptlrpc_import_delay_req()) @@@ IMP_INVALID  req@ffff8102fa59c800 x1368947485953065/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:/ffffffff/ffffffff rc 0/-1
May 17 14:45:03 client-18 kernel: LustreError: 6048:0:(client.c:1057:ptlrpc_import_delay_req()) Skipped 19 previous similar messages
May 17 14:45:07 client-18 xinetd[3390]: START: shell pid=9720 from=192.168.4.5
May 17 14:45:07 client-18 rshd[9721]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 14:45:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=9720 duration=1(sec)
May 17 14:45:08 client-18 xinetd[3390]: START: shell pid=9754 from=192.168.4.5
May 17 14:45:08 client-18 rshd[9755]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:45:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=9754 duration=0(sec)
May 17 14:45:08 client-18 xinetd[3390]: START: shell pid=9790 from=192.168.4.5
May 17 14:45:08 client-18 rshd[9791]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:45:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=9790 duration=0(sec)
May 17 14:45:08 client-18 xinetd[3390]: START: shell pid=9827 from=192.168.4.5
May 17 14:45:08 client-18 rshd[9828]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 18: \|X\| open\(O_CREAT\), unlink, touch new, close, touch, unlink ================= 14:45:08 \(1305668708\)");echo XXRETCODE:$?'
May 17 14:45:08 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 18: |X| open(O_CREAT), unlink, touch new, close, touch, unlink ================= 14:45:08 (1305668708)
May 17 14:45:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=9827 duration=0(sec)
May 17 14:45:08 client-18 xinetd[3390]: START: shell pid=9862 from=192.168.4.5
May 17 14:45:08 client-18 rshd[9863]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 14:45:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=9862 duration=0(sec)
May 17 14:45:18 client-18 kernel: Lustre: 3861:0:(import.c:529:import_select_connection()) MGC192.168.4.128@o2ib: tried all connections, increasing latency to 11s
May 17 14:45:27 client-18 xinetd[3390]: START: shell pid=9901 from=192.168.4.5
May 17 14:45:27 client-18 rshd[9902]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 14:45:28 client-18 xinetd[3390]: EXIT: shell status=0 pid=9901 duration=1(sec)
May 17 14:45:28 client-18 xinetd[3390]: START: shell pid=9935 from=192.168.4.5
May 17 14:45:28 client-18 rshd[9936]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:45:28 client-18 xinetd[3390]: EXIT: shell status=0 pid=9935 duration=0(sec)
May 17 14:45:28 client-18 xinetd[3390]: START: shell pid=9971 from=192.168.4.5
May 17 14:45:28 client-18 rshd[9972]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:45:28 client-18 xinetd[3390]: EXIT: shell status=0 pid=9971 duration=0(sec)
May 17 14:45:28 client-18 xinetd[3390]: START: shell pid=10008 from=192.168.4.5
May 17 14:45:28 client-18 rshd[10009]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 19: \|X\| mcreate, open, write, rename =========================================== 14:45:28 \(1305668728\)");echo XXRETCODE:$?'
May 17 14:45:28 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 19: |X| mcreate, open, write, rename =========================================== 14:45:28 (1305668728)
May 17 14:45:28 client-18 xinetd[3390]: EXIT: shell status=0 pid=10008 duration=0(sec)
May 17 14:45:28 client-18 xinetd[3390]: START: shell pid=10043 from=192.168.4.5
May 17 14:45:28 client-18 rshd[10044]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 14:45:28 client-18 xinetd[3390]: EXIT: shell status=0 pid=10043 duration=0(sec)
May 17 14:45:38 client-18 kernel: Lustre: 3861:0:(import.c:529:import_select_connection()) MGC192.168.4.128@o2ib: tried all connections, increasing latency to 16s
May 17 14:45:47 client-18 xinetd[3390]: START: shell pid=10082 from=192.168.4.5
May 17 14:45:47 client-18 rshd[10083]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 14:45:48 client-18 xinetd[3390]: EXIT: shell status=0 pid=10082 duration=1(sec)
May 17 14:45:48 client-18 xinetd[3390]: START: shell pid=10116 from=192.168.4.5
May 17 14:45:48 client-18 rshd[10117]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:45:48 client-18 xinetd[3390]: EXIT: shell status=0 pid=10116 duration=0(sec)
May 17 14:45:48 client-18 xinetd[3390]: START: shell pid=10152 from=192.168.4.5
May 17 14:45:48 client-18 rshd[10153]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:45:48 client-18 xinetd[3390]: EXIT: shell status=0 pid=10152 duration=0(sec)
May 17 14:45:48 client-18 xinetd[3390]: START: shell pid=10189 from=192.168.4.5
May 17 14:45:48 client-18 rshd[10190]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 20a: \|X\| open\(O_CREAT\), unlink, replay, close \(test mds_cleanup_orphans\) ======= 14:45:48 \(1305668748\)");echo XXRETCODE:$?'
May 17 14:45:48 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 20a: |X| open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ======= 14:45:48 (1305668748)
May 17 14:45:48 client-18 xinetd[3390]: EXIT: shell status=0 pid=10189 duration=0(sec)
May 17 14:45:48 client-18 xinetd[3390]: START: shell pid=10224 from=192.168.4.5
May 17 14:45:48 client-18 rshd[10225]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 14:45:48 client-18 xinetd[3390]: EXIT: shell status=0 pid=10224 duration=0(sec)
May 17 14:46:03 client-18 kernel: Lustre: 3861:0:(import.c:529:import_select_connection()) MGC192.168.4.128@o2ib: tried all connections, increasing latency to 21s
May 17 14:46:07 client-18 xinetd[3390]: START: shell pid=10263 from=192.168.4.5
May 17 14:46:07 client-18 rshd[10264]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 14:46:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=10263 duration=1(sec)
May 17 14:46:08 client-18 xinetd[3390]: START: shell pid=10297 from=192.168.4.5
May 17 14:46:08 client-18 rshd[10298]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:46:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=10297 duration=0(sec)
May 17 14:46:08 client-18 xinetd[3390]: START: shell pid=10333 from=192.168.4.5
May 17 14:46:08 client-18 rshd[10334]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:46:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=10333 duration=0(sec)
May 17 14:46:08 client-18 xinetd[3390]: START: shell pid=10370 from=192.168.4.5
May 17 14:46:08 client-18 rshd[10371]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 20b: write, unlink, eviction, replay, \(test mds_cleanup_orphans\) =============== 14:46:08 \(1305668768\)");echo XXRETCODE:$?'
May 17 14:46:08 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 20b: write, unlink, eviction, replay, (test mds_cleanup_orphans) =============== 14:46:08 (1305668768)
May 17 14:46:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=10370 duration=0(sec)
May 17 14:46:28 client-18 xinetd[3390]: START: shell pid=10409 from=192.168.4.5
May 17 14:46:28 client-18 rshd[10410]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 14:46:28 client-18 xinetd[3390]: EXIT: shell status=0 pid=10409 duration=0(sec)
May 17 14:46:33 client-18 xinetd[3390]: START: shell pid=10443 from=192.168.4.5
May 17 14:46:33 client-18 rshd[10444]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark before 3024924, after 3024924");echo XXRETCODE:$?'
May 17 14:46:33 client-18 kernel: Lustre: DEBUG MARKER: before 3024924, after 3024924
May 17 14:46:33 client-18 xinetd[3390]: EXIT: shell status=0 pid=10443 duration=0(sec)
May 17 14:46:33 client-18 xinetd[3390]: START: shell pid=10478 from=192.168.4.5
May 17 14:46:33 client-18 rshd[10479]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:46:33 client-18 xinetd[3390]: EXIT: shell status=0 pid=10478 duration=0(sec)
May 17 14:46:33 client-18 xinetd[3390]: START: shell pid=10514 from=192.168.4.5
May 17 14:46:33 client-18 rshd[10515]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:46:33 client-18 xinetd[3390]: EXIT: shell status=0 pid=10514 duration=0(sec)
May 17 14:46:33 client-18 kernel: Lustre: 3860:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74c962c5 to 0x62e5a8d74c97e55
May 17 14:46:33 client-18 xinetd[3390]: START: shell pid=10552 from=192.168.4.5
May 17 14:46:33 client-18 rshd[10553]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 20c: check that client eviction does not affect file content =================== 14:46:33 \(1305668793\)");echo XXRETCODE:$?'
May 17 14:46:33 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 20c: check that client eviction does not affect file content =================== 14:46:33 (1305668793)
May 17 14:46:33 client-18 xinetd[3390]: EXIT: shell status=0 pid=10552 duration=0(sec)
May 17 14:46:36 client-18 xinetd[3390]: START: shell pid=10587 from=192.168.4.5
May 17 14:46:36 client-18 rshd[10588]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:46:36 client-18 xinetd[3390]: EXIT: shell status=0 pid=10587 duration=0(sec)
May 17 14:46:36 client-18 xinetd[3390]: START: shell pid=10623 from=192.168.4.5
May 17 14:46:36 client-18 rshd[10624]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:46:36 client-18 xinetd[3390]: EXIT: shell status=0 pid=10623 duration=0(sec)
May 17 14:46:36 client-18 xinetd[3390]: START: shell pid=10660 from=192.168.4.5
May 17 14:46:36 client-18 rshd[10661]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 21: \|X\| open\(O_CREAT\), unlink touch new, replay, close \(test mds_cleanup_orphans\) ====================================================================================================== 14:46:36 \(1305668796\)");echo XXRETCODE:$?'
May 17 14:46:36 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 21: |X| open(O_CREAT), unlink touch new, replay, close (test mds_cleanup_orphans) ====================================================================================================== 14:46:36 (1305668796)
May 17 14:46:36 client-18 xinetd[3390]: EXIT: shell status=0 pid=10660 duration=0(sec)
May 17 14:46:36 client-18 xinetd[3390]: START: shell pid=10697 from=192.168.4.5
May 17 14:46:36 client-18 rshd[10698]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 14:46:36 client-18 xinetd[3390]: EXIT: shell status=0 pid=10697 duration=0(sec)
May 17 14:46:55 client-18 xinetd[3390]: START: shell pid=10736 from=192.168.4.5
May 17 14:46:55 client-18 rshd[10737]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 14:46:56 client-18 xinetd[3390]: EXIT: shell status=0 pid=10736 duration=1(sec)
May 17 14:46:56 client-18 xinetd[3390]: START: shell pid=10770 from=192.168.4.5
May 17 14:46:56 client-18 rshd[10771]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:46:56 client-18 xinetd[3390]: EXIT: shell status=0 pid=10770 duration=0(sec)
May 17 14:46:56 client-18 xinetd[3390]: START: shell pid=10806 from=192.168.4.5
May 17 14:46:56 client-18 rshd[10807]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:46:56 client-18 xinetd[3390]: EXIT: shell status=0 pid=10806 duration=0(sec)
May 17 14:46:56 client-18 xinetd[3390]: START: shell pid=10843 from=192.168.4.5
May 17 14:46:56 client-18 rshd[10844]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 22: open\(O_CREAT\), \|X\| unlink, replay, close \(test mds_cleanup_orphans\) ======== 14:46:56 \(1305668816\)");echo XXRETCODE:$?'
May 17 14:46:56 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 22: open(O_CREAT), |X| unlink, replay, close (test mds_cleanup_orphans) ======== 14:46:56 (1305668816)
May 17 14:46:56 client-18 xinetd[3390]: EXIT: shell status=0 pid=10843 duration=0(sec)
May 17 14:46:56 client-18 xinetd[3390]: START: shell pid=10878 from=192.168.4.5
May 17 14:46:56 client-18 rshd[10879]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 14:46:56 client-18 xinetd[3390]: EXIT: shell status=0 pid=10878 duration=0(sec)
May 17 14:47:09 client-18 kernel: Lustre: 3859:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1368947485953297 sent from lustre-MDT0000-mdc-ffff810317273c00 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305668803] [real_sent 1305668803] [current 1305668829] [deadline 26s] [delay 0s]  req@ffff8103037c9800 x1368947485953297/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 192/192 e 0 to 1 dl 1305668829 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
May 17 14:47:09 client-18 kernel: Lustre: 3859:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 31 previous similar messages
May 17 14:47:15 client-18 xinetd[3390]: START: shell pid=10917 from=192.168.4.5
May 17 14:47:15 client-18 rshd[10918]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 14:47:16 client-18 kernel: Lustre: lustre-MDT0000-mdc-ffff810317273c00: Connection restored to service lustre-MDT0000 using nid 192.168.4.128@o2ib.
May 17 14:47:16 client-18 kernel: Lustre: Skipped 20 previous similar messages
May 17 14:47:16 client-18 xinetd[3390]: EXIT: shell status=0 pid=10917 duration=1(sec)
May 17 14:47:16 client-18 xinetd[3390]: START: shell pid=10951 from=192.168.4.5
May 17 14:47:16 client-18 rshd[10952]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:47:16 client-18 xinetd[3390]: EXIT: shell status=0 pid=10951 duration=0(sec)
May 17 14:47:16 client-18 xinetd[3390]: START: shell pid=10987 from=192.168.4.5
May 17 14:47:16 client-18 rshd[10988]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:47:16 client-18 xinetd[3390]: EXIT: shell status=0 pid=10987 duration=0(sec)
May 17 14:47:16 client-18 xinetd[3390]: START: shell pid=11024 from=192.168.4.5
May 17 14:47:16 client-18 rshd[11025]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 23: open\(O_CREAT\), \|X\| unlink touch new, replay, close \(test mds_cleanup_orphans\) ====================================================================================================== 14:47:16 \(1305668836\)");echo XXRETCODE:$?'
May 17 14:47:16 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 23: open(O_CREAT), |X| unlink touch new, replay, close (test mds_cleanup_orphans) ====================================================================================================== 14:47:16 (1305668836)
May 17 14:47:16 client-18 xinetd[3390]: EXIT: shell status=0 pid=11024 duration=0(sec)
May 17 14:47:16 client-18 xinetd[3390]: START: shell pid=11059 from=192.168.4.5
May 17 14:47:16 client-18 rshd[11060]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 14:47:16 client-18 xinetd[3390]: EXIT: shell status=0 pid=11059 duration=0(sec)
May 17 14:47:35 client-18 xinetd[3390]: START: shell pid=11102 from=192.168.4.5
May 17 14:47:35 client-18 rshd[11103]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 14:47:36 client-18 xinetd[3390]: EXIT: shell status=0 pid=11102 duration=1(sec)
May 17 14:47:36 client-18 xinetd[3390]: START: shell pid=11136 from=192.168.4.5
May 17 14:47:36 client-18 rshd[11137]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:47:36 client-18 xinetd[3390]: EXIT: shell status=0 pid=11136 duration=0(sec)
May 17 14:47:36 client-18 xinetd[3390]: START: shell pid=11172 from=192.168.4.5
May 17 14:47:36 client-18 rshd[11173]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:47:36 client-18 xinetd[3390]: EXIT: shell status=0 pid=11172 duration=0(sec)
May 17 14:47:36 client-18 xinetd[3390]: START: shell pid=11209 from=192.168.4.5
May 17 14:47:36 client-18 rshd[11210]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 24: open\(O_CREAT\), replay, unlink, close \(test mds_cleanup_orphans\) ============ 14:47:36 \(1305668856\)");echo XXRETCODE:$?'
May 17 14:47:36 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 24: open(O_CREAT), replay, unlink, close (test mds_cleanup_orphans) ============ 14:47:36 (1305668856)
May 17 14:47:36 client-18 xinetd[3390]: EXIT: shell status=0 pid=11209 duration=0(sec)
May 17 14:47:36 client-18 xinetd[3390]: START: shell pid=11244 from=192.168.4.5
May 17 14:47:36 client-18 rshd[11245]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 14:47:36 client-18 xinetd[3390]: EXIT: shell status=0 pid=11244 duration=0(sec)
May 17 14:47:40 client-18 kernel: Lustre: 3861:0:(import.c:529:import_select_connection()) MGC192.168.4.128@o2ib: tried all connections, increasing latency to 21s
May 17 14:47:40 client-18 kernel: Lustre: 3861:0:(import.c:529:import_select_connection()) Skipped 1 previous similar message
May 17 14:47:55 client-18 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The obd_ping operation failed with -107
May 17 14:47:55 client-18 kernel: LustreError: Skipped 16 previous similar messages
May 17 14:47:55 client-18 kernel: Lustre: lustre-MDT0000-mdc-ffff810317273c00: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete.
May 17 14:47:55 client-18 kernel: Lustre: Skipped 12 previous similar messages
May 17 14:47:56 client-18 xinetd[3390]: START: shell pid=11283 from=192.168.4.5
May 17 14:47:56 client-18 rshd[11284]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 14:47:56 client-18 kernel: LustreError: 3860:0:(mdc_request.c:642:mdc_replay_open()) @@@ Can't properly replay without open data.  req@ffff8103021fcc00 x1368947485447692/t17185310206(17185310206) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 528/544 e 0 to 0 dl 1305668902 ref 2 fl Interpret:RP/ffffffff/ffffffff rc 301/-1
May 17 14:47:56 client-18 kernel: LustreError: 3860:0:(mdc_request.c:642:mdc_replay_open()) Skipped 558 previous similar messages
May 17 14:47:56 client-18 xinetd[3390]: EXIT: shell status=0 pid=11283 duration=0(sec)
May 17 14:47:56 client-18 xinetd[3390]: START: shell pid=11317 from=192.168.4.5
May 17 14:47:56 client-18 rshd[11318]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:47:56 client-18 xinetd[3390]: EXIT: shell status=0 pid=11317 duration=0(sec)
May 17 14:47:56 client-18 xinetd[3390]: START: shell pid=11353 from=192.168.4.5
May 17 14:47:56 client-18 rshd[11354]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:47:56 client-18 xinetd[3390]: EXIT: shell status=0 pid=11353 duration=0(sec)
May 17 14:47:56 client-18 xinetd[3390]: START: shell pid=11390 from=192.168.4.5
May 17 14:47:56 client-18 rshd[11391]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 25: open\(O_CREAT\), unlink, replay, close \(test mds_cleanup_orphans\) ============ 14:47:56 \(1305668876\)");echo XXRETCODE:$?'
May 17 14:47:56 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 25: open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ============ 14:47:56 (1305668876)
May 17 14:47:56 client-18 xinetd[3390]: EXIT: shell status=0 pid=11390 duration=0(sec)
May 17 14:47:56 client-18 xinetd[3390]: START: shell pid=11425 from=192.168.4.5
May 17 14:47:56 client-18 rshd[11426]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 14:47:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=11425 duration=1(sec)
May 17 14:48:16 client-18 xinetd[3390]: START: shell pid=11464 from=192.168.4.5
May 17 14:48:16 client-18 rshd[11465]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 14:48:16 client-18 xinetd[3390]: EXIT: shell status=0 pid=11464 duration=0(sec)
May 17 14:48:16 client-18 xinetd[3390]: START: shell pid=11498 from=192.168.4.5
May 17 14:48:16 client-18 rshd[11499]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:48:16 client-18 xinetd[3390]: EXIT: shell status=0 pid=11498 duration=0(sec)
May 17 14:48:16 client-18 xinetd[3390]: START: shell pid=11534 from=192.168.4.5
May 17 14:48:16 client-18 rshd[11535]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:48:16 client-18 xinetd[3390]: EXIT: shell status=0 pid=11534 duration=0(sec)
May 17 14:48:16 client-18 xinetd[3390]: START: shell pid=11571 from=192.168.4.5
May 17 14:48:17 client-18 rshd[11572]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 26: \|X\| open\(O_CREAT\), unlink two, close one, replay, close one \(test mds_cleanup_orphans\) ====================================================================================================== 14:48:16 \(1305668896\)");echo XXRETCODE:$?'
May 17 14:48:17 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 26: |X| open(O_CREAT), unlink two, close one, replay, close one (test mds_cleanup_orphans) ====================================================================================================== 14:48:16 (1305668896)
May 17 14:48:17 client-18 xinetd[3390]: EXIT: shell status=0 pid=11571 duration=1(sec)
May 17 14:48:17 client-18 xinetd[3390]: START: shell pid=11606 from=192.168.4.5
May 17 14:48:17 client-18 rshd[11607]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 14:48:17 client-18 xinetd[3390]: EXIT: shell status=0 pid=11606 duration=0(sec)
May 17 14:48:36 client-18 xinetd[3390]: START: shell pid=11645 from=192.168.4.5
May 17 14:48:36 client-18 rshd[11646]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 14:48:36 client-18 xinetd[3390]: EXIT: shell status=0 pid=11645 duration=0(sec)
May 17 14:48:36 client-18 xinetd[3390]: START: shell pid=11679 from=192.168.4.5
May 17 14:48:36 client-18 rshd[11680]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:48:36 client-18 xinetd[3390]: EXIT: shell status=0 pid=11679 duration=0(sec)
May 17 14:48:36 client-18 xinetd[3390]: START: shell pid=11715 from=192.168.4.5
May 17 14:48:36 client-18 rshd[11716]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:48:37 client-18 xinetd[3390]: EXIT: shell status=0 pid=11715 duration=1(sec)
May 17 14:48:37 client-18 xinetd[3390]: START: shell pid=11752 from=192.168.4.5
May 17 14:48:37 client-18 rshd[11753]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 27: \|X\| open\(O_CREAT\), unlink two, replay, close two \(test mds_cleanup_orphans\) ====================================================================================================== 14:48:37 \(1305668917\)");echo XXRETCODE:$?'
May 17 14:48:37 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 27: |X| open(O_CREAT), unlink two, replay, close two (test mds_cleanup_orphans) ====================================================================================================== 14:48:37 (1305668917)
May 17 14:48:37 client-18 xinetd[3390]: EXIT: shell status=0 pid=11752 duration=0(sec)
May 17 14:48:37 client-18 xinetd[3390]: START: shell pid=11787 from=192.168.4.5
May 17 14:48:37 client-18 rshd[11788]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 14:48:37 client-18 xinetd[3390]: EXIT: shell status=0 pid=11787 duration=0(sec)
May 17 14:48:56 client-18 xinetd[3390]: START: shell pid=11830 from=192.168.4.5
May 17 14:48:56 client-18 rshd[11831]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 14:48:56 client-18 xinetd[3390]: EXIT: shell status=0 pid=11830 duration=0(sec)
May 17 14:48:56 client-18 xinetd[3390]: START: shell pid=11864 from=192.168.4.5
May 17 14:48:56 client-18 rshd[11865]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:48:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=11864 duration=1(sec)
May 17 14:48:57 client-18 xinetd[3390]: START: shell pid=11900 from=192.168.4.5
May 17 14:48:57 client-18 rshd[11901]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:48:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=11900 duration=0(sec)
May 17 14:48:57 client-18 xinetd[3390]: START: shell pid=11937 from=192.168.4.5
May 17 14:48:57 client-18 rshd[11938]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 28: open\(O_CREAT\), \|X\| unlink two, close one, replay, close one \(test mds_cleanup_orphans\) ====================================================================================================== 14:48:57 \(1305668937\)");echo XXRETCODE:$?'
May 17 14:48:57 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 28: open(O_CREAT), |X| unlink two, close one, replay, close one (test mds_cleanup_orphans) ====================================================================================================== 14:48:57 (1305668937)
May 17 14:48:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=11937 duration=0(sec)
May 17 14:48:57 client-18 xinetd[3390]: START: shell pid=11972 from=192.168.4.5
May 17 14:48:57 client-18 rshd[11973]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 14:48:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=11972 duration=0(sec)
May 17 14:49:10 client-18 kernel: Lustre: 3861:0:(import.c:529:import_select_connection()) MGC192.168.4.128@o2ib: tried all connections, increasing latency to 21s
May 17 14:49:10 client-18 kernel: Lustre: 3861:0:(import.c:529:import_select_connection()) Skipped 2 previous similar messages
May 17 14:49:16 client-18 xinetd[3390]: START: shell pid=12011 from=192.168.4.5
May 17 14:49:16 client-18 rshd[12012]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 14:49:16 client-18 xinetd[3390]: EXIT: shell status=0 pid=12011 duration=0(sec)
May 17 14:49:17 client-18 xinetd[3390]: START: shell pid=12045 from=192.168.4.5
May 17 14:49:17 client-18 rshd[12046]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:49:17 client-18 xinetd[3390]: EXIT: shell status=0 pid=12045 duration=0(sec)
May 17 14:49:17 client-18 xinetd[3390]: START: shell pid=12081 from=192.168.4.5
May 17 14:49:17 client-18 rshd[12082]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:49:17 client-18 xinetd[3390]: EXIT: shell status=0 pid=12081 duration=0(sec)
May 17 14:49:17 client-18 xinetd[3390]: START: shell pid=12118 from=192.168.4.5
May 17 14:49:17 client-18 rshd[12119]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 29: open\(O_CREAT\), \|X\| unlink two, replay, close two \(test mds_cleanup_orphans\) ====================================================================================================== 14:49:17 \(1305668957\)");echo XXRETCODE:$?'
May 17 14:49:17 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 29: open(O_CREAT), |X| unlink two, replay, close two (test mds_cleanup_orphans) ====================================================================================================== 14:49:17 (1305668957)
May 17 14:49:17 client-18 xinetd[3390]: EXIT: shell status=0 pid=12118 duration=0(sec)
May 17 14:49:17 client-18 xinetd[3390]: START: shell pid=12153 from=192.168.4.5
May 17 14:49:17 client-18 rshd[12154]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 14:49:17 client-18 xinetd[3390]: EXIT: shell status=0 pid=12153 duration=0(sec)
May 17 14:49:36 client-18 xinetd[3390]: START: shell pid=12192 from=192.168.4.5
May 17 14:49:36 client-18 rshd[12193]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 14:49:37 client-18 xinetd[3390]: EXIT: shell status=0 pid=12192 duration=1(sec)
May 17 14:49:37 client-18 xinetd[3390]: START: shell pid=12226 from=192.168.4.5
May 17 14:49:37 client-18 rshd[12227]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:49:37 client-18 xinetd[3390]: EXIT: shell status=0 pid=12226 duration=0(sec)
May 17 14:49:37 client-18 xinetd[3390]: START: shell pid=12262 from=192.168.4.5
May 17 14:49:37 client-18 rshd[12263]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:49:37 client-18 xinetd[3390]: EXIT: shell status=0 pid=12262 duration=0(sec)
May 17 14:49:37 client-18 xinetd[3390]: START: shell pid=12299 from=192.168.4.5
May 17 14:49:37 client-18 rshd[12300]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 30: open\(O_CREAT\) two, unlink two, replay, close two \(test mds_cleanup_orphans\) ====================================================================================================== 14:49:37 \(1305668977\)");echo XXRETCODE:$?'
May 17 14:49:37 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 30: open(O_CREAT) two, unlink two, replay, close two (test mds_cleanup_orphans) ====================================================================================================== 14:49:37 (1305668977)
May 17 14:49:37 client-18 xinetd[3390]: EXIT: shell status=0 pid=12299 duration=0(sec)
May 17 14:49:37 client-18 xinetd[3390]: START: shell pid=12334 from=192.168.4.5
May 17 14:49:37 client-18 rshd[12335]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 14:49:37 client-18 xinetd[3390]: EXIT: shell status=0 pid=12334 duration=0(sec)
May 17 14:49:56 client-18 xinetd[3390]: START: shell pid=12377 from=192.168.4.5
May 17 14:49:57 client-18 rshd[12378]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 14:50:00 client-18 xinetd[3390]: EXIT: shell status=0 pid=12377 duration=4(sec)
May 17 14:50:00 client-18 xinetd[3390]: START: shell pid=12411 from=192.168.4.5
May 17 14:50:00 client-18 rshd[12412]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:50:01 client-18 xinetd[3390]: EXIT: shell status=0 pid=12411 duration=1(sec)
May 17 14:50:01 client-18 xinetd[3390]: START: shell pid=12447 from=192.168.4.5
May 17 14:50:01 client-18 rshd[12448]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:50:01 client-18 xinetd[3390]: EXIT: shell status=0 pid=12447 duration=0(sec)
May 17 14:50:01 client-18 xinetd[3390]: START: shell pid=12485 from=192.168.4.5
May 17 14:50:01 client-18 rshd[12486]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 31: open\(O_CREAT\) two, unlink one, \|X\| unlink one, close two \(test mds_cleanup_orphans\) ====================================================================================================== 14:50:01 \(1305669001\)");echo XXRETCODE:$?'
May 17 14:50:01 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 31: open(O_CREAT) two, unlink one, |X| unlink one, close two (test mds_cleanup_orphans) ====================================================================================================== 14:50:01 (1305669001)
May 17 14:50:01 client-18 xinetd[3390]: EXIT: shell status=0 pid=12485 duration=0(sec)
May 17 14:50:01 client-18 xinetd[3390]: START: shell pid=12520 from=192.168.4.5
May 17 14:50:01 client-18 rshd[12521]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 14:50:01 client-18 xinetd[3390]: EXIT: shell status=0 pid=12520 duration=0(sec)
May 17 14:50:20 client-18 xinetd[3390]: START: shell pid=12559 from=192.168.4.5
May 17 14:50:20 client-18 rshd[12560]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 14:50:21 client-18 xinetd[3390]: EXIT: shell status=0 pid=12559 duration=1(sec)
May 17 14:50:21 client-18 xinetd[3390]: START: shell pid=12593 from=192.168.4.5
May 17 14:50:21 client-18 rshd[12594]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:50:21 client-18 xinetd[3390]: EXIT: shell status=0 pid=12593 duration=0(sec)
May 17 14:50:21 client-18 xinetd[3390]: START: shell pid=12629 from=192.168.4.5
May 17 14:50:21 client-18 rshd[12630]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:50:21 client-18 xinetd[3390]: EXIT: shell status=0 pid=12629 duration=0(sec)
May 17 14:50:21 client-18 xinetd[3390]: START: shell pid=12666 from=192.168.4.5
May 17 14:50:21 client-18 rshd[12667]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 32: close\(\) notices client eviction\; close\(\) after client eviction ============= 14:50:21 \(1305669021\)");echo XXRETCODE:$?'
May 17 14:50:21 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 32: close() notices client eviction; close() after client eviction ============= 14:50:21 (1305669021)
May 17 14:50:21 client-18 xinetd[3390]: EXIT: shell status=0 pid=12666 duration=0(sec)
May 17 14:50:23 client-18 xinetd[3390]: START: shell pid=12701 from=192.168.4.5
May 17 14:50:23 client-18 rshd[12702]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:50:23 client-18 xinetd[3390]: EXIT: shell status=0 pid=12701 duration=0(sec)
May 17 14:50:23 client-18 xinetd[3390]: START: shell pid=12737 from=192.168.4.5
May 17 14:50:23 client-18 rshd[12738]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:50:23 client-18 xinetd[3390]: EXIT: shell status=0 pid=12737 duration=0(sec)
May 17 14:50:24 client-18 xinetd[3390]: START: shell pid=12774 from=192.168.4.5
May 17 14:50:24 client-18 rshd[12775]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark  SKIP: replay-single test_33a skipping ALWAYS excluded test 33a");echo XXRETCODE:$?'
May 17 14:50:24 client-18 kernel: Lustre: DEBUG MARKER: SKIP: replay-single test_33a skipping ALWAYS excluded test 33a
May 17 14:50:24 client-18 xinetd[3390]: EXIT: shell status=0 pid=12774 duration=0(sec)
May 17 14:50:24 client-18 xinetd[3390]: START: shell pid=12809 from=192.168.4.5
May 17 14:50:24 client-18 rshd[12810]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark  SKIP: replay-single test_33b skipping ALWAYS excluded test 33b");echo XXRETCODE:$?'
May 17 14:50:24 client-18 kernel: Lustre: DEBUG MARKER: SKIP: replay-single test_33b skipping ALWAYS excluded test 33b
May 17 14:50:24 client-18 xinetd[3390]: EXIT: shell status=0 pid=12809 duration=0(sec)
May 17 14:50:24 client-18 xinetd[3390]: START: shell pid=12844 from=192.168.4.5
May 17 14:50:24 client-18 rshd[12845]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 34: abort recovery before client does replay \(test mds_cleanup_orphans\) ======== 14:50:24 \(1305669024\)");echo XXRETCODE:$?'
May 17 14:50:24 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 34: abort recovery before client does replay (test mds_cleanup_orphans) ======== 14:50:24 (1305669024)
May 17 14:50:24 client-18 xinetd[3390]: EXIT: shell status=0 pid=12844 duration=0(sec)
May 17 14:50:24 client-18 xinetd[3390]: START: shell pid=12879 from=192.168.4.5
May 17 14:50:24 client-18 rshd[12880]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 14:50:24 client-18 xinetd[3390]: EXIT: shell status=0 pid=12879 duration=0(sec)
May 17 14:50:35 client-18 xinetd[3390]: START: shell pid=12918 from=192.168.4.5
May 17 14:50:35 client-18 rshd[12919]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 14:50:35 client-18 kernel: LustreError: 167-0: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
May 17 14:50:35 client-18 kernel: LustreError: 12951:0:(lmv_obd.c:1198:lmv_statfs()) can't stat MDS #0 (lustre-MDT0000-mdc-ffff810317273c00), error -4
May 17 14:50:35 client-18 kernel: LustreError: 12952:0:(client.c:1057:ptlrpc_import_delay_req()) @@@ IMP_INVALID  req@ffff8102eb16d800 x1368947485953726/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:23/10 lens 360/1048 e 0 to 0 dl 0 ref 2 fl Rpc:/ffffffff/ffffffff rc 0/-1
May 17 14:50:35 client-18 kernel: LustreError: 12952:0:(client.c:1057:ptlrpc_import_delay_req()) Skipped 64 previous similar messages
May 17 14:50:35 client-18 kernel: LustreError: 12952:0:(file.c:155:ll_close_inode_openhandle()) inode 144115305986177651 mdc close failed: rc = -108
May 17 14:50:35 client-18 kernel: LustreError: 12951:0:(llite_lib.c:1370:ll_statfs_internal()) md_statfs fails: rc = -4
May 17 14:50:35 client-18 xinetd[3390]: EXIT: shell status=0 pid=12918 duration=0(sec)
May 17 14:50:36 client-18 xinetd[3390]: START: shell pid=12953 from=192.168.4.5
May 17 14:50:36 client-18 rshd[12954]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 14:50:36 client-18 xinetd[3390]: EXIT: shell status=0 pid=12953 duration=0(sec)
May 17 14:50:36 client-18 xinetd[3390]: START: shell pid=12987 from=192.168.4.5
May 17 14:50:36 client-18 rshd[12988]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:50:36 client-18 xinetd[3390]: EXIT: shell status=0 pid=12987 duration=0(sec)
May 17 14:50:36 client-18 xinetd[3390]: START: shell pid=13023 from=192.168.4.5
May 17 14:50:36 client-18 rshd[13024]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:50:36 client-18 xinetd[3390]: EXIT: shell status=0 pid=13023 duration=0(sec)
May 17 14:50:36 client-18 xinetd[3390]: START: shell pid=13060 from=192.168.4.5
May 17 14:50:36 client-18 rshd[13061]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 35: test recovery from llog for unlink op ====================================== 14:50:36 \(1305669036\)");echo XXRETCODE:$?'
May 17 14:50:37 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 35: test recovery from llog for unlink op ====================================== 14:50:36 (1305669036)
May 17 14:50:37 client-18 xinetd[3390]: EXIT: shell status=0 pid=13060 duration=1(sec)
May 17 14:50:48 client-18 xinetd[3390]: START: shell pid=13095 from=192.168.4.5
May 17 14:50:48 client-18 rshd[13096]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 14:50:48 client-18 kernel: LustreError: 167-0: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
May 17 14:50:48 client-18 kernel: LustreError: 13129:0:(lmv_obd.c:1198:lmv_statfs()) can't stat MDS #0 (lustre-MDT0000-mdc-ffff810317273c00), error -4
May 17 14:50:48 client-18 kernel: LustreError: 13129:0:(llite_lib.c:1370:ll_statfs_internal()) md_statfs fails: rc = -4
May 17 14:50:48 client-18 xinetd[3390]: EXIT: shell status=0 pid=13095 duration=0(sec)
May 17 14:50:55 client-18 xinetd[3390]: START: shell pid=13131 from=192.168.4.5
May 17 14:50:55 client-18 rshd[13132]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 14:50:55 client-18 xinetd[3390]: EXIT: shell status=0 pid=13131 duration=0(sec)
May 17 14:50:55 client-18 xinetd[3390]: START: shell pid=13165 from=192.168.4.5
May 17 14:50:55 client-18 rshd[13166]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:50:55 client-18 xinetd[3390]: EXIT: shell status=0 pid=13165 duration=0(sec)
May 17 14:50:55 client-18 xinetd[3390]: START: shell pid=13201 from=192.168.4.5
May 17 14:50:55 client-18 rshd[13202]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:50:55 client-18 xinetd[3390]: EXIT: shell status=0 pid=13201 duration=0(sec)
May 17 14:50:55 client-18 xinetd[3390]: START: shell pid=13238 from=192.168.4.5
May 17 14:50:55 client-18 rshd[13239]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 36: don\'t resend cancel ======================================================== 14:50:55 \(1305669055\)");echo XXRETCODE:$?'
May 17 14:50:55 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 36: don't resend cancel ======================================================== 14:50:55 (1305669055)
May 17 14:50:55 client-18 xinetd[3390]: EXIT: shell status=0 pid=13238 duration=0(sec)
May 17 14:50:55 client-18 xinetd[3390]: START: shell pid=13273 from=192.168.4.5
May 17 14:50:55 client-18 rshd[13274]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 14:50:56 client-18 xinetd[3390]: EXIT: shell status=0 pid=13273 duration=1(sec)
May 17 14:51:14 client-18 xinetd[3390]: START: shell pid=13316 from=192.168.4.5
May 17 14:51:14 client-18 rshd[13317]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:51:14 client-18 xinetd[3390]: EXIT: shell status=0 pid=13316 duration=0(sec)
May 17 14:51:14 client-18 xinetd[3390]: START: shell pid=13352 from=192.168.4.5
May 17 14:51:14 client-18 rshd[13353]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:51:14 client-18 xinetd[3390]: EXIT: shell status=0 pid=13352 duration=0(sec)
May 17 14:51:14 client-18 xinetd[3390]: START: shell pid=13389 from=192.168.4.5
May 17 14:51:14 client-18 rshd[13390]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl set_param debug_mb=150");echo XXRETCODE:$?'
May 17 14:51:14 client-18 xinetd[3390]: EXIT: shell status=0 pid=13389 duration=0(sec)
May 17 14:51:14 client-18 xinetd[3390]: START: shell pid=13424 from=192.168.4.5
May 17 14:51:14 client-18 rshd[13425]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl set_param debug=-1;");echo XXRETCODE:$?'
May 17 14:51:14 client-18 kernel: Lustre: 13458:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release.
May 17 14:51:14 client-18 kernel: Lustre: 13458:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message
May 17 14:51:14 client-18 xinetd[3390]: EXIT: shell status=0 pid=13424 duration=0(sec)
May 17 14:51:14 client-18 xinetd[3390]: START: shell pid=13459 from=192.168.4.5
May 17 14:51:14 client-18 rshd[13460]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 37: abort recovery before client does replay \(test mds_cleanup_orphans for directories\) ====================================================================================================== 14:51:14 \(1305669074\)");echo XXRETCODE:$?'
May 17 14:51:15 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 37: abort recovery before client does replay (test mds_cleanup_orphans for directories) ====================================================================================================== 14:51:14 (1305669074)
May 17 14:51:15 client-18 xinetd[3390]: EXIT: shell status=0 pid=13459 duration=1(sec)
May 17 14:51:22 client-18 kernel: Lustre: 3861:0:(import.c:529:import_select_connection()) lustre-MDT0000-mdc-ffff810317273c00: tried all connections, increasing latency to 6s
May 17 14:51:22 client-18 kernel: Lustre: 3861:0:(import.c:529:import_select_connection()) Skipped 4 previous similar messages
May 17 14:51:22 client-18 xinetd[3390]: START: shell pid=13494 from=192.168.4.5
May 17 14:51:22 client-18 rshd[13495]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 14:51:23 client-18 xinetd[3390]: EXIT: shell status=0 pid=13494 duration=1(sec)
May 17 14:51:32 client-18 kernel: LustreError: 167-0: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
May 17 14:51:32 client-18 xinetd[3390]: START: shell pid=13534 from=192.168.4.5
May 17 14:51:32 client-18 rshd[13535]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 14:51:32 client-18 xinetd[3390]: EXIT: shell status=0 pid=13534 duration=0(sec)
May 17 14:51:34 client-18 xinetd[3390]: START: shell pid=13568 from=192.168.4.5
May 17 14:51:34 client-18 rshd[13569]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 14:51:34 client-18 xinetd[3390]: EXIT: shell status=0 pid=13568 duration=0(sec)
May 17 14:51:34 client-18 xinetd[3390]: START: shell pid=13602 from=192.168.4.5
May 17 14:51:34 client-18 rshd[13603]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:51:34 client-18 xinetd[3390]: EXIT: shell status=0 pid=13602 duration=0(sec)
May 17 14:51:34 client-18 xinetd[3390]: START: shell pid=13638 from=192.168.4.5
May 17 14:51:34 client-18 rshd[13639]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:51:34 client-18 xinetd[3390]: EXIT: shell status=0 pid=13638 duration=0(sec)
May 17 14:51:34 client-18 xinetd[3390]: START: shell pid=13675 from=192.168.4.5
May 17 14:51:34 client-18 rshd[13676]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl set_param debug_mb=24");echo XXRETCODE:$?'
May 17 14:51:34 client-18 xinetd[3390]: EXIT: shell status=0 pid=13675 duration=0(sec)
May 17 14:51:34 client-18 xinetd[3390]: START: shell pid=13710 from=192.168.4.5
May 17 14:51:34 client-18 rshd[13711]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl set_param debug=\"inode super warning dlmtrace error emerg ha rpctrace vfstrace config console\";");echo XXRETCODE:$?'
May 17 14:51:34 client-18 xinetd[3390]: EXIT: shell status=0 pid=13710 duration=0(sec)
May 17 14:51:34 client-18 xinetd[3390]: START: shell pid=13745 from=192.168.4.5
May 17 14:51:34 client-18 rshd[13746]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 38: test recovery from unlink llog \(test llog_gen_rec\) ========================= 14:51:34 \(1305669094\)");echo XXRETCODE:$?'
May 17 14:51:34 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 38: test recovery from unlink llog (test llog_gen_rec) ========================= 14:51:34 (1305669094)
May 17 14:51:34 client-18 xinetd[3390]: EXIT: shell status=0 pid=13745 duration=0(sec)
May 17 14:51:36 client-18 xinetd[3390]: START: shell pid=13780 from=192.168.4.5
May 17 14:51:36 client-18 rshd[13781]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 14:51:36 client-18 xinetd[3390]: EXIT: shell status=0 pid=13780 duration=0(sec)
May 17 14:51:55 client-18 xinetd[3390]: START: shell pid=13820 from=192.168.4.5
May 17 14:51:55 client-18 rshd[13821]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 14:51:59 client-18 xinetd[3390]: EXIT: shell status=0 pid=13820 duration=4(sec)
May 17 14:52:01 client-18 xinetd[3390]: START: shell pid=13854 from=192.168.4.5
May 17 14:52:02 client-18 rshd[13855]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:52:02 client-18 xinetd[3390]: EXIT: shell status=0 pid=13854 duration=1(sec)
May 17 14:52:02 client-18 xinetd[3390]: START: shell pid=13890 from=192.168.4.5
May 17 14:52:02 client-18 rshd[13891]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:52:02 client-18 xinetd[3390]: EXIT: shell status=0 pid=13890 duration=0(sec)
May 17 14:52:02 client-18 xinetd[3390]: START: shell pid=13927 from=192.168.4.5
May 17 14:52:02 client-18 rshd[13928]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 39: test recovery from unlink llog \(test llog_gen_rec\) ========================= 14:52:02 \(1305669122\)");echo XXRETCODE:$?'
May 17 14:52:02 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 39: test recovery from unlink llog (test llog_gen_rec) ========================= 14:52:02 (1305669122)
May 17 14:52:02 client-18 xinetd[3390]: EXIT: shell status=0 pid=13927 duration=0(sec)
May 17 14:52:03 client-18 xinetd[3390]: START: shell pid=13965 from=192.168.4.5
May 17 14:52:03 client-18 rshd[13966]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 14:52:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=13965 duration=0(sec)
May 17 14:52:04 client-18 kernel: Lustre: 3860:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74c97e55 to 0x62e5a8d74cb5d96
May 17 14:52:04 client-18 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import
May 17 14:52:04 client-18 kernel: Lustre: Skipped 4 previous similar messages
May 17 14:52:23 client-18 xinetd[3390]: START: shell pid=14006 from=192.168.4.5
May 17 14:52:23 client-18 rshd[14007]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 14:52:23 client-18 xinetd[3390]: EXIT: shell status=0 pid=14006 duration=0(sec)
May 17 14:52:24 client-18 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail.
May 17 14:52:24 client-18 kernel: LustreError: Skipped 5 previous similar messages
May 17 14:52:24 client-18 kernel: Lustre: 3860:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74cb5d96 to 0x62e5a8d74cbcc62
May 17 14:52:25 client-18 xinetd[3390]: START: shell pid=14045 from=192.168.4.5
May 17 14:52:26 client-18 rshd[14046]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:52:26 client-18 xinetd[3390]: EXIT: shell status=0 pid=14045 duration=1(sec)
May 17 14:52:26 client-18 xinetd[3390]: START: shell pid=14081 from=192.168.4.5
May 17 14:52:26 client-18 rshd[14082]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:52:26 client-18 xinetd[3390]: EXIT: shell status=0 pid=14081 duration=0(sec)
May 17 14:52:26 client-18 xinetd[3390]: START: shell pid=14118 from=192.168.4.5
May 17 14:52:26 client-18 rshd[14119]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 40: cause recovery in ptlrpc, ensure IO continues ============================== 14:52:26 \(1305669146\)");echo XXRETCODE:$?'
May 17 14:52:26 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 40: cause recovery in ptlrpc, ensure IO continues ============================== 14:52:26 (1305669146)
May 17 14:52:26 client-18 xinetd[3390]: EXIT: shell status=0 pid=14118 duration=0(sec)
May 17 14:52:44 client-18 kernel: Lustre: 3860:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74cbcc62 to 0x62e5a8d74cbcd5e
May 17 14:53:05 client-18 xinetd[3390]: START: shell pid=14156 from=192.168.4.5
May 17 14:53:05 client-18 rshd[14157]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:53:05 client-18 xinetd[3390]: EXIT: shell status=0 pid=14156 duration=0(sec)
May 17 14:53:05 client-18 xinetd[3390]: START: shell pid=14192 from=192.168.4.5
May 17 14:53:05 client-18 rshd[14193]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:53:05 client-18 xinetd[3390]: EXIT: shell status=0 pid=14192 duration=0(sec)
May 17 14:53:05 client-18 xinetd[3390]: START: shell pid=14229 from=192.168.4.5
May 17 14:53:05 client-18 rshd[14230]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 41: read from a valid osc while other oscs are invalid ========================= 14:53:05 \(1305669185\)");echo XXRETCODE:$?'
May 17 14:53:05 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 41: read from a valid osc while other oscs are invalid ========================= 14:53:05 (1305669185)
May 17 14:53:05 client-18 xinetd[3390]: EXIT: shell status=0 pid=14229 duration=0(sec)
May 17 14:53:06 client-18 xinetd[3390]: START: shell pid=14264 from=192.168.4.5
May 17 14:53:06 client-18 rshd[14265]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:53:06 client-18 xinetd[3390]: EXIT: shell status=0 pid=14264 duration=0(sec)
May 17 14:53:06 client-18 xinetd[3390]: START: shell pid=14300 from=192.168.4.5
May 17 14:53:06 client-18 rshd[14301]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:53:07 client-18 xinetd[3390]: EXIT: shell status=0 pid=14300 duration=1(sec)
May 17 14:53:07 client-18 xinetd[3390]: START: shell pid=14337 from=192.168.4.5
May 17 14:53:07 client-18 rshd[14338]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 42: recovery after ost failure ================================================= 14:53:07 \(1305669187\)");echo XXRETCODE:$?'
May 17 14:53:07 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 42: recovery after ost failure ================================================= 14:53:07 (1305669187)
May 17 14:53:07 client-18 xinetd[3390]: EXIT: shell status=0 pid=14337 duration=0(sec)
May 17 14:53:08 client-18 xinetd[3390]: START: shell pid=14372 from=192.168.4.5
May 17 14:53:08 client-18 rshd[14373]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 14:53:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=14372 duration=0(sec)
May 17 14:54:01 client-18 xinetd[3390]: START: shell pid=14415 from=192.168.4.5
May 17 14:54:01 client-18 rshd[14416]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl set_param debug=\"inode super warning dlmtrace error emerg ha rpctrace vfstrace config console\";");echo XXRETCODE:$?'
May 17 14:54:01 client-18 xinetd[3390]: EXIT: shell status=0 pid=14415 duration=0(sec)
May 17 14:54:02 client-18 xinetd[3390]: START: shell pid=14450 from=192.168.4.5
May 17 14:54:02 client-18 rshd[14451]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:54:02 client-18 xinetd[3390]: EXIT: shell status=0 pid=14450 duration=0(sec)
May 17 14:54:02 client-18 xinetd[3390]: START: shell pid=14486 from=192.168.4.5
May 17 14:54:02 client-18 rshd[14487]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:54:02 client-18 xinetd[3390]: EXIT: shell status=0 pid=14486 duration=0(sec)
May 17 14:54:02 client-18 xinetd[3390]: START: shell pid=14523 from=192.168.4.5
May 17 14:54:02 client-18 rshd[14524]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 43: mds osc import failure during recovery\; don\'t LBUG ========================= 14:54:02 \(1305669242\)");echo XXRETCODE:$?'
May 17 14:54:02 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 43: mds osc import failure during recovery; don't LBUG ========================= 14:54:02 (1305669242)
May 17 14:54:02 client-18 xinetd[3390]: EXIT: shell status=0 pid=14523 duration=0(sec)
May 17 14:54:02 client-18 xinetd[3390]: START: shell pid=14558 from=192.168.4.5
May 17 14:54:02 client-18 rshd[14559]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 14:54:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=14558 duration=1(sec)
May 17 14:54:22 client-18 xinetd[3390]: START: shell pid=14597 from=192.168.4.5
May 17 14:54:22 client-18 rshd[14598]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 14:54:22 client-18 xinetd[3390]: EXIT: shell status=0 pid=14597 duration=0(sec)
May 17 14:54:24 client-18 kernel: Lustre: 3860:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74cbcd5e to 0x62e5a8d74ccc29d
May 17 14:54:32 client-18 xinetd[3390]: START: shell pid=14635 from=192.168.4.5
May 17 14:54:32 client-18 rshd[14636]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 14:54:32 client-18 xinetd[3390]: EXIT: shell status=0 pid=14635 duration=0(sec)
May 17 14:54:32 client-18 xinetd[3390]: START: shell pid=14671 from=192.168.4.5
May 17 14:54:32 client-18 rshd[14672]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 14:54:32 client-18 xinetd[3390]: EXIT: shell status=0 pid=14671 duration=0(sec)
May 17 14:54:33 client-18 xinetd[3390]: START: shell pid=14708 from=192.168.4.5
May 17 14:54:33 client-18 rshd[14709]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 44a: race in target handle connect ============================================= 14:54:32 \(1305669272\)");echo XXRETCODE:$?'
May 17 14:54:33 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 44a: race in target handle connect ============================================= 14:54:32 (1305669272)
May 17 14:54:33 client-18 xinetd[3390]: EXIT: shell status=0 pid=14708 duration=0(sec)
May 17 15:00:14 client-18 xinetd[3390]: START: shell pid=14763 from=192.168.4.5
May 17 15:00:14 client-18 rshd[14764]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 15:00:14 client-18 xinetd[3390]: EXIT: shell status=0 pid=14763 duration=0(sec)
May 17 15:00:14 client-18 xinetd[3390]: START: shell pid=14799 from=192.168.4.5
May 17 15:00:14 client-18 rshd[14800]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 15:00:14 client-18 xinetd[3390]: EXIT: shell status=0 pid=14799 duration=0(sec)
May 17 15:00:14 client-18 xinetd[3390]: START: shell pid=14836 from=192.168.4.5
May 17 15:00:14 client-18 rshd[14837]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 44b: race in target handle connect ============================================= 15:00:14 \(1305669614\)");echo XXRETCODE:$?'
May 17 15:00:14 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 44b: race in target handle connect ============================================= 15:00:14 (1305669614)
May 17 15:00:14 client-18 xinetd[3390]: EXIT: shell status=0 pid=14836 duration=0(sec)
May 17 15:07:44 client-18 xinetd[3390]: START: shell pid=14901 from=192.168.4.5
May 17 15:07:44 client-18 rshd[14902]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 15:07:44 client-18 xinetd[3390]: EXIT: shell status=0 pid=14901 duration=0(sec)
May 17 15:07:44 client-18 xinetd[3390]: START: shell pid=14937 from=192.168.4.5
May 17 15:07:44 client-18 rshd[14938]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 15:07:44 client-18 xinetd[3390]: EXIT: shell status=0 pid=14937 duration=0(sec)
May 17 15:07:44 client-18 xinetd[3390]: START: shell pid=14974 from=192.168.4.5
May 17 15:07:44 client-18 rshd[14975]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 44c: race in target handle connect ============================================= 15:07:44 \(1305670064\)");echo XXRETCODE:$?'
May 17 15:07:44 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 44c: race in target handle connect ============================================= 15:07:44 (1305670064)
May 17 15:07:44 client-18 xinetd[3390]: EXIT: shell status=0 pid=14974 duration=0(sec)
May 17 15:07:44 client-18 xinetd[3390]: START: shell pid=15009 from=192.168.4.5
May 17 15:07:44 client-18 rshd[15010]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 15:07:44 client-18 xinetd[3390]: EXIT: shell status=0 pid=15009 duration=0(sec)
May 17 15:07:54 client-18 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The obd_ping operation failed with -107
May 17 15:07:54 client-18 kernel: LustreError: Skipped 21 previous similar messages
May 17 15:07:54 client-18 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail.
May 17 15:07:54 client-18 kernel: LustreError: Skipped 2 previous similar messages
May 17 15:07:54 client-18 kernel: Lustre: lustre-MDT0000-mdc-ffff810317273c00: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete.
May 17 15:07:54 client-18 kernel: Lustre: 3860:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74ccc29d to 0x62e5a8d74cccb17
May 17 15:07:54 client-18 kernel: Lustre: Skipped 16 previous similar messages
May 17 15:07:54 client-18 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import
May 17 15:07:54 client-18 kernel: Lustre: Skipped 3 previous similar messages
May 17 15:07:54 client-18 kernel: Lustre: MGC192.168.4.128@o2ib: Connection restored to service MGS using nid 192.168.4.128@o2ib.
May 17 15:07:54 client-18 kernel: Lustre: Skipped 22 previous similar messages
May 17 15:07:54 client-18 kernel: LustreError: 167-0: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
May 17 15:07:55 client-18 xinetd[3390]: START: shell pid=15051 from=192.168.4.5
May 17 15:07:55 client-18 rshd[15052]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 15:07:55 client-18 xinetd[3390]: EXIT: shell status=0 pid=15051 duration=0(sec)
May 17 15:08:05 client-18 xinetd[3390]: START: shell pid=15087 from=192.168.4.5
May 17 15:08:05 client-18 rshd[15088]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 15:08:05 client-18 xinetd[3390]: EXIT: shell status=0 pid=15087 duration=0(sec)
May 17 15:08:16 client-18 kernel: Lustre: 3859:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1368947485955474 sent from MGC192.168.4.128@o2ib to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305670089] [real_sent 1305670089] [current 1305670096] [deadline 7s] [delay 0s]  req@ffff810306309800 x1368947485955474/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 192/192 e 0 to 1 dl 1305670096 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
May 17 15:08:16 client-18 kernel: Lustre: 3859:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 46 previous similar messages
May 17 15:08:24 client-18 xinetd[3390]: START: shell pid=15121 from=192.168.4.5
May 17 15:08:24 client-18 rshd[15122]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 15:08:26 client-18 kernel: Lustre: 3861:0:(import.c:529:import_select_connection()) MGC192.168.4.128@o2ib: tried all connections, increasing latency to 6s
May 17 15:08:26 client-18 kernel: Lustre: 3861:0:(import.c:529:import_select_connection()) Skipped 4 previous similar messages
May 17 15:08:26 client-18 kernel: Lustre: 3860:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74cccb17 to 0x62e5a8d74cccd0f
May 17 15:08:31 client-18 xinetd[3390]: EXIT: shell status=0 pid=15121 duration=7(sec)
May 17 15:08:31 client-18 xinetd[3390]: START: shell pid=15158 from=192.168.4.5
May 17 15:08:31 client-18 rshd[15159]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 15:08:31 client-18 xinetd[3390]: EXIT: shell status=0 pid=15158 duration=0(sec)
May 17 15:08:31 client-18 xinetd[3390]: START: shell pid=15194 from=192.168.4.5
May 17 15:08:31 client-18 rshd[15195]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 15:08:32 client-18 xinetd[3390]: EXIT: shell status=0 pid=15194 duration=1(sec)
May 17 15:08:32 client-18 xinetd[3390]: START: shell pid=15231 from=192.168.4.5
May 17 15:08:32 client-18 rshd[15232]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 45: Handle failed close ======================================================== 15:08:32 \(1305670112\)");echo XXRETCODE:$?'
May 17 15:08:32 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 45: Handle failed close ======================================================== 15:08:32 (1305670112)
May 17 15:08:32 client-18 xinetd[3390]: EXIT: shell status=0 pid=15231 duration=0(sec)
May 17 15:08:33 client-18 xinetd[3390]: START: shell pid=15266 from=192.168.4.5
May 17 15:08:33 client-18 rshd[15267]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 15:08:33 client-18 xinetd[3390]: EXIT: shell status=0 pid=15266 duration=0(sec)
May 17 15:08:33 client-18 xinetd[3390]: START: shell pid=15302 from=192.168.4.5
May 17 15:08:33 client-18 rshd[15303]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 15:08:33 client-18 xinetd[3390]: EXIT: shell status=0 pid=15302 duration=0(sec)
May 17 15:08:33 client-18 xinetd[3390]: START: shell pid=15339 from=192.168.4.5
May 17 15:08:33 client-18 rshd[15340]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 46: Don\'t leak file handle after open resend \(3325\) ============================ 15:08:33 \(1305670113\)");echo XXRETCODE:$?'
May 17 15:08:33 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 46: Don't leak file handle after open resend (3325) ============================ 15:08:33 (1305670113)
May 17 15:08:33 client-18 xinetd[3390]: EXIT: shell status=0 pid=15339 duration=0(sec)
May 17 15:09:18 client-18 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The obd_ping operation failed with -107
May 17 15:09:18 client-18 kernel: LustreError: Skipped 2 previous similar messages
May 17 15:09:18 client-18 kernel: Lustre: lustre-MDT0000-mdc-ffff810317273c00: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete.
May 17 15:09:18 client-18 kernel: Lustre: Skipped 1 previous similar message
May 17 15:09:20 client-18 xinetd[3390]: START: shell pid=15378 from=192.168.4.5
May 17 15:09:20 client-18 rshd[15379]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 15:09:24 client-18 kernel: Lustre: 3860:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1368947485955582 sent from MGC192.168.4.128@o2ib to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305670153] [real_sent 1305670153] [current 1305670164] [deadline 11s] [delay 0s]  req@ffff810302421c00 x1368947485955582/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 368/392 e 0 to 1 dl 1305670164 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
May 17 15:09:24 client-18 kernel: Lustre: 3860:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 4 previous similar messages
May 17 15:09:28 client-18 kernel: Lustre: 3861:0:(import.c:529:import_select_connection()) MGC192.168.4.128@o2ib: tried all connections, increasing latency to 11s
May 17 15:09:28 client-18 kernel: Lustre: 3861:0:(import.c:529:import_select_connection()) Skipped 1 previous similar message
May 17 15:09:28 client-18 kernel: Lustre: 3860:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74cccd0f to 0x62e5a8d74cccee4
May 17 15:09:28 client-18 kernel: Lustre: MGC192.168.4.128@o2ib: Connection restored to service MGS using nid 192.168.4.128@o2ib.
May 17 15:09:28 client-18 kernel: Lustre: Skipped 3 previous similar messages
May 17 15:09:28 client-18 xinetd[3390]: EXIT: shell status=0 pid=15378 duration=8(sec)
May 17 15:09:28 client-18 xinetd[3390]: START: shell pid=15413 from=192.168.4.5
May 17 15:09:28 client-18 rshd[15414]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 15:09:28 client-18 xinetd[3390]: EXIT: shell status=0 pid=15413 duration=0(sec)
May 17 15:09:29 client-18 xinetd[3390]: START: shell pid=15449 from=192.168.4.5
May 17 15:09:29 client-18 rshd[15450]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 15:09:29 client-18 xinetd[3390]: EXIT: shell status=0 pid=15449 duration=0(sec)
May 17 15:09:29 client-18 xinetd[3390]: START: shell pid=15486 from=192.168.4.5
May 17 15:09:29 client-18 rshd[15487]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 47: MDS-\>OSC failure during precreate cleanup \(2824\) =========================== 15:09:29 \(1305670169\)");echo XXRETCODE:$?'
May 17 15:09:29 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 47: MDS->OSC failure during precreate cleanup (2824) =========================== 15:09:29 (1305670169)
May 17 15:09:29 client-18 xinetd[3390]: EXIT: shell status=0 pid=15486 duration=0(sec)
May 17 15:09:43 client-18 xinetd[3390]: START: shell pid=15523 from=192.168.4.5
May 17 15:09:43 client-18 rshd[15524]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 15:09:48 client-18 xinetd[3390]: EXIT: shell status=0 pid=15523 duration=5(sec)
May 17 15:10:50 client-18 xinetd[3390]: START: shell pid=15561 from=192.168.4.5
May 17 15:10:50 client-18 rshd[15562]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 15:10:50 client-18 xinetd[3390]: EXIT: shell status=0 pid=15561 duration=0(sec)
May 17 15:10:50 client-18 xinetd[3390]: START: shell pid=15597 from=192.168.4.5
May 17 15:10:50 client-18 rshd[15598]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 15:10:50 client-18 xinetd[3390]: EXIT: shell status=0 pid=15597 duration=0(sec)
May 17 15:10:50 client-18 xinetd[3390]: START: shell pid=15634 from=192.168.4.5
May 17 15:10:50 client-18 rshd[15635]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 48: MDS-\>OSC failure during precreate cleanup \(2824\) =========================== 15:10:50 \(1305670250\)");echo XXRETCODE:$?'
May 17 15:10:50 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 48: MDS->OSC failure during precreate cleanup (2824) =========================== 15:10:50 (1305670250)
May 17 15:10:50 client-18 xinetd[3390]: EXIT: shell status=0 pid=15634 duration=0(sec)
May 17 15:10:50 client-18 xinetd[3390]: START: shell pid=15669 from=192.168.4.5
May 17 15:10:50 client-18 rshd[15670]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 15:10:50 client-18 xinetd[3390]: EXIT: shell status=0 pid=15669 duration=0(sec)
May 17 15:11:08 client-18 kernel: Lustre: 3860:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74cccee4 to 0x62e5a8d74ccdb0f
May 17 15:11:13 client-18 xinetd[3390]: START: shell pid=15716 from=192.168.4.5
May 17 15:11:13 client-18 rshd[15717]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 15:11:14 client-18 xinetd[3390]: EXIT: shell status=0 pid=15716 duration=1(sec)
May 17 15:11:14 client-18 xinetd[3390]: START: shell pid=15752 from=192.168.4.5
May 17 15:11:14 client-18 rshd[15753]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 15:11:14 client-18 xinetd[3390]: EXIT: shell status=0 pid=15752 duration=0(sec)
May 17 15:11:14 client-18 xinetd[3390]: START: shell pid=15789 from=192.168.4.5
May 17 15:11:14 client-18 rshd[15790]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 50: Double OSC recovery, don\'t LASSERT \(3812\) ================================== 15:11:14 \(1305670274\)");echo XXRETCODE:$?'
May 17 15:11:14 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 50: Double OSC recovery, don't LASSERT (3812) ================================== 15:11:14 (1305670274)
May 17 15:11:14 client-18 xinetd[3390]: EXIT: shell status=0 pid=15789 duration=0(sec)
May 17 15:11:20 client-18 xinetd[3390]: START: shell pid=15824 from=192.168.4.5
May 17 15:11:20 client-18 rshd[15825]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 15:11:20 client-18 xinetd[3390]: EXIT: shell status=0 pid=15824 duration=0(sec)
May 17 15:11:20 client-18 xinetd[3390]: START: shell pid=15860 from=192.168.4.5
May 17 15:11:20 client-18 rshd[15861]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 15:11:20 client-18 xinetd[3390]: EXIT: shell status=0 pid=15860 duration=0(sec)
May 17 15:11:20 client-18 xinetd[3390]: START: shell pid=15897 from=192.168.4.5
May 17 15:11:20 client-18 rshd[15898]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 52: time out lock replay \(3764\) ================================================ 15:11:20 \(1305670280\)");echo XXRETCODE:$?'
May 17 15:11:20 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 52: time out lock replay (3764) ================================================ 15:11:20 (1305670280)
May 17 15:11:20 client-18 xinetd[3390]: EXIT: shell status=0 pid=15897 duration=0(sec)
May 17 15:11:20 client-18 xinetd[3390]: START: shell pid=15932 from=192.168.4.5
May 17 15:11:20 client-18 rshd[15933]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 15:11:20 client-18 xinetd[3390]: EXIT: shell status=0 pid=15932 duration=0(sec)
May 17 15:11:34 client-18 kernel: Lustre: 3859:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1368947485955775 sent from lustre-MDT0000-mdc-ffff810317273c00 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305670263] [real_sent 1305670263] [current 1305670294] [deadline 31s] [delay 0s]  req@ffff810300294800 x1368947485955775/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 192/192 e 0 to 1 dl 1305670294 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
May 17 15:11:34 client-18 kernel: Lustre: 3859:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 4 previous similar messages
May 17 15:11:38 client-18 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The obd_ping operation failed with -107
May 17 15:11:38 client-18 kernel: LustreError: Skipped 6 previous similar messages
May 17 15:11:38 client-18 kernel: Lustre: 3860:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74ccdb0f to 0x62e5a8d74cce9a2
May 17 15:11:38 client-18 kernel: Lustre: MGC192.168.4.128@o2ib: Connection restored to service MGS using nid 192.168.4.128@o2ib.
May 17 15:11:38 client-18 kernel: Lustre: Skipped 4 previous similar messages
May 17 15:11:40 client-18 xinetd[3390]: START: shell pid=15975 from=192.168.4.5
May 17 15:11:40 client-18 rshd[15976]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 15:11:40 client-18 kernel: Lustre: lustre-MDT0000-mdc-ffff810317273c00: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete.
May 17 15:11:40 client-18 kernel: Lustre: Skipped 2 previous similar messages
May 17 15:11:40 client-18 xinetd[3390]: EXIT: shell status=0 pid=15975 duration=0(sec)
May 17 15:11:40 client-18 xinetd[3390]: START: shell pid=16009 from=192.168.4.5
May 17 15:11:40 client-18 rshd[16010]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 15:11:41 client-18 xinetd[3390]: EXIT: shell status=0 pid=16009 duration=1(sec)
May 17 15:11:41 client-18 xinetd[3390]: START: shell pid=16045 from=192.168.4.5
May 17 15:11:41 client-18 rshd[16046]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 15:11:41 client-18 xinetd[3390]: EXIT: shell status=0 pid=16045 duration=0(sec)
May 17 15:11:41 client-18 xinetd[3390]: START: shell pid=16082 from=192.168.4.5
May 17 15:11:41 client-18 rshd[16083]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 53a: \|X\| close request while two MDC requests in flight ======================== 15:11:41 \(1305670301\)");echo XXRETCODE:$?'
May 17 15:11:41 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 53a: |X| close request while two MDC requests in flight ======================== 15:11:41 (1305670301)
May 17 15:11:41 client-18 xinetd[3390]: EXIT: shell status=0 pid=16082 duration=0(sec)
May 17 15:12:01 client-18 kernel: LustreError: 15709:0:(client.c:1057:ptlrpc_import_delay_req()) @@@ IMP_INVALID  req@ffff810317273000 x1368947485955883/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:/ffffffff/ffffffff rc 0/-1
May 17 15:12:01 client-18 kernel: LustreError: 15709:0:(client.c:1057:ptlrpc_import_delay_req()) Skipped 6 previous similar messages
May 17 15:12:01 client-18 kernel: Lustre: 3860:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74cce9a2 to 0x62e5a8d74ccecff
May 17 15:12:02 client-18 xinetd[3390]: START: shell pid=16118 from=192.168.4.5
May 17 15:12:02 client-18 rshd[16119]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 15:12:02 client-18 xinetd[3390]: EXIT: shell status=0 pid=16118 duration=0(sec)
May 17 15:12:02 client-18 xinetd[3390]: START: shell pid=16152 from=192.168.4.5
May 17 15:12:02 client-18 rshd[16153]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 15:12:02 client-18 xinetd[3390]: EXIT: shell status=0 pid=16152 duration=0(sec)
May 17 15:12:02 client-18 xinetd[3390]: START: shell pid=16188 from=192.168.4.5
May 17 15:12:02 client-18 rshd[16189]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 15:12:02 client-18 xinetd[3390]: EXIT: shell status=0 pid=16188 duration=0(sec)
May 17 15:12:02 client-18 xinetd[3390]: START: shell pid=16225 from=192.168.4.5
May 17 15:12:02 client-18 rshd[16226]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 53b: \|X\| open request while two MDC requests in flight ========================= 15:12:02 \(1305670322\)");echo XXRETCODE:$?'
May 17 15:12:02 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 53b: |X| open request while two MDC requests in flight ========================= 15:12:02 (1305670322)
May 17 15:12:02 client-18 xinetd[3390]: EXIT: shell status=0 pid=16225 duration=0(sec)
May 17 15:12:23 client-18 kernel: Lustre: 3860:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74ccecff to 0x62e5a8d74ccf0b7
May 17 15:12:23 client-18 xinetd[3390]: START: shell pid=16261 from=192.168.4.5
May 17 15:12:23 client-18 rshd[16262]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 15:12:23 client-18 xinetd[3390]: EXIT: shell status=0 pid=16261 duration=0(sec)
May 17 15:12:24 client-18 xinetd[3390]: START: shell pid=16295 from=192.168.4.5
May 17 15:12:24 client-18 rshd[16296]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 15:12:24 client-18 xinetd[3390]: EXIT: shell status=0 pid=16295 duration=0(sec)
May 17 15:12:24 client-18 xinetd[3390]: START: shell pid=16331 from=192.168.4.5
May 17 15:12:24 client-18 rshd[16332]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 15:12:24 client-18 xinetd[3390]: EXIT: shell status=0 pid=16331 duration=0(sec)
May 17 15:12:24 client-18 xinetd[3390]: START: shell pid=16372 from=192.168.4.5
May 17 15:12:24 client-18 rshd[16373]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 53c: \|X\| open request and close request while two MDC requests in flight ======= 15:12:24 \(1305670344\)");echo XXRETCODE:$?'
May 17 15:12:24 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 53c: |X| open request and close request while two MDC requests in flight ======= 15:12:24 (1305670344)
May 17 15:12:24 client-18 xinetd[3390]: EXIT: shell status=0 pid=16372 duration=0(sec)
May 17 15:12:44 client-18 kernel: Lustre: 3860:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74ccf0b7 to 0x62e5a8d74ccf48b
May 17 15:12:51 client-18 xinetd[3390]: START: shell pid=16410 from=192.168.4.5
May 17 15:12:51 client-18 rshd[16411]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 15:12:51 client-18 xinetd[3390]: EXIT: shell status=0 pid=16410 duration=0(sec)
May 17 15:12:51 client-18 xinetd[3390]: START: shell pid=16446 from=192.168.4.5
May 17 15:12:51 client-18 rshd[16447]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 15:12:51 client-18 xinetd[3390]: EXIT: shell status=0 pid=16446 duration=0(sec)
May 17 15:12:52 client-18 xinetd[3390]: START: shell pid=16483 from=192.168.4.5
May 17 15:12:52 client-18 rshd[16484]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 53d: \|X\| close reply while two MDC requests in flight ========================== 15:12:51 \(1305670371\)");echo XXRETCODE:$?'
May 17 15:12:52 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 53d: |X| close reply while two MDC requests in flight ========================== 15:12:51 (1305670371)
May 17 15:12:52 client-18 xinetd[3390]: EXIT: shell status=0 pid=16483 duration=0(sec)
May 17 15:13:11 client-18 kernel: Lustre: 3860:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74ccf48b to 0x62e5a8d74ccf8d6
May 17 15:13:12 client-18 xinetd[3390]: START: shell pid=16519 from=192.168.4.5
May 17 15:13:12 client-18 rshd[16520]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 15:13:12 client-18 xinetd[3390]: EXIT: shell status=0 pid=16519 duration=0(sec)
May 17 15:13:12 client-18 xinetd[3390]: START: shell pid=16553 from=192.168.4.5
May 17 15:13:12 client-18 rshd[16554]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 15:13:12 client-18 xinetd[3390]: EXIT: shell status=0 pid=16553 duration=0(sec)
May 17 15:13:12 client-18 xinetd[3390]: START: shell pid=16589 from=192.168.4.5
May 17 15:13:12 client-18 rshd[16590]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 15:13:12 client-18 xinetd[3390]: EXIT: shell status=0 pid=16589 duration=0(sec)
May 17 15:13:12 client-18 xinetd[3390]: START: shell pid=16626 from=192.168.4.5
May 17 15:13:13 client-18 rshd[16627]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 53e: \|X\| open reply while two MDC requests in flight =========================== 15:13:12 \(1305670392\)");echo XXRETCODE:$?'
May 17 15:13:13 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 53e: |X| open reply while two MDC requests in flight =========================== 15:13:12 (1305670392)
May 17 15:13:13 client-18 xinetd[3390]: EXIT: shell status=0 pid=16626 duration=1(sec)
May 17 15:13:33 client-18 xinetd[3390]: START: shell pid=16663 from=192.168.4.5
May 17 15:13:33 client-18 rshd[16664]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 15:13:34 client-18 xinetd[3390]: EXIT: shell status=0 pid=16663 duration=1(sec)
May 17 15:13:34 client-18 xinetd[3390]: START: shell pid=16697 from=192.168.4.5
May 17 15:13:34 client-18 rshd[16698]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 15:13:34 client-18 xinetd[3390]: EXIT: shell status=0 pid=16697 duration=0(sec)
May 17 15:13:34 client-18 xinetd[3390]: START: shell pid=16733 from=192.168.4.5
May 17 15:13:34 client-18 rshd[16734]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 15:13:35 client-18 xinetd[3390]: EXIT: shell status=0 pid=16733 duration=1(sec)
May 17 15:13:35 client-18 xinetd[3390]: START: shell pid=16770 from=192.168.4.5
May 17 15:13:35 client-18 rshd[16771]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 53f: \|X\| open reply and close reply while two MDC requests in flight =========== 15:13:35 \(1305670415\)");echo XXRETCODE:$?'
May 17 15:13:35 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 53f: |X| open reply and close reply while two MDC requests in flight =========== 15:13:35 (1305670415)
May 17 15:13:35 client-18 xinetd[3390]: EXIT: shell status=0 pid=16770 duration=0(sec)
May 17 15:13:36 client-18 kernel: Lustre: 3860:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74ccf8d6 to 0x62e5a8d74ccffa5
May 17 15:13:56 client-18 kernel: LustreError: 15709:0:(client.c:1057:ptlrpc_import_delay_req()) @@@ IMP_INVALID  req@ffff810314ea4800 x1368947485956090/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:/ffffffff/ffffffff rc 0/-1
May 17 15:13:56 client-18 kernel: LustreError: 15709:0:(client.c:1057:ptlrpc_import_delay_req()) Skipped 5 previous similar messages
May 17 15:13:56 client-18 kernel: Lustre: 3860:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74ccffa5 to 0x62e5a8d74cd00af
May 17 15:13:59 client-18 xinetd[3390]: START: shell pid=16811 from=192.168.4.5
May 17 15:13:59 client-18 rshd[16812]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 15:13:59 client-18 xinetd[3390]: EXIT: shell status=0 pid=16811 duration=0(sec)
May 17 15:13:59 client-18 xinetd[3390]: START: shell pid=16847 from=192.168.4.5
May 17 15:13:59 client-18 rshd[16848]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 15:13:59 client-18 xinetd[3390]: EXIT: shell status=0 pid=16847 duration=0(sec)
May 17 15:13:59 client-18 xinetd[3390]: START: shell pid=16884 from=192.168.4.5
May 17 15:13:59 client-18 rshd[16885]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 53g: \|X\| drop open reply and close request while close and open are both in flight ====================================================================================================== 15:13:59 \(1305670439\)");echo XXRETCODE:$?'
May 17 15:13:59 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 53g: |X| drop open reply and close request while close and open are both in flight ====================================================================================================== 15:13:59 (1305670439)
May 17 15:13:59 client-18 xinetd[3390]: EXIT: shell status=0 pid=16884 duration=0(sec)
May 17 15:14:21 client-18 kernel: Lustre: 3860:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74cd00af to 0x62e5a8d74cd04c2
May 17 15:14:36 client-18 xinetd[3390]: START: shell pid=16924 from=192.168.4.5
May 17 15:14:36 client-18 rshd[16925]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 15:14:36 client-18 xinetd[3390]: EXIT: shell status=0 pid=16924 duration=0(sec)
May 17 15:14:36 client-18 xinetd[3390]: START: shell pid=16960 from=192.168.4.5
May 17 15:14:36 client-18 rshd[16961]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 15:14:36 client-18 xinetd[3390]: EXIT: shell status=0 pid=16960 duration=0(sec)
May 17 15:14:36 client-18 xinetd[3390]: START: shell pid=16997 from=192.168.4.5
May 17 15:14:36 client-18 rshd[16998]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 53h: \|X\| open request and close reply while two MDC requests in flight ========= 15:14:36 \(1305670476\)");echo XXRETCODE:$?'
May 17 15:14:36 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 53h: |X| open request and close reply while two MDC requests in flight ========= 15:14:36 (1305670476)
May 17 15:14:36 client-18 xinetd[3390]: EXIT: shell status=0 pid=16997 duration=0(sec)
May 17 15:14:58 client-18 kernel: Lustre: 3860:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74cd04c2 to 0x62e5a8d74cd08c7
May 17 15:15:17 client-18 xinetd[3390]: START: shell pid=17039 from=192.168.4.5
May 17 15:15:17 client-18 rshd[17040]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 15:15:17 client-18 xinetd[3390]: EXIT: shell status=0 pid=17039 duration=0(sec)
May 17 15:15:17 client-18 xinetd[3390]: START: shell pid=17075 from=192.168.4.5
May 17 15:15:17 client-18 rshd[17076]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 15:15:17 client-18 xinetd[3390]: EXIT: shell status=0 pid=17075 duration=0(sec)
May 17 15:15:17 client-18 xinetd[3390]: START: shell pid=17112 from=192.168.4.5
May 17 15:15:17 client-18 rshd[17113]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 55: let MDS_CHECK_RESENT return the original return code instead of 0 ========== 15:15:17 \(1305670517\)");echo XXRETCODE:$?'
May 17 15:15:17 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 55: let MDS_CHECK_RESENT return the original return code instead of 0 ========== 15:15:17 (1305670517)
May 17 15:15:17 client-18 xinetd[3390]: EXIT: shell status=0 pid=17112 duration=0(sec)
May 17 15:15:58 client-18 xinetd[3390]: START: shell pid=17147 from=192.168.4.5
May 17 15:15:58 client-18 rshd[17148]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 15:15:58 client-18 xinetd[3390]: EXIT: shell status=0 pid=17147 duration=0(sec)
May 17 15:15:58 client-18 xinetd[3390]: START: shell pid=17183 from=192.168.4.5
May 17 15:15:59 client-18 rshd[17184]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 15:15:59 client-18 xinetd[3390]: EXIT: shell status=0 pid=17183 duration=1(sec)
May 17 15:15:59 client-18 xinetd[3390]: START: shell pid=17220 from=192.168.4.5
May 17 15:15:59 client-18 rshd[17221]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 56: don\'t replay a symlink open request \(3440\) ================================= 15:15:59 \(1305670559\)");echo XXRETCODE:$?'
May 17 15:15:59 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 56: don't replay a symlink open request (3440) ================================= 15:15:59 (1305670559)
May 17 15:15:59 client-18 xinetd[3390]: EXIT: shell status=0 pid=17220 duration=0(sec)
May 17 15:15:59 client-18 xinetd[3390]: START: shell pid=17255 from=192.168.4.5
May 17 15:15:59 client-18 rshd[17256]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 15:15:59 client-18 xinetd[3390]: EXIT: shell status=0 pid=17255 duration=0(sec)
May 17 15:16:18 client-18 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The obd_ping operation failed with -107
May 17 15:16:18 client-18 kernel: LustreError: Skipped 12 previous similar messages
May 17 15:16:18 client-18 kernel: Lustre: lustre-MDT0000-mdc-ffff810317273c00: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete.
May 17 15:16:18 client-18 kernel: Lustre: 3860:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74cd08c7 to 0x62e5a8d74cd0cef
May 17 15:16:18 client-18 kernel: Lustre: Skipped 8 previous similar messages
May 17 15:16:18 client-18 kernel: Lustre: MGC192.168.4.128@o2ib: Connection restored to service MGS using nid 192.168.4.128@o2ib.
May 17 15:16:18 client-18 kernel: Lustre: Skipped 17 previous similar messages
May 17 15:16:19 client-18 xinetd[3390]: START: shell pid=17300 from=192.168.4.5
May 17 15:16:19 client-18 rshd[17301]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 15:16:19 client-18 xinetd[3390]: EXIT: shell status=0 pid=17300 duration=0(sec)
May 17 15:16:29 client-18 xinetd[3390]: START: shell pid=17336 from=192.168.4.5
May 17 15:16:29 client-18 rshd[17337]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 15:16:29 client-18 xinetd[3390]: EXIT: shell status=0 pid=17336 duration=0(sec)
May 17 15:16:29 client-18 xinetd[3390]: START: shell pid=17372 from=192.168.4.5
May 17 15:16:29 client-18 rshd[17373]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 15:16:29 client-18 xinetd[3390]: EXIT: shell status=0 pid=17372 duration=0(sec)
May 17 15:16:29 client-18 xinetd[3390]: START: shell pid=17409 from=192.168.4.5
May 17 15:16:29 client-18 rshd[17410]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 57: test recovery from llog for setattr op ===================================== 15:16:29 \(1305670589\)");echo XXRETCODE:$?'
May 17 15:16:29 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 57: test recovery from llog for setattr op ===================================== 15:16:29 (1305670589)
May 17 15:16:29 client-18 xinetd[3390]: EXIT: shell status=0 pid=17409 duration=0(sec)
May 17 15:16:29 client-18 xinetd[3390]: START: shell pid=17444 from=192.168.4.5
May 17 15:16:30 client-18 rshd[17445]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 15:16:30 client-18 xinetd[3390]: EXIT: shell status=0 pid=17444 duration=1(sec)
May 17 15:16:39 client-18 kernel: Lustre: 3859:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1368947485956313 sent from lustre-MDT0000-mdc-ffff810317273c00 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305670568] [real_sent 1305670568] [current 1305670599] [deadline 31s] [delay 0s]  req@ffff8103067c9000 x1368947485956313/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 192/192 e 0 to 1 dl 1305670599 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
May 17 15:16:39 client-18 kernel: Lustre: 3859:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 34 previous similar messages
May 17 15:16:48 client-18 kernel: Lustre: 3860:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74cd0cef to 0x62e5a8d74cd0f81
May 17 15:16:49 client-18 xinetd[3390]: START: shell pid=17484 from=192.168.4.5
May 17 15:16:49 client-18 rshd[17485]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 15:16:49 client-18 xinetd[3390]: EXIT: shell status=0 pid=17484 duration=0(sec)
May 17 15:16:50 client-18 xinetd[3390]: START: shell pid=17518 from=192.168.4.5
May 17 15:16:50 client-18 rshd[17519]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 15:16:50 client-18 xinetd[3390]: EXIT: shell status=0 pid=17518 duration=0(sec)
May 17 15:16:50 client-18 xinetd[3390]: START: shell pid=17554 from=192.168.4.5
May 17 15:16:51 client-18 rshd[17555]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 15:16:51 client-18 xinetd[3390]: EXIT: shell status=0 pid=17554 duration=1(sec)
May 17 15:16:51 client-18 xinetd[3390]: START: shell pid=17591 from=192.168.4.5
May 17 15:16:51 client-18 rshd[17592]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 58a: test recovery from llog for setattr op \(test llog_gen_rec\) ================ 15:16:51 \(1305670611\)");echo XXRETCODE:$?'
May 17 15:16:51 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 58a: test recovery from llog for setattr op (test llog_gen_rec) ================ 15:16:51 (1305670611)
May 17 15:16:51 client-18 xinetd[3390]: EXIT: shell status=0 pid=17591 duration=0(sec)
May 17 15:16:53 client-18 xinetd[3390]: START: shell pid=17628 from=192.168.4.5
May 17 15:16:53 client-18 rshd[17629]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 15:16:53 client-18 xinetd[3390]: EXIT: shell status=0 pid=17628 duration=0(sec)
May 17 15:17:13 client-18 kernel: Lustre: 3860:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74cd0f81 to 0x62e5a8d74ce2408
May 17 15:17:13 client-18 xinetd[3390]: START: shell pid=17668 from=192.168.4.5
May 17 15:17:13 client-18 rshd[17669]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 15:17:13 client-18 xinetd[3390]: EXIT: shell status=0 pid=17668 duration=0(sec)
May 17 15:17:20 client-18 xinetd[3390]: START: shell pid=17704 from=192.168.4.5
May 17 15:17:20 client-18 rshd[17705]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 15:17:20 client-18 xinetd[3390]: EXIT: shell status=0 pid=17704 duration=0(sec)
May 17 15:17:20 client-18 xinetd[3390]: START: shell pid=17740 from=192.168.4.5
May 17 15:17:20 client-18 rshd[17741]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 15:17:20 client-18 xinetd[3390]: EXIT: shell status=0 pid=17740 duration=0(sec)
May 17 15:17:20 client-18 xinetd[3390]: START: shell pid=17777 from=192.168.4.5
May 17 15:17:20 client-18 rshd[17778]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 58b: test replay of setxattr op ================================================ 15:17:20 \(1305670640\)");echo XXRETCODE:$?'
May 17 15:17:20 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 58b: test replay of setxattr op ================================================ 15:17:20 (1305670640)
May 17 15:17:20 client-18 xinetd[3390]: EXIT: shell status=0 pid=17777 duration=0(sec)
May 17 15:17:21 client-18 xinetd[3390]: START: shell pid=17812 from=192.168.4.5
May 17 15:17:21 client-18 rshd[17813]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 15:17:21 client-18 xinetd[3390]: EXIT: shell status=0 pid=17812 duration=0(sec)
May 17 15:17:40 client-18 kernel: Lustre: 3860:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74ce2408 to 0x62e5a8d74d00652
May 17 15:17:40 client-18 xinetd[3390]: START: shell pid=17856 from=192.168.4.5
May 17 15:17:40 client-18 rshd[17857]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 15:17:41 client-18 xinetd[3390]: EXIT: shell status=0 pid=17856 duration=1(sec)
May 17 15:17:41 client-18 xinetd[3390]: START: shell pid=17890 from=192.168.4.5
May 17 15:17:41 client-18 rshd[17891]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 15:17:41 client-18 xinetd[3390]: EXIT: shell status=0 pid=17890 duration=0(sec)
May 17 15:17:41 client-18 xinetd[3390]: START: shell pid=17926 from=192.168.4.5
May 17 15:17:41 client-18 rshd[17927]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 15:17:41 client-18 xinetd[3390]: EXIT: shell status=0 pid=17926 duration=0(sec)
May 17 15:17:41 client-18 xinetd[3390]: START: shell pid=17963 from=192.168.4.5
May 17 15:17:41 client-18 rshd[17964]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 58c: resend\/reconstruct setxattr op ============================================ 15:17:41 \(1305670661\)");echo XXRETCODE:$?'
May 17 15:17:41 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 58c: resend/reconstruct setxattr op ============================================ 15:17:41 (1305670661)
May 17 15:17:41 client-18 xinetd[3390]: EXIT: shell status=0 pid=17963 duration=0(sec)
May 17 15:19:04 client-18 xinetd[3390]: START: shell pid=18004 from=192.168.4.5
May 17 15:19:04 client-18 rshd[18005]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 15:19:05 client-18 xinetd[3390]: EXIT: shell status=0 pid=18004 duration=1(sec)
May 17 15:19:05 client-18 xinetd[3390]: START: shell pid=18040 from=192.168.4.5
May 17 15:19:05 client-18 rshd[18041]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 15:19:05 client-18 xinetd[3390]: EXIT: shell status=0 pid=18040 duration=0(sec)
May 17 15:19:05 client-18 xinetd[3390]: START: shell pid=18077 from=192.168.4.5
May 17 15:19:05 client-18 rshd[18078]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 59: test log_commit_thread vs filter_destroy race ============================== 15:19:05 \(1305670745\)");echo XXRETCODE:$?'
May 17 15:19:05 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 59: test log_commit_thread vs filter_destroy race ============================== 15:19:05 (1305670745)
May 17 15:19:05 client-18 xinetd[3390]: EXIT: shell status=0 pid=18077 duration=0(sec)
May 17 15:19:19 client-18 xinetd[3390]: START: shell pid=18112 from=192.168.4.5
May 17 15:19:19 client-18 rshd[18113]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 15:19:20 client-18 kernel: Lustre: 3861:0:(import.c:529:import_select_connection()) lustre-OST0000-osc-ffff810317273c00: tried all connections, increasing latency to 11s
May 17 15:19:20 client-18 kernel: Lustre: 3861:0:(import.c:529:import_select_connection()) Skipped 1 previous similar message
May 17 15:19:24 client-18 xinetd[3390]: EXIT: shell status=0 pid=18112 duration=5(sec)
May 17 15:19:32 client-18 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail.
May 17 15:19:32 client-18 kernel: LustreError: Skipped 16 previous similar messages
May 17 15:19:42 client-18 kernel: Lustre: 3860:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74d00652 to 0x62e5a8d74d0476d
May 17 15:19:42 client-18 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import
May 17 15:19:42 client-18 kernel: Lustre: Skipped 16 previous similar messages
May 17 15:19:43 client-18 xinetd[3390]: START: shell pid=18148 from=192.168.4.5
May 17 15:19:43 client-18 rshd[18149]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 15:19:43 client-18 xinetd[3390]: EXIT: shell status=0 pid=18148 duration=0(sec)
May 17 15:20:03 client-18 xinetd[3390]: START: shell pid=18188 from=192.168.4.5
May 17 15:20:04 client-18 rshd[18189]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 15:20:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=18188 duration=1(sec)
May 17 15:20:04 client-18 xinetd[3390]: START: shell pid=18224 from=192.168.4.5
May 17 15:20:04 client-18 rshd[18225]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 15:20:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=18224 duration=0(sec)
May 17 15:20:04 client-18 xinetd[3390]: START: shell pid=18261 from=192.168.4.5
May 17 15:20:04 client-18 rshd[18262]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 60: test llog post recovery init vs llog unlink ================================ 15:20:04 \(1305670804\)");echo XXRETCODE:$?'
May 17 15:20:04 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 60: test llog post recovery init vs llog unlink ================================ 15:20:04 (1305670804)
May 17 15:20:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=18261 duration=0(sec)
May 17 15:20:04 client-18 xinetd[3390]: START: shell pid=18296 from=192.168.4.5
May 17 15:20:04 client-18 rshd[18297]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 15:20:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=18296 duration=0(sec)
May 17 15:20:24 client-18 xinetd[3390]: START: shell pid=18335 from=192.168.4.5
May 17 15:20:24 client-18 rshd[18336]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 15:20:24 client-18 xinetd[3390]: EXIT: shell status=0 pid=18335 duration=0(sec)
May 17 15:20:24 client-18 xinetd[3390]: START: shell pid=18369 from=192.168.4.5
May 17 15:20:24 client-18 rshd[18370]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 15:20:24 client-18 xinetd[3390]: EXIT: shell status=0 pid=18369 duration=0(sec)
May 17 15:20:25 client-18 xinetd[3390]: START: shell pid=18405 from=192.168.4.5
May 17 15:20:25 client-18 rshd[18406]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 15:20:25 client-18 xinetd[3390]: EXIT: shell status=0 pid=18405 duration=0(sec)
May 17 15:20:25 client-18 xinetd[3390]: START: shell pid=18442 from=192.168.4.5
May 17 15:20:25 client-18 rshd[18443]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 61a: test race llog recovery vs llog cleanup =================================== 15:20:25 \(1305670825\)");echo XXRETCODE:$?'
May 17 15:20:25 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 61a: test race llog recovery vs llog cleanup =================================== 15:20:25 (1305670825)
May 17 15:20:25 client-18 xinetd[3390]: EXIT: shell status=0 pid=18442 duration=0(sec)
May 17 15:20:26 client-18 xinetd[3390]: START: shell pid=18477 from=192.168.4.5
May 17 15:20:26 client-18 rshd[18478]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 15:20:26 client-18 xinetd[3390]: EXIT: shell status=0 pid=18477 duration=0(sec)
May 17 15:20:34 client-18 kernel: Lustre: 3860:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74d0476d to 0x62e5a8d74d18124
May 17 15:21:26 client-18 xinetd[3390]: START: shell pid=18523 from=192.168.4.5
May 17 15:21:26 client-18 rshd[18524]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 15:21:29 client-18 kernel: Lustre: 3861:0:(import.c:529:import_select_connection()) lustre-OST0000-osc-ffff810317273c00: tried all connections, increasing latency to 21s
May 17 15:21:29 client-18 kernel: Lustre: 3861:0:(import.c:529:import_select_connection()) Skipped 9 previous similar messages
May 17 15:21:33 client-18 xinetd[3390]: EXIT: shell status=0 pid=18523 duration=7(sec)
May 17 15:22:03 client-18 xinetd[3390]: START: shell pid=18557 from=192.168.4.5
May 17 15:22:03 client-18 rshd[18558]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 15:22:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=18557 duration=0(sec)
May 17 15:22:03 client-18 xinetd[3390]: START: shell pid=18593 from=192.168.4.5
May 17 15:22:03 client-18 rshd[18594]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 15:22:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=18593 duration=0(sec)
May 17 15:22:03 client-18 xinetd[3390]: START: shell pid=18630 from=192.168.4.5
May 17 15:22:03 client-18 rshd[18631]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 61b: test race mds llog sync vs llog cleanup =================================== 15:22:03 \(1305670923\)");echo XXRETCODE:$?'
May 17 15:22:03 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 61b: test race mds llog sync vs llog cleanup =================================== 15:22:03 (1305670923)
May 17 15:22:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=18630 duration=0(sec)
May 17 15:22:24 client-18 kernel: Lustre: 3860:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74d18124 to 0x62e5a8d74d182f2
May 17 15:23:42 client-18 kernel: Lustre: 3860:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74d182f2 to 0x62e5a8d74d18450
May 17 15:23:43 client-18 xinetd[3390]: START: shell pid=18679 from=192.168.4.5
May 17 15:23:43 client-18 rshd[18680]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 15:24:09 client-18 xinetd[3390]: EXIT: shell status=0 pid=18679 duration=26(sec)
May 17 15:24:09 client-18 xinetd[3390]: START: shell pid=18715 from=192.168.4.5
May 17 15:24:09 client-18 rshd[18716]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 15:24:09 client-18 xinetd[3390]: EXIT: shell status=0 pid=18715 duration=0(sec)
May 17 15:24:09 client-18 xinetd[3390]: START: shell pid=18751 from=192.168.4.5
May 17 15:24:09 client-18 rshd[18752]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 15:24:09 client-18 xinetd[3390]: EXIT: shell status=0 pid=18751 duration=0(sec)
May 17 15:24:09 client-18 xinetd[3390]: START: shell pid=18788 from=192.168.4.5
May 17 15:24:09 client-18 rshd[18789]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 61c: test race mds llog sync vs llog cleanup =================================== 15:24:09 \(1305671049\)");echo XXRETCODE:$?'
May 17 15:24:10 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 61c: test race mds llog sync vs llog cleanup =================================== 15:24:09 (1305671049)
May 17 15:24:10 client-18 xinetd[3390]: EXIT: shell status=0 pid=18788 duration=1(sec)
May 17 15:24:45 client-18 xinetd[3390]: START: shell pid=18823 from=192.168.4.5
May 17 15:24:45 client-18 rshd[18824]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 15:24:45 client-18 xinetd[3390]: EXIT: shell status=0 pid=18823 duration=0(sec)
May 17 15:24:45 client-18 xinetd[3390]: START: shell pid=18857 from=192.168.4.5
May 17 15:24:45 client-18 rshd[18858]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 15:24:46 client-18 xinetd[3390]: EXIT: shell status=0 pid=18857 duration=1(sec)
May 17 15:24:46 client-18 xinetd[3390]: START: shell pid=18893 from=192.168.4.5
May 17 15:24:46 client-18 rshd[18894]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 15:24:46 client-18 xinetd[3390]: EXIT: shell status=0 pid=18893 duration=0(sec)
May 17 15:24:46 client-18 xinetd[3390]: START: shell pid=18930 from=192.168.4.5
May 17 15:24:46 client-18 rshd[18931]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark  SKIP: replay-single test_61d skipping ALWAYS excluded test 61d");echo XXRETCODE:$?'
May 17 15:24:46 client-18 kernel: Lustre: DEBUG MARKER: SKIP: replay-single test_61d skipping ALWAYS excluded test 61d
May 17 15:24:46 client-18 xinetd[3390]: EXIT: shell status=0 pid=18930 duration=0(sec)
May 17 15:24:46 client-18 xinetd[3390]: START: shell pid=18965 from=192.168.4.5
May 17 15:24:46 client-18 rshd[18966]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 62: don\'t mis-drop resent replay =============================================== 15:24:46 \(1305671086\)");echo XXRETCODE:$?'
May 17 15:24:46 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 62: don't mis-drop resent replay =============================================== 15:24:46 (1305671086)
May 17 15:24:46 client-18 xinetd[3390]: EXIT: shell status=0 pid=18965 duration=0(sec)
May 17 15:24:46 client-18 xinetd[3390]: START: shell pid=19000 from=192.168.4.5
May 17 15:24:46 client-18 rshd[19001]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 15:24:46 client-18 xinetd[3390]: EXIT: shell status=0 pid=19000 duration=0(sec)
May 17 15:25:06 client-18 xinetd[3390]: START: shell pid=19043 from=192.168.4.5
May 17 15:25:06 client-18 rshd[19044]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 15:25:06 client-18 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_statfs operation failed with -107
May 17 15:25:06 client-18 kernel: LustreError: Skipped 26 previous similar messages
May 17 15:25:06 client-18 kernel: Lustre: lustre-MDT0000-mdc-ffff810317273c00: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete.
May 17 15:25:06 client-18 kernel: Lustre: Skipped 11 previous similar messages
May 17 15:25:07 client-18 kernel: Lustre: 3860:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74d18450 to 0x62e5a8d74d18a4d
May 17 15:25:07 client-18 kernel: Lustre: MGC192.168.4.128@o2ib: Connection restored to service MGS using nid 192.168.4.128@o2ib.
May 17 15:25:07 client-18 kernel: Lustre: Skipped 19 previous similar messages
May 17 15:25:39 client-18 kernel: Lustre: 3859:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1368947485957271 sent from lustre-MDT0000-mdc-ffff810317273c00 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305671092] [real_sent 1305671092] [current 1305671139] [deadline 47s] [delay 0s]  req@ffff8103067c9c00 x1368947485957271/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 192/192 e 0 to 1 dl 1305671139 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
May 17 15:25:39 client-18 kernel: Lustre: 3859:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 32 previous similar messages
May 17 15:25:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=19043 duration=51(sec)
May 17 15:25:58 client-18 xinetd[3390]: START: shell pid=19081 from=192.168.4.5
May 17 15:25:58 client-18 rshd[19082]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 15:25:58 client-18 xinetd[3390]: EXIT: shell status=0 pid=19081 duration=0(sec)
May 17 15:25:58 client-18 xinetd[3390]: START: shell pid=19117 from=192.168.4.5
May 17 15:25:58 client-18 rshd[19118]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 15:25:58 client-18 xinetd[3390]: EXIT: shell status=0 pid=19117 duration=0(sec)
May 17 15:25:58 client-18 xinetd[3390]: START: shell pid=19154 from=192.168.4.5
May 17 15:25:58 client-18 rshd[19155]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 65a: AT: verify early replies ================================================== 15:25:58 \(1305671158\)");echo XXRETCODE:$?'
May 17 15:25:58 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 65a: AT: verify early replies ================================================== 15:25:58 (1305671158)
May 17 15:25:58 client-18 xinetd[3390]: EXIT: shell status=0 pid=19154 duration=0(sec)
May 17 15:26:25 client-18 xinetd[3390]: START: shell pid=19193 from=192.168.4.5
May 17 15:26:25 client-18 rshd[19194]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl set_param debug=\"inode super warning dlmtrace error emerg ha rpctrace vfstrace config console\";");echo XXRETCODE:$?'
May 17 15:26:25 client-18 xinetd[3390]: EXIT: shell status=0 pid=19193 duration=0(sec)
May 17 15:26:35 client-18 xinetd[3390]: START: shell pid=19228 from=192.168.4.5
May 17 15:26:35 client-18 rshd[19229]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 15:26:35 client-18 xinetd[3390]: EXIT: shell status=0 pid=19228 duration=0(sec)
May 17 15:26:35 client-18 xinetd[3390]: START: shell pid=19273 from=192.168.4.5
May 17 15:26:35 client-18 rshd[19274]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 15:26:35 client-18 xinetd[3390]: EXIT: shell status=0 pid=19273 duration=0(sec)
May 17 15:26:35 client-18 xinetd[3390]: START: shell pid=19310 from=192.168.4.5
May 17 15:26:35 client-18 rshd[19311]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 65b: AT: verify early replies on packed reply \/ bulk =========================== 15:26:35 \(1305671195\)");echo XXRETCODE:$?'
May 17 15:26:35 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 65b: AT: verify early replies on packed reply / bulk =========================== 15:26:35 (1305671195)
May 17 15:26:35 client-18 xinetd[3390]: EXIT: shell status=0 pid=19310 duration=0(sec)
May 17 15:27:02 client-18 xinetd[3390]: START: shell pid=19345 from=192.168.4.5
May 17 15:27:02 client-18 rshd[19346]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl set_param debug=\"inode super warning dlmtrace error emerg ha rpctrace vfstrace config console\";");echo XXRETCODE:$?'
May 17 15:27:02 client-18 xinetd[3390]: EXIT: shell status=0 pid=19345 duration=0(sec)
May 17 15:27:02 client-18 xinetd[3390]: START: shell pid=19380 from=192.168.4.5
May 17 15:27:02 client-18 rshd[19381]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 15:27:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=19380 duration=1(sec)
May 17 15:27:03 client-18 xinetd[3390]: START: shell pid=19416 from=192.168.4.5
May 17 15:27:03 client-18 rshd[19417]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 15:27:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=19416 duration=0(sec)
May 17 15:27:03 client-18 xinetd[3390]: START: shell pid=19453 from=192.168.4.5
May 17 15:27:03 client-18 rshd[19454]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 66a: AT: verify MDT service time adjusts with no early replies ================= 15:27:03 \(1305671223\)");echo XXRETCODE:$?'
May 17 15:27:03 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 66a: AT: verify MDT service time adjusts with no early replies ================= 15:27:03 (1305671223)
May 17 15:27:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=19453 duration=0(sec)
May 17 15:27:48 client-18 xinetd[3390]: START: shell pid=19492 from=192.168.4.5
May 17 15:27:48 client-18 rshd[19493]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 15:27:48 client-18 xinetd[3390]: EXIT: shell status=0 pid=19492 duration=0(sec)
May 17 15:27:48 client-18 xinetd[3390]: START: shell pid=19528 from=192.168.4.5
May 17 15:27:49 client-18 rshd[19529]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 15:27:49 client-18 xinetd[3390]: EXIT: shell status=0 pid=19528 duration=1(sec)
May 17 15:27:49 client-18 xinetd[3390]: START: shell pid=19565 from=192.168.4.5
May 17 15:27:49 client-18 rshd[19566]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 66b: AT: verify net latency adjusts ============================================ 15:27:49 \(1305671269\)");echo XXRETCODE:$?'
May 17 15:27:49 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 66b: AT: verify net latency adjusts ============================================ 15:27:49 (1305671269)
May 17 15:27:49 client-18 xinetd[3390]: EXIT: shell status=0 pid=19565 duration=0(sec)
May 17 15:29:02 client-18 xinetd[3390]: START: shell pid=19604 from=192.168.4.5
May 17 15:29:02 client-18 rshd[19605]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 15:29:02 client-18 xinetd[3390]: EXIT: shell status=0 pid=19604 duration=0(sec)
May 17 15:29:02 client-18 xinetd[3390]: START: shell pid=19640 from=192.168.4.5
May 17 15:29:02 client-18 rshd[19641]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 15:29:02 client-18 xinetd[3390]: EXIT: shell status=0 pid=19640 duration=0(sec)
May 17 15:29:02 client-18 xinetd[3390]: START: shell pid=19677 from=192.168.4.5
May 17 15:29:02 client-18 rshd[19678]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 67a: AT: verify slow request processing doesn\'t induce reconnects ============== 15:29:02 \(1305671342\)");echo XXRETCODE:$?'
May 17 15:29:02 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 67a: AT: verify slow request processing doesn't induce reconnects ============== 15:29:02 (1305671342)
May 17 15:29:02 client-18 xinetd[3390]: EXIT: shell status=0 pid=19677 duration=0(sec)
May 17 15:29:24 client-18 xinetd[3390]: START: shell pid=19712 from=192.168.4.5
May 17 15:29:24 client-18 rshd[19713]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 15:29:24 client-18 xinetd[3390]: EXIT: shell status=0 pid=19712 duration=0(sec)
May 17 15:29:24 client-18 xinetd[3390]: START: shell pid=19748 from=192.168.4.5
May 17 15:29:24 client-18 rshd[19749]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 15:29:24 client-18 xinetd[3390]: EXIT: shell status=0 pid=19748 duration=0(sec)
May 17 15:29:24 client-18 xinetd[3390]: START: shell pid=19785 from=192.168.4.5
May 17 15:29:24 client-18 rshd[19786]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 67b: AT: verify instant slowdown doesn\'t induce reconnects ===================== 15:29:24 \(1305671364\)");echo XXRETCODE:$?'
May 17 15:29:24 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 67b: AT: verify instant slowdown doesn't induce reconnects ===================== 15:29:24 (1305671364)
May 17 15:29:24 client-18 xinetd[3390]: EXIT: shell status=0 pid=19785 duration=0(sec)
May 17 15:29:46 client-18 xinetd[3390]: START: shell pid=19820 from=192.168.4.5
May 17 15:29:46 client-18 rshd[19821]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "df /mnt/lustre; uname -n >> /mnt/lustre/recon");echo XXRETCODE:$?'
May 17 15:29:46 client-18 xinetd[3390]: EXIT: shell status=0 pid=19820 duration=0(sec)
May 17 15:29:46 client-18 xinetd[3390]: START: shell pid=19857 from=192.168.4.5
May 17 15:29:46 client-18 rshd[19858]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark phase 2");echo XXRETCODE:$?'
May 17 15:29:46 client-18 kernel: Lustre: DEBUG MARKER: phase 2
May 17 15:29:46 client-18 xinetd[3390]: EXIT: shell status=0 pid=19857 duration=0(sec)
May 17 15:29:47 client-18 xinetd[3390]: START: shell pid=19892 from=192.168.4.5
May 17 15:29:47 client-18 rshd[19893]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "df /mnt/lustre; uname -n >> /mnt/lustre/recon");echo XXRETCODE:$?'
May 17 15:29:47 client-18 xinetd[3390]: EXIT: shell status=0 pid=19892 duration=0(sec)
May 17 15:29:47 client-18 xinetd[3390]: START: shell pid=19929 from=192.168.4.5
May 17 15:29:47 client-18 rshd[19930]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 15:29:47 client-18 xinetd[3390]: EXIT: shell status=0 pid=19929 duration=0(sec)
May 17 15:29:47 client-18 xinetd[3390]: START: shell pid=19965 from=192.168.4.5
May 17 15:29:47 client-18 rshd[19966]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 15:29:47 client-18 xinetd[3390]: EXIT: shell status=0 pid=19965 duration=0(sec)
May 17 15:29:47 client-18 xinetd[3390]: START: shell pid=20002 from=192.168.4.5
May 17 15:29:47 client-18 rshd[20003]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 68: AT: verify slowing locks =================================================== 15:29:47 \(1305671387\)");echo XXRETCODE:$?'
May 17 15:29:47 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 68: AT: verify slowing locks =================================================== 15:29:47 (1305671387)
May 17 15:29:47 client-18 xinetd[3390]: EXIT: shell status=0 pid=20002 duration=0(sec)
May 17 15:30:53 client-18 xinetd[3390]: START: shell pid=20041 from=192.168.4.5
May 17 15:30:53 client-18 rshd[20042]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 15:30:53 client-18 xinetd[3390]: EXIT: shell status=0 pid=20041 duration=0(sec)
May 17 15:30:53 client-18 xinetd[3390]: START: shell pid=20077 from=192.168.4.5
May 17 15:30:53 client-18 rshd[20078]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 15:30:53 client-18 xinetd[3390]: EXIT: shell status=0 pid=20077 duration=0(sec)
May 17 15:30:53 client-18 xinetd[3390]: START: shell pid=20114 from=192.168.4.5
May 17 15:30:53 client-18 rshd[20115]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 70a: check multi client t-f ==================================================== 15:30:53 \(1305671453\)");echo XXRETCODE:$?'
May 17 15:30:53 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 70a: check multi client t-f ==================================================== 15:30:53 (1305671453)
May 17 15:30:53 client-18 xinetd[3390]: EXIT: shell status=0 pid=20114 duration=0(sec)
May 17 15:30:53 client-18 xinetd[3390]: START: shell pid=20149 from=192.168.4.5
May 17 15:30:53 client-18 rshd[20150]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c " running=\$(mount | grep -c /mnt/lustre' '); rc=0; if [ \$running -eq 0 ] ; then     mkdir -p /mnt/lustre;     mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre;     rc=\$?; fi; exit \$rc");echo XXRETCODE:$?'
May 17 15:30:53 client-18 xinetd[3390]: EXIT: shell status=0 pid=20149 duration=0(sec)
May 17 15:30:53 client-18 xinetd[3390]: START: shell pid=20187 from=192.168.4.5
May 17 15:30:53 client-18 rshd[20188]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "mount | grep -w /mnt/lustre");echo XXRETCODE:$?'
May 17 15:30:53 client-18 xinetd[3390]: EXIT: shell status=0 pid=20187 duration=0(sec)
May 17 15:30:53 client-18 xinetd[3390]: START: shell pid=20224 from=192.168.4.5
May 17 15:30:53 client-18 rshd[20225]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param debug=\"0x33f0406\";         lctl set_param subsystem_debug=\"0xffb7e3ff\";         lctl set_param debug_mb=32;");echo XXRETCODE:$?'
May 17 15:30:53 client-18 kernel: Lustre: 20259:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release.
May 17 15:30:53 client-18 kernel: Lustre: 20260:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release.
May 17 15:30:53 client-18 xinetd[3390]: EXIT: shell status=0 pid=20224 duration=0(sec)
May 17 15:30:54 client-18 xinetd[3390]: START: shell pid=20261 from=192.168.4.5
May 17 15:30:54 client-18 rshd[20262]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "dd bs=1M count=10 if=/dev/zero of=/mnt/lustre/f70a_client-18-ib");echo XXRETCODE:$?'
May 17 15:30:54 client-18 xinetd[3390]: EXIT: shell status=0 pid=20261 duration=0(sec)
May 17 15:30:54 client-18 xinetd[3390]: START: shell pid=20296 from=192.168.4.5
May 17 15:30:54 client-18 rshd[20297]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "dd if=/mnt/lustre/f70a_client-5-ib of=/dev/null");echo XXRETCODE:$?'
May 17 15:30:56 client-18 xinetd[3390]: EXIT: shell status=0 pid=20296 duration=2(sec)
May 17 15:30:58 client-18 xinetd[3390]: START: shell pid=20331 from=192.168.4.5
May 17 15:30:58 client-18 rshd[20332]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 15:30:58 client-18 xinetd[3390]: EXIT: shell status=0 pid=20331 duration=0(sec)
May 17 15:30:58 client-18 xinetd[3390]: START: shell pid=20367 from=192.168.4.5
May 17 15:30:58 client-18 rshd[20368]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 15:30:58 client-18 xinetd[3390]: EXIT: shell status=0 pid=20367 duration=0(sec)
May 17 15:30:58 client-18 xinetd[3390]: START: shell pid=20404 from=192.168.4.5
May 17 15:30:58 client-18 rshd[20405]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 70b: mds recovery\; 2 clients =================================================== 15:30:58 \(1305671458\)");echo XXRETCODE:$?'
May 17 15:30:58 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 70b: mds recovery; 2 clients =================================================== 15:30:58 (1305671458)
May 17 15:30:58 client-18 xinetd[3390]: EXIT: shell status=0 pid=20404 duration=0(sec)
May 17 15:30:58 client-18 xinetd[3390]: START: shell pid=20439 from=192.168.4.5
May 17 15:30:58 client-18 rshd[20440]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c " running=\$(mount | grep -c /mnt/lustre' '); rc=0; if [ \$running -eq 0 ] ; then     mkdir -p /mnt/lustre;     mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre;     rc=\$?; fi; exit \$rc");echo XXRETCODE:$?'
May 17 15:30:58 client-18 xinetd[3390]: EXIT: shell status=0 pid=20439 duration=0(sec)
May 17 15:30:58 client-18 xinetd[3390]: START: shell pid=20477 from=192.168.4.5
May 17 15:30:58 client-18 rshd[20478]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "mount | grep -w /mnt/lustre");echo XXRETCODE:$?'
May 17 15:30:58 client-18 xinetd[3390]: EXIT: shell status=0 pid=20477 duration=0(sec)
May 17 15:30:58 client-18 xinetd[3390]: START: shell pid=20514 from=192.168.4.5
May 17 15:30:59 client-18 rshd[20515]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param debug=\"0x33f0406\";         lctl set_param subsystem_debug=\"0xffb7e3ff\";         lctl set_param debug_mb=32;");echo XXRETCODE:$?'
May 17 15:30:59 client-18 kernel: Lustre: 20549:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release.
May 17 15:30:59 client-18 xinetd[3390]: EXIT: shell status=0 pid=20514 duration=1(sec)
May 17 15:30:59 client-18 xinetd[3390]: START: shell pid=20551 from=192.168.4.5
May 17 15:30:59 client-18 rshd[20552]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "set -x; MISSING_DBENCH_OK= 		PATH=:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/tests:/usr/lib64/openmpi/1.4-gcc/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/usr/lib64/lustre/utils:/usr/lib64/lustre/utils/gss:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey::/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests/:/usr/share/doc/dbench/loadfiles 		DBENCH_LIB=/usr/share/doc/dbench/loadfiles TESTSUITE=replay-single TESTNAME=test_70b 		LCTL=/usr/sbin/lctl rundbench 1 -t 300");echo XXRETCODE:$?'
May 17 15:30:59 client-18 xinetd[3390]: START: shell pid=20572 from=192.168.4.5
May 17 15:30:59 client-18 rshd[20577]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark Started rundbench load pid=13056 ...");echo XXRETCODE:$?'
May 17 15:30:59 client-18 kernel: Lustre: DEBUG MARKER: Started rundbench load pid=13056 ...
May 17 15:30:59 client-18 xinetd[3390]: EXIT: shell status=0 pid=20572 duration=0(sec)
May 17 15:31:01 client-18 xinetd[3390]: START: shell pid=20631 from=192.168.4.5
May 17 15:31:01 client-18 rshd[20632]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "killall -0 rundbench");echo XXRETCODE:$?'
May 17 15:31:01 client-18 xinetd[3390]: EXIT: shell status=0 pid=20631 duration=0(sec)
May 17 15:31:02 client-18 xinetd[3390]: START: shell pid=20666 from=192.168.4.5
May 17 15:31:02 client-18 rshd[20667]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 15:31:02 client-18 xinetd[3390]: EXIT: shell status=0 pid=20666 duration=0(sec)
May 17 15:31:04 client-18 xinetd[3390]: START: shell pid=20706 from=192.168.4.5
May 17 15:31:04 client-18 rshd[20707]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark test_70b fail mds1 1 times");echo XXRETCODE:$?'
May 17 15:31:04 client-18 kernel: Lustre: DEBUG MARKER: test_70b fail mds1 1 times
May 17 15:31:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=20706 duration=0(sec)
May 17 15:31:12 client-18 kernel: Lustre: 3861:0:(import.c:529:import_select_connection()) lustre-MDT0000-mdc-ffff810317273c00: tried all connections, increasing latency to 21s
May 17 15:31:12 client-18 kernel: Lustre: 3861:0:(import.c:529:import_select_connection()) Skipped 6 previous similar messages
May 17 15:31:14 client-18 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail.
May 17 15:31:14 client-18 kernel: LustreError: Skipped 4 previous similar messages
May 17 15:31:23 client-18 xinetd[3390]: START: shell pid=20745 from=192.168.4.5
May 17 15:31:23 client-18 rshd[20746]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 15:31:24 client-18 kernel: Lustre: 3860:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74d18a4d to 0x62e5a8d74d4c629
May 17 15:31:24 client-18 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import
May 17 15:31:24 client-18 kernel: Lustre: Skipped 4 previous similar messages
May 17 15:31:39 client-18 kernel: LustreError: 3860:0:(client.c:2570:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff810317273800 x1368947485957949/t313532613065(313532613065) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 568/544 e 0 to 0 dl 1305671545 ref 2 fl Interpret:RP/ffffffff/ffffffff rc 301/-1
May 17 15:31:39 client-18 kernel: LustreError: 3860:0:(client.c:2570:ptlrpc_replay_interpret()) Skipped 1 previous similar message
May 17 15:31:40 client-18 xinetd[3390]: EXIT: shell status=0 pid=20745 duration=17(sec)
May 17 15:31:40 client-18 xinetd[3390]: START: shell pid=20783 from=192.168.4.5
May 17 15:31:40 client-18 rshd[20784]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "killall -0 rundbench");echo XXRETCODE:$?'
May 17 15:31:40 client-18 xinetd[3390]: EXIT: shell status=0 pid=20783 duration=0(sec)
May 17 15:31:41 client-18 xinetd[3390]: START: shell pid=20818 from=192.168.4.5
May 17 15:31:41 client-18 rshd[20819]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 15:31:41 client-18 xinetd[3390]: EXIT: shell status=0 pid=20818 duration=0(sec)
May 17 15:31:43 client-18 xinetd[3390]: START: shell pid=20857 from=192.168.4.5
May 17 15:31:43 client-18 rshd[20858]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark test_70b fail mds1 2 times");echo XXRETCODE:$?'
May 17 15:31:43 client-18 kernel: Lustre: DEBUG MARKER: test_70b fail mds1 2 times
May 17 15:31:43 client-18 xinetd[3390]: EXIT: shell status=0 pid=20857 duration=0(sec)
May 17 15:32:02 client-18 xinetd[3390]: START: shell pid=20892 from=192.168.4.5
May 17 15:32:02 client-18 rshd[20893]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 15:32:11 client-18 kernel: Lustre: 3860:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74d4c629 to 0x62e5a8d74d72ad3
May 17 15:32:24 client-18 xinetd[3390]: EXIT: shell status=0 pid=20892 duration=22(sec)
May 17 15:32:24 client-18 xinetd[3390]: START: shell pid=20929 from=192.168.4.5
May 17 15:32:24 client-18 rshd[20930]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "killall -0 rundbench");echo XXRETCODE:$?'
May 17 15:32:24 client-18 xinetd[3390]: EXIT: shell status=0 pid=20929 duration=0(sec)
May 17 15:32:25 client-18 xinetd[3390]: START: shell pid=20968 from=192.168.4.5
May 17 15:32:25 client-18 rshd[20969]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 15:32:25 client-18 xinetd[3390]: EXIT: shell status=0 pid=20968 duration=0(sec)
May 17 15:32:27 client-18 xinetd[3390]: START: shell pid=21007 from=192.168.4.5
May 17 15:32:27 client-18 rshd[21008]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark test_70b fail mds1 3 times");echo XXRETCODE:$?'
May 17 15:32:27 client-18 kernel: Lustre: DEBUG MARKER: test_70b fail mds1 3 times
May 17 15:32:27 client-18 xinetd[3390]: EXIT: shell status=0 pid=21007 duration=0(sec)
May 17 15:32:46 client-18 kernel: Lustre: 3860:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74d72ad3 to 0x62e5a8d74d9901e
May 17 15:32:46 client-18 xinetd[3390]: START: shell pid=21043 from=192.168.4.5
May 17 15:32:46 client-18 rshd[21044]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 15:33:07 client-18 xinetd[3390]: EXIT: shell status=0 pid=21043 duration=21(sec)
May 17 15:33:07 client-18 xinetd[3390]: START: shell pid=21079 from=192.168.4.5
May 17 15:33:07 client-18 rshd[21080]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "killall -0 rundbench");echo XXRETCODE:$?'
May 17 15:33:07 client-18 xinetd[3390]: EXIT: shell status=0 pid=21079 duration=0(sec)
May 17 15:33:08 client-18 xinetd[3390]: START: shell pid=21114 from=192.168.4.5
May 17 15:33:08 client-18 rshd[21115]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 15:33:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=21114 duration=0(sec)
May 17 15:33:09 client-18 xinetd[3390]: START: shell pid=21153 from=192.168.4.5
May 17 15:33:10 client-18 rshd[21154]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark test_70b fail mds1 4 times");echo XXRETCODE:$?'
May 17 15:33:10 client-18 kernel: Lustre: DEBUG MARKER: test_70b fail mds1 4 times
May 17 15:33:10 client-18 xinetd[3390]: EXIT: shell status=0 pid=21153 duration=1(sec)
May 17 15:33:28 client-18 kernel: Lustre: 3860:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74d9901e to 0x62e5a8d74dbe9c3
May 17 15:33:29 client-18 xinetd[3390]: START: shell pid=21189 from=192.168.4.5
May 17 15:33:29 client-18 rshd[21190]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 15:33:50 client-18 kernel: LustreError: 3860:0:(client.c:2570:ptlrpc_replay_interpret()) @@@ status -116, old was 0  req@ffff8102e46e2000 x1368947485977033/t326417516324(326417516324) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:23/10 lens 360/424 e 0 to 0 dl 1305671657 ref 2 fl Interpret:R/ffffffff/ffffffff rc -116/-1
May 17 15:33:51 client-18 xinetd[3390]: EXIT: shell status=0 pid=21189 duration=22(sec)
May 17 15:33:51 client-18 xinetd[3390]: START: shell pid=21230 from=192.168.4.5
May 17 15:33:51 client-18 rshd[21231]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "killall -0 rundbench");echo XXRETCODE:$?'
May 17 15:33:51 client-18 xinetd[3390]: EXIT: shell status=0 pid=21230 duration=0(sec)
May 17 15:33:52 client-18 xinetd[3390]: START: shell pid=21265 from=192.168.4.5
May 17 15:33:52 client-18 rshd[21266]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 15:33:52 client-18 xinetd[3390]: EXIT: shell status=0 pid=21265 duration=0(sec)
May 17 15:33:54 client-18 xinetd[3390]: START: shell pid=21304 from=192.168.4.5
May 17 15:33:54 client-18 rshd[21305]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark test_70b fail mds1 5 times");echo XXRETCODE:$?'
May 17 15:33:54 client-18 kernel: Lustre: DEBUG MARKER: test_70b fail mds1 5 times
May 17 15:33:54 client-18 xinetd[3390]: EXIT: shell status=0 pid=21304 duration=0(sec)
May 17 15:34:13 client-18 xinetd[3390]: START: shell pid=21339 from=192.168.4.5
May 17 15:34:13 client-18 kernel: Lustre: 3860:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74dbe9c3 to 0x62e5a8d74de5814
May 17 15:34:13 client-18 rshd[21341]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 15:34:33 client-18 kernel: LustreError: 3860:0:(client.c:2570:ptlrpc_replay_interpret()) @@@ status -116, old was 0  req@ffff810302db5000 x1368947485981791/t330712483721(330712483721) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:23/10 lens 360/424 e 0 to 0 dl 1305671700 ref 2 fl Interpret:R/ffffffff/ffffffff rc -116/-1
May 17 15:34:33 client-18 kernel: LustreError: 3860:0:(client.c:2570:ptlrpc_replay_interpret()) Skipped 2 previous similar messages
May 17 15:34:34 client-18 xinetd[3390]: EXIT: shell status=0 pid=21339 duration=21(sec)
May 17 15:34:34 client-18 xinetd[3390]: START: shell pid=21376 from=192.168.4.5
May 17 15:34:34 client-18 rshd[21377]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "killall -0 rundbench");echo XXRETCODE:$?'
May 17 15:34:34 client-18 xinetd[3390]: EXIT: shell status=0 pid=21376 duration=0(sec)
May 17 15:34:35 client-18 xinetd[3390]: START: shell pid=21411 from=192.168.4.5
May 17 15:34:35 client-18 rshd[21412]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 15:34:35 client-18 xinetd[3390]: EXIT: shell status=0 pid=21411 duration=0(sec)
May 17 15:34:37 client-18 xinetd[3390]: START: shell pid=21450 from=192.168.4.5
May 17 15:34:37 client-18 rshd[21451]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark test_70b fail mds1 6 times");echo XXRETCODE:$?'
May 17 15:34:37 client-18 kernel: Lustre: DEBUG MARKER: test_70b fail mds1 6 times
May 17 15:34:37 client-18 xinetd[3390]: EXIT: shell status=0 pid=21450 duration=0(sec)
May 17 15:34:55 client-18 kernel: Lustre: 3860:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74de5814 to 0x62e5a8d74e0be4d
May 17 15:34:56 client-18 xinetd[3390]: START: shell pid=21490 from=192.168.4.5
May 17 15:34:56 client-18 rshd[21491]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 15:35:17 client-18 kernel: LustreError: 3860:0:(client.c:2570:ptlrpc_replay_interpret()) @@@ status -116, old was 0  req@ffff8102f2fe1c00 x1368947485986731/t335007451052(335007451052) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:23/10 lens 360/424 e 0 to 0 dl 1305671744 ref 2 fl Interpret:R/ffffffff/ffffffff rc -116/-1
May 17 15:35:18 client-18 kernel: Lustre: lustre-MDT0000-mdc-ffff810317273c00: Connection restored to service lustre-MDT0000 using nid 192.168.4.128@o2ib.
May 17 15:35:18 client-18 kernel: Lustre: Skipped 12 previous similar messages
May 17 15:35:18 client-18 xinetd[3390]: EXIT: shell status=0 pid=21490 duration=22(sec)
May 17 15:35:18 client-18 xinetd[3390]: START: shell pid=21526 from=192.168.4.5
May 17 15:35:18 client-18 rshd[21527]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "killall -0 rundbench");echo XXRETCODE:$?'
May 17 15:35:18 client-18 xinetd[3390]: EXIT: shell status=0 pid=21526 duration=0(sec)
May 17 15:35:19 client-18 xinetd[3390]: START: shell pid=21561 from=192.168.4.5
May 17 15:35:19 client-18 rshd[21562]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 15:35:19 client-18 xinetd[3390]: EXIT: shell status=0 pid=21561 duration=0(sec)
May 17 15:35:21 client-18 xinetd[3390]: START: shell pid=21600 from=192.168.4.5
May 17 15:35:21 client-18 rshd[21601]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark test_70b fail mds1 7 times");echo XXRETCODE:$?'
May 17 15:35:21 client-18 kernel: Lustre: DEBUG MARKER: test_70b fail mds1 7 times
May 17 15:35:21 client-18 xinetd[3390]: EXIT: shell status=0 pid=21600 duration=0(sec)
May 17 15:35:21 client-18 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_close operation failed with -19
May 17 15:35:21 client-18 kernel: LustreError: Skipped 15 previous similar messages
May 17 15:35:21 client-18 kernel: Lustre: lustre-MDT0000-mdc-ffff810317273c00: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete.
May 17 15:35:21 client-18 kernel: Lustre: Skipped 6 previous similar messages
May 17 15:35:40 client-18 kernel: Lustre: 3860:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74e0be4d to 0x62e5a8d74e331c9
May 17 15:35:40 client-18 xinetd[3390]: START: shell pid=21636 from=192.168.4.5
May 17 15:35:40 client-18 rshd[21637]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 15:35:47 client-18 kernel: Lustre: 3860:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1368947485994781 sent from lustre-MDT0000-mdc-ffff810317273c00 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305671721] [real_sent 1305671721] [current 1305671747] [deadline 26s] [delay 0s]  req@ffff8102fd188400 x1368947485994781/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 368/392 e 0 to 1 dl 1305671747 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
May 17 15:35:47 client-18 kernel: Lustre: 3860:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 14 previous similar messages
May 17 15:36:00 client-18 xinetd[3390]: EXIT: shell status=0 pid=21636 duration=20(sec)
May 17 15:36:00 client-18 xinetd[3390]: START: shell pid=21672 from=192.168.4.5
May 17 15:36:00 client-18 rshd[21673]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "killall -0 rundbench");echo XXRETCODE:$?'
May 17 15:36:00 client-18 xinetd[3390]: EXIT: shell status=0 pid=21672 duration=0(sec)
May 17 15:36:01 client-18 xinetd[3390]: START: shell pid=21707 from=192.168.4.5
May 17 15:36:01 client-18 rshd[21708]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 15:36:01 client-18 xinetd[3390]: EXIT: shell status=0 pid=21707 duration=0(sec)
May 17 15:36:03 client-18 xinetd[3390]: START: shell pid=21746 from=192.168.4.5
May 17 15:36:03 client-18 rshd[21747]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark test_70b fail mds1 8 times");echo XXRETCODE:$?'
May 17 15:36:03 client-18 kernel: Lustre: DEBUG MARKER: test_70b fail mds1 8 times
May 17 15:36:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=21746 duration=0(sec)
May 17 15:36:20 client-18 kernel: Lustre: 3860:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74e331c9 to 0x62e5a8d74e58e00
May 17 15:36:22 client-18 xinetd[3390]: START: shell pid=21786 from=192.168.4.5
May 17 15:36:22 client-18 rshd[21787]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 15:36:41 client-18 kernel: LustreError: 3860:0:(client.c:2570:ptlrpc_replay_interpret()) @@@ status -116, old was 0  req@ffff8102eccf5000 x1368947485996317/t343597385502(343597385502) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:23/10 lens 360/424 e 0 to 0 dl 1305671828 ref 2 fl Interpret:R/ffffffff/ffffffff rc -116/-1
May 17 15:36:42 client-18 xinetd[3390]: EXIT: shell status=0 pid=21786 duration=20(sec)
May 17 15:37:00 client-18 xinetd[3390]: EXIT: shell status=0 pid=20551 duration=361(sec)
May 17 15:37:00 client-18 xinetd[3390]: START: shell pid=21834 from=192.168.4.5
May 17 15:37:00 client-18 rshd[21835]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 15:37:00 client-18 xinetd[3390]: EXIT: shell status=0 pid=21834 duration=0(sec)
May 17 15:37:00 client-18 xinetd[3390]: START: shell pid=21871 from=192.168.4.5
May 17 15:37:00 client-18 rshd[21872]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 15:37:00 client-18 xinetd[3390]: EXIT: shell status=0 pid=21871 duration=0(sec)
May 17 15:37:00 client-18 xinetd[3390]: START: shell pid=21908 from=192.168.4.5
May 17 15:37:00 client-18 rshd[21909]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 73a: open\(O_CREAT\), unlink, replay, reconnect before open replay , close ======= 15:37:00 \(1305671820\)");echo XXRETCODE:$?'
May 17 15:37:00 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 73a: open(O_CREAT), unlink, replay, reconnect before open replay , close ======= 15:37:00 (1305671820)
May 17 15:37:00 client-18 xinetd[3390]: EXIT: shell status=0 pid=21908 duration=0(sec)
May 17 15:37:00 client-18 xinetd[3390]: START: shell pid=21943 from=192.168.4.5
May 17 15:37:00 client-18 rshd[21944]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 15:37:01 client-18 xinetd[3390]: EXIT: shell status=0 pid=21943 duration=1(sec)
May 17 15:37:20 client-18 kernel: Lustre: 3860:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74e58e00 to 0x62e5a8d74f21db5
May 17 15:37:21 client-18 xinetd[3390]: START: shell pid=21983 from=192.168.4.5
May 17 15:37:21 client-18 rshd[21984]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 15:37:47 client-18 kernel: LustreError: 3860:0:(client.c:2515:ptlrpc_replay_interpret()) request replay timed out, restarting recovery
May 17 15:37:47 client-18 kernel: LustreError: 3860:0:(client.c:2570:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff810317273800 x1368947485957949/t313532613065(313532613065) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 568/544 e 0 to 0 dl 1305671913 ref 2 fl Interpret:RP/ffffffff/ffffffff rc 301/-1
May 17 15:37:47 client-18 kernel: LustreError: 3860:0:(client.c:2570:ptlrpc_replay_interpret()) Skipped 3 previous similar messages
May 17 15:37:47 client-18 xinetd[3390]: EXIT: shell status=0 pid=21983 duration=26(sec)
May 17 15:37:47 client-18 xinetd[3390]: START: shell pid=22023 from=192.168.4.5
May 17 15:37:47 client-18 rshd[22024]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 15:37:47 client-18 xinetd[3390]: EXIT: shell status=0 pid=22023 duration=0(sec)
May 17 15:37:47 client-18 xinetd[3390]: START: shell pid=22059 from=192.168.4.5
May 17 15:37:47 client-18 rshd[22060]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 15:37:47 client-18 xinetd[3390]: EXIT: shell status=0 pid=22059 duration=0(sec)
May 17 15:37:47 client-18 xinetd[3390]: START: shell pid=22096 from=192.168.4.5
May 17 15:37:48 client-18 rshd[22097]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 73b: open\(O_CREAT\), unlink, replay, reconnect at open_replay reply, close ====== 15:37:47 \(1305671867\)");echo XXRETCODE:$?'
May 17 15:37:48 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 73b: open(O_CREAT), unlink, replay, reconnect at open_replay reply, close ====== 15:37:47 (1305671867)
May 17 15:37:48 client-18 xinetd[3390]: EXIT: shell status=0 pid=22096 duration=1(sec)
May 17 15:37:48 client-18 xinetd[3390]: START: shell pid=22131 from=192.168.4.5
May 17 15:37:48 client-18 rshd[22132]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 15:37:48 client-18 xinetd[3390]: EXIT: shell status=0 pid=22131 duration=0(sec)
May 17 15:38:07 client-18 kernel: Lustre: 3860:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74f21db5 to 0x62e5a8d74f220e8
May 17 15:38:08 client-18 xinetd[3390]: START: shell pid=22171 from=192.168.4.5
May 17 15:38:08 client-18 rshd[22172]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 15:38:54 client-18 kernel: LustreError: 3860:0:(client.c:2515:ptlrpc_replay_interpret()) request replay timed out, restarting recovery
May 17 15:38:54 client-18 kernel: LustreError: 3860:0:(client.c:2570:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req@ffff810317273800 x1368947485957949/t313532613065(313532613065) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 568/544 e 0 to 0 dl 1305671961 ref 2 fl Interpret:RP/ffffffff/ffffffff rc 301/-1
May 17 15:38:54 client-18 xinetd[3390]: EXIT: shell status=0 pid=22171 duration=46(sec)
May 17 15:38:54 client-18 xinetd[3390]: START: shell pid=22211 from=192.168.4.5
May 17 15:38:54 client-18 rshd[22212]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 15:38:54 client-18 xinetd[3390]: EXIT: shell status=0 pid=22211 duration=0(sec)
May 17 15:38:54 client-18 xinetd[3390]: START: shell pid=22247 from=192.168.4.5
May 17 15:38:54 client-18 rshd[22248]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 15:38:54 client-18 xinetd[3390]: EXIT: shell status=0 pid=22247 duration=0(sec)
May 17 15:38:55 client-18 xinetd[3390]: START: shell pid=22284 from=192.168.4.5
May 17 15:38:55 client-18 rshd[22285]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 73c: open\(O_CREAT\), unlink, replay, reconnect at last_replay, close ============ 15:38:54 \(1305671934\)");echo XXRETCODE:$?'
May 17 15:38:55 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 73c: open(O_CREAT), unlink, replay, reconnect at last_replay, close ============ 15:38:54 (1305671934)
May 17 15:38:55 client-18 xinetd[3390]: EXIT: shell status=0 pid=22284 duration=0(sec)
May 17 15:38:55 client-18 xinetd[3390]: START: shell pid=22319 from=192.168.4.5
May 17 15:38:55 client-18 rshd[22320]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 15:38:55 client-18 xinetd[3390]: EXIT: shell status=0 pid=22319 duration=0(sec)
May 17 15:39:14 client-18 kernel: Lustre: 3860:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74f220e8 to 0x62e5a8d74f22414
May 17 15:39:15 client-18 xinetd[3390]: START: shell pid=22359 from=192.168.4.5
May 17 15:39:15 client-18 rshd[22360]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 15:39:15 client-18 xinetd[3390]: EXIT: shell status=0 pid=22359 duration=0(sec)
May 17 15:39:15 client-18 xinetd[3390]: START: shell pid=22393 from=192.168.4.5
May 17 15:39:16 client-18 rshd[22394]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 15:39:16 client-18 xinetd[3390]: EXIT: shell status=0 pid=22393 duration=1(sec)
May 17 15:39:16 client-18 xinetd[3390]: START: shell pid=22429 from=192.168.4.5
May 17 15:39:16 client-18 rshd[22430]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 15:39:16 client-18 xinetd[3390]: EXIT: shell status=0 pid=22429 duration=0(sec)
May 17 15:39:16 client-18 xinetd[3390]: START: shell pid=22466 from=192.168.4.5
May 17 15:39:16 client-18 rshd[22467]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 74: Ensure applications don\'t fail waiting for OST recovery ==================== 15:39:16 \(1305671956\)");echo XXRETCODE:$?'
May 17 15:39:16 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 74: Ensure applications don't fail waiting for OST recovery ==================== 15:39:16 (1305671956)
May 17 15:39:16 client-18 xinetd[3390]: EXIT: shell status=0 pid=22466 duration=0(sec)
May 17 15:39:17 client-18 xinetd[3390]: START: shell pid=22501 from=192.168.4.5
May 17 15:39:17 client-18 rshd[22502]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:; lsof -t /mnt/lustre || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount  /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre is still busy, wait one second && sleep 1;     umount  /mnt/lustre; fi fi");echo XXRETCODE:$?'
May 17 15:39:18 client-18 kernel: LustreError: 22543:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -107 from cancel RPC: canceling anyway
May 17 15:39:18 client-18 kernel: LustreError: 22543:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -107
May 17 15:39:18 client-18 kernel: LustreError: 22543:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -11 from cancel RPC: canceling anyway
May 17 15:39:18 client-18 kernel: LustreError: 22543:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -11
May 17 15:39:18 client-18 kernel: Lustre: client ffff810317273c00 umount complete
May 17 15:39:18 client-18 xinetd[3390]: EXIT: shell status=0 pid=22501 duration=1(sec)
May 17 15:39:38 client-18 xinetd[3390]: START: shell pid=22546 from=192.168.4.5
May 17 15:39:38 client-18 rshd[22547]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c " running=\$(mount | grep -c /mnt/lustre' '); rc=0; if [ \$running -eq 0 ] ; then     mkdir -p /mnt/lustre;     mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre;     rc=\$?; fi; exit \$rc");echo XXRETCODE:$?'
May 17 15:39:38 client-18 kernel: Lustre: 22586:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null
May 17 15:39:38 client-18 kernel: Lustre: 22586:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 6 previous similar messages
May 17 15:39:38 client-18 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'.
May 17 15:39:38 client-18 kernel: Lustre: Increasing default stripe size to min 1048576
May 17 15:39:38 client-18 kernel: Lustre: 22586:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff810325d3f400->192.168.4.128@o2ib netid 50000: select flavor null
May 17 15:39:38 client-18 kernel: Lustre: Client lustre-client has started
May 17 15:39:38 client-18 xinetd[3390]: EXIT: shell status=0 pid=22546 duration=0(sec)
May 17 15:39:38 client-18 xinetd[3390]: START: shell pid=22599 from=192.168.4.5
May 17 15:39:38 client-18 rshd[22600]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "mount | grep -w /mnt/lustre");echo XXRETCODE:$?'
May 17 15:39:38 client-18 xinetd[3390]: EXIT: shell status=0 pid=22599 duration=0(sec)
May 17 15:39:38 client-18 xinetd[3390]: START: shell pid=22636 from=192.168.4.5
May 17 15:39:38 client-18 rshd[22637]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param debug=\"0x33f0406\";         lctl set_param subsystem_debug=\"0xffb7e3ff\";         lctl set_param debug_mb=32;");echo XXRETCODE:$?'
May 17 15:39:38 client-18 kernel: Lustre: 22671:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release.
May 17 15:39:38 client-18 kernel: Lustre: 22671:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message
May 17 15:39:38 client-18 xinetd[3390]: EXIT: shell status=0 pid=22636 duration=0(sec)
May 17 15:39:44 client-18 xinetd[3390]: START: shell pid=22673 from=192.168.4.5
May 17 15:39:44 client-18 rshd[22674]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 15:40:53 client-18 xinetd[3390]: EXIT: shell status=0 pid=22673 duration=69(sec)
May 17 15:40:53 client-18 xinetd[3390]: START: shell pid=22711 from=192.168.4.5
May 17 15:40:53 client-18 rshd[22712]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 15:40:53 client-18 xinetd[3390]: EXIT: shell status=0 pid=22711 duration=0(sec)
May 17 15:40:53 client-18 xinetd[3390]: START: shell pid=22747 from=192.168.4.5
May 17 15:40:53 client-18 rshd[22748]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 15:40:53 client-18 xinetd[3390]: EXIT: shell status=0 pid=22747 duration=0(sec)
May 17 15:40:53 client-18 xinetd[3390]: START: shell pid=22784 from=192.168.4.5
May 17 15:40:53 client-18 rshd[22785]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 80a: CMD: unlink cross-node dir \(fail mds with inode\) ========================== 15:40:53 \(1305672053\)");echo XXRETCODE:$?'
May 17 15:40:53 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 80a: CMD: unlink cross-node dir (fail mds with inode) ========================== 15:40:53 (1305672053)
May 17 15:40:53 client-18 xinetd[3390]: EXIT: shell status=0 pid=22784 duration=0(sec)
May 17 15:40:54 client-18 xinetd[3390]: START: shell pid=22819 from=192.168.4.5
May 17 15:40:54 client-18 rshd[22820]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark  SKIP: replay-single test_80a needs \>= 2 MDTs");echo XXRETCODE:$?'
May 17 15:40:54 client-18 kernel: Lustre: DEBUG MARKER: SKIP: replay-single test_80a needs >= 2 MDTs
May 17 15:40:54 client-18 xinetd[3390]: EXIT: shell status=0 pid=22819 duration=0(sec)
May 17 15:40:54 client-18 xinetd[3390]: START: shell pid=22854 from=192.168.4.5
May 17 15:40:54 client-18 rshd[22855]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 15:40:54 client-18 xinetd[3390]: EXIT: shell status=0 pid=22854 duration=0(sec)
May 17 15:40:54 client-18 xinetd[3390]: START: shell pid=22890 from=192.168.4.5
May 17 15:40:54 client-18 rshd[22891]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 15:40:54 client-18 xinetd[3390]: EXIT: shell status=0 pid=22890 duration=0(sec)
May 17 15:40:54 client-18 xinetd[3390]: START: shell pid=22927 from=192.168.4.5
May 17 15:40:54 client-18 rshd[22928]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 80b: CMD: unlink cross-node dir \(fail mds with name\) =========================== 15:40:54 \(1305672054\)");echo XXRETCODE:$?'
May 17 15:40:54 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 80b: CMD: unlink cross-node dir (fail mds with name) =========================== 15:40:54 (1305672054)
May 17 15:40:54 client-18 xinetd[3390]: EXIT: shell status=0 pid=22927 duration=0(sec)
May 17 15:40:54 client-18 xinetd[3390]: START: shell pid=22962 from=192.168.4.5
May 17 15:40:54 client-18 rshd[22963]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark  SKIP: replay-single test_80b needs \>= 2 MDTs");echo XXRETCODE:$?'
May 17 15:40:54 client-18 kernel: Lustre: DEBUG MARKER: SKIP: replay-single test_80b needs >= 2 MDTs
May 17 15:40:54 client-18 xinetd[3390]: EXIT: shell status=0 pid=22962 duration=0(sec)
May 17 15:40:54 client-18 xinetd[3390]: START: shell pid=22997 from=192.168.4.5
May 17 15:40:54 client-18 rshd[22998]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 15:40:54 client-18 xinetd[3390]: EXIT: shell status=0 pid=22997 duration=0(sec)
May 17 15:40:54 client-18 xinetd[3390]: START: shell pid=23033 from=192.168.4.5
May 17 15:40:55 client-18 rshd[23034]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 15:40:55 client-18 xinetd[3390]: EXIT: shell status=0 pid=23033 duration=1(sec)
May 17 15:40:55 client-18 xinetd[3390]: START: shell pid=23070 from=192.168.4.5
May 17 15:40:55 client-18 rshd[23071]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 81a: CMD: unlink cross-node file \(fail mds with name\) ========================== 15:40:55 \(1305672055\)");echo XXRETCODE:$?'
May 17 15:40:55 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 81a: CMD: unlink cross-node file (fail mds with name) ========================== 15:40:55 (1305672055)
May 17 15:40:55 client-18 xinetd[3390]: EXIT: shell status=0 pid=23070 duration=0(sec)
May 17 15:40:55 client-18 xinetd[3390]: START: shell pid=23105 from=192.168.4.5
May 17 15:40:55 client-18 rshd[23106]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark  SKIP: replay-single test_81a needs \>= 2 MDTs");echo XXRETCODE:$?'
May 17 15:40:55 client-18 kernel: Lustre: DEBUG MARKER: SKIP: replay-single test_81a needs >= 2 MDTs
May 17 15:40:55 client-18 xinetd[3390]: EXIT: shell status=0 pid=23105 duration=0(sec)
May 17 15:40:55 client-18 xinetd[3390]: START: shell pid=23140 from=192.168.4.5
May 17 15:40:55 client-18 rshd[23141]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 15:40:55 client-18 xinetd[3390]: EXIT: shell status=0 pid=23140 duration=0(sec)
May 17 15:40:55 client-18 xinetd[3390]: START: shell pid=23176 from=192.168.4.5
May 17 15:40:55 client-18 rshd[23177]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 15:40:55 client-18 xinetd[3390]: EXIT: shell status=0 pid=23176 duration=0(sec)
May 17 15:40:55 client-18 xinetd[3390]: START: shell pid=23213 from=192.168.4.5
May 17 15:40:55 client-18 rshd[23214]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 82a: CMD: mkdir cross-node dir \(fail mds with inode\) =========================== 15:40:55 \(1305672055\)");echo XXRETCODE:$?'
May 17 15:40:55 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 82a: CMD: mkdir cross-node dir (fail mds with inode) =========================== 15:40:55 (1305672055)
May 17 15:40:55 client-18 xinetd[3390]: EXIT: shell status=0 pid=23213 duration=0(sec)
May 17 15:40:55 client-18 xinetd[3390]: START: shell pid=23248 from=192.168.4.5
May 17 15:40:55 client-18 rshd[23249]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark  SKIP: replay-single test_82a needs \>= 2 MDTs");echo XXRETCODE:$?'
May 17 15:40:56 client-18 kernel: Lustre: DEBUG MARKER: SKIP: replay-single test_82a needs >= 2 MDTs
May 17 15:40:56 client-18 xinetd[3390]: EXIT: shell status=0 pid=23248 duration=1(sec)
May 17 15:40:56 client-18 xinetd[3390]: START: shell pid=23283 from=192.168.4.5
May 17 15:40:56 client-18 rshd[23284]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 15:40:56 client-18 xinetd[3390]: EXIT: shell status=0 pid=23283 duration=0(sec)
May 17 15:40:56 client-18 xinetd[3390]: START: shell pid=23319 from=192.168.4.5
May 17 15:40:56 client-18 rshd[23320]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 15:40:56 client-18 xinetd[3390]: EXIT: shell status=0 pid=23319 duration=0(sec)
May 17 15:40:56 client-18 xinetd[3390]: START: shell pid=23356 from=192.168.4.5
May 17 15:40:56 client-18 rshd[23357]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 82b: CMD: mkdir cross-node dir \(fail mds with name\) ============================ 15:40:56 \(1305672056\)");echo XXRETCODE:$?'
May 17 15:40:56 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 82b: CMD: mkdir cross-node dir (fail mds with name) ============================ 15:40:56 (1305672056)
May 17 15:40:56 client-18 xinetd[3390]: EXIT: shell status=0 pid=23356 duration=0(sec)
May 17 15:40:56 client-18 xinetd[3390]: START: shell pid=23391 from=192.168.4.5
May 17 15:40:56 client-18 rshd[23392]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark  SKIP: replay-single test_82b needs \>= 2 MDTs");echo XXRETCODE:$?'
May 17 15:40:56 client-18 kernel: Lustre: DEBUG MARKER: SKIP: replay-single test_82b needs >= 2 MDTs
May 17 15:40:56 client-18 xinetd[3390]: EXIT: shell status=0 pid=23391 duration=0(sec)
May 17 15:40:56 client-18 xinetd[3390]: START: shell pid=23426 from=192.168.4.5
May 17 15:40:56 client-18 rshd[23427]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 15:40:56 client-18 xinetd[3390]: EXIT: shell status=0 pid=23426 duration=0(sec)
May 17 15:40:56 client-18 xinetd[3390]: START: shell pid=23462 from=192.168.4.5
May 17 15:40:56 client-18 rshd[23463]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 15:40:56 client-18 xinetd[3390]: EXIT: shell status=0 pid=23462 duration=0(sec)
May 17 15:40:57 client-18 xinetd[3390]: START: shell pid=23499 from=192.168.4.5
May 17 15:40:57 client-18 rshd[23500]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 83a: fail log_add during unlink recovery ======================================= 15:40:57 \(1305672057\)");echo XXRETCODE:$?'
May 17 15:40:57 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 83a: fail log_add during unlink recovery ======================================= 15:40:57 (1305672057)
May 17 15:40:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=23499 duration=0(sec)
May 17 15:40:57 client-18 xinetd[3390]: START: shell pid=23534 from=192.168.4.5
May 17 15:40:57 client-18 rshd[23535]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 15:40:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=23534 duration=0(sec)
May 17 15:40:57 client-18 xinetd[3390]: START: shell pid=23570 from=192.168.4.5
May 17 15:40:57 client-18 rshd[23571]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 15:40:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=23570 duration=0(sec)
May 17 15:40:57 client-18 xinetd[3390]: START: shell pid=23607 from=192.168.4.5
May 17 15:40:57 client-18 rshd[23608]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 83b: fail log_add during unlink recovery ======================================= 15:40:57 \(1305672057\)");echo XXRETCODE:$?'
May 17 15:40:57 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 83b: fail log_add during unlink recovery ======================================= 15:40:57 (1305672057)
May 17 15:40:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=23607 duration=0(sec)
May 17 15:40:57 client-18 xinetd[3390]: START: shell pid=23642 from=192.168.4.5
May 17 15:40:58 client-18 rshd[23643]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 15:40:58 client-18 xinetd[3390]: EXIT: shell status=0 pid=23642 duration=1(sec)
May 17 15:41:18 client-18 xinetd[3390]: START: shell pid=23685 from=192.168.4.5
May 17 15:41:18 client-18 rshd[23686]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 15:41:20 client-18 kernel: Lustre: 3861:0:(import.c:529:import_select_connection()) MGC192.168.4.128@o2ib: tried all connections, increasing latency to 6s
May 17 15:41:20 client-18 kernel: Lustre: 3861:0:(import.c:529:import_select_connection()) Skipped 17 previous similar messages
May 17 15:41:20 client-18 kernel: Lustre: 3860:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74f225f0 to 0x62e5a8d74f22f43
May 17 15:41:20 client-18 xinetd[3390]: EXIT: shell status=0 pid=23685 duration=2(sec)
May 17 15:41:20 client-18 xinetd[3390]: START: shell pid=23721 from=192.168.4.5
May 17 15:41:20 client-18 rshd[23722]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 15:41:20 client-18 xinetd[3390]: EXIT: shell status=0 pid=23721 duration=0(sec)
May 17 15:41:20 client-18 xinetd[3390]: START: shell pid=23757 from=192.168.4.5
May 17 15:41:20 client-18 rshd[23758]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 15:41:20 client-18 xinetd[3390]: EXIT: shell status=0 pid=23757 duration=0(sec)
May 17 15:41:21 client-18 xinetd[3390]: START: shell pid=23794 from=192.168.4.5
May 17 15:41:21 client-18 rshd[23795]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 84a: stale open during export disconnect ======================================= 15:41:20 \(1305672080\)");echo XXRETCODE:$?'
May 17 15:41:21 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 84a: stale open during export disconnect ======================================= 15:41:20 (1305672080)
May 17 15:41:21 client-18 xinetd[3390]: EXIT: shell status=0 pid=23794 duration=0(sec)
May 17 15:41:31 client-18 xinetd[3390]: START: shell pid=23831 from=192.168.4.5
May 17 15:41:31 client-18 rshd[23832]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 15:41:31 client-18 xinetd[3390]: EXIT: shell status=0 pid=23831 duration=0(sec)
May 17 15:41:31 client-18 xinetd[3390]: START: shell pid=23867 from=192.168.4.5
May 17 15:41:31 client-18 rshd[23868]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 15:41:31 client-18 xinetd[3390]: EXIT: shell status=0 pid=23867 duration=0(sec)
May 17 15:41:32 client-18 xinetd[3390]: START: shell pid=23904 from=192.168.4.5
May 17 15:41:32 client-18 rshd[23905]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 85a: check the cancellation of unused locks during recovery\(IBITS\) ============= 15:41:31 \(1305672091\)");echo XXRETCODE:$?'
May 17 15:41:32 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 85a: check the cancellation of unused locks during recovery(IBITS) ============= 15:41:31 (1305672091)
May 17 15:41:32 client-18 xinetd[3390]: EXIT: shell status=0 pid=23904 duration=0(sec)
May 17 15:41:47 client-18 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail.
May 17 15:41:47 client-18 kernel: LustreError: Skipped 11 previous similar messages
May 17 15:41:52 client-18 xinetd[3390]: START: shell pid=23939 from=192.168.4.5
May 17 15:41:52 client-18 rshd[23940]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 15:42:02 client-18 kernel: Lustre: 3860:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74f22f43 to 0x62e5a8d74f25315
May 17 15:42:02 client-18 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import
May 17 15:42:02 client-18 kernel: Lustre: Skipped 12 previous similar messages
May 17 15:42:02 client-18 xinetd[3390]: EXIT: shell status=0 pid=23939 duration=10(sec)
May 17 15:42:02 client-18 xinetd[3390]: START: shell pid=23974 from=192.168.4.5
May 17 15:42:02 client-18 rshd[23975]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 15:42:02 client-18 xinetd[3390]: EXIT: shell status=0 pid=23974 duration=0(sec)
May 17 15:42:02 client-18 xinetd[3390]: START: shell pid=24010 from=192.168.4.5
May 17 15:42:02 client-18 rshd[24011]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 15:42:02 client-18 xinetd[3390]: EXIT: shell status=0 pid=24010 duration=0(sec)
May 17 15:42:02 client-18 xinetd[3390]: START: shell pid=24047 from=192.168.4.5
May 17 15:42:03 client-18 rshd[24048]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 85b: check the cancellation of unused locks during recovery\(EXTENT\) ============ 15:42:02 \(1305672122\)");echo XXRETCODE:$?'
May 17 15:42:03 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 85b: check the cancellation of unused locks during recovery(EXTENT) ============ 15:42:02 (1305672122)
May 17 15:42:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=24047 duration=1(sec)
May 17 15:42:07 client-18 kernel: Lustre: Increasing default stripe size to min 1048576
May 17 15:42:22 client-18 xinetd[3390]: START: shell pid=24084 from=192.168.4.5
May 17 15:42:22 client-18 rshd[24085]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 15:42:31 client-18 xinetd[3390]: EXIT: shell status=0 pid=24084 duration=9(sec)
May 17 15:42:31 client-18 xinetd[3390]: START: shell pid=24122 from=192.168.4.5
May 17 15:42:31 client-18 rshd[24123]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 15:42:31 client-18 xinetd[3390]: EXIT: shell status=0 pid=24122 duration=0(sec)
May 17 15:42:31 client-18 xinetd[3390]: START: shell pid=24158 from=192.168.4.5
May 17 15:42:31 client-18 rshd[24159]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 15:42:31 client-18 xinetd[3390]: EXIT: shell status=0 pid=24158 duration=0(sec)
May 17 15:42:31 client-18 xinetd[3390]: START: shell pid=24195 from=192.168.4.5
May 17 15:42:31 client-18 rshd[24196]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 86: umount server after clear nid_stats should not hit LBUG ==================== 15:42:31 \(1305672151\)");echo XXRETCODE:$?'
May 17 15:42:31 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 86: umount server after clear nid_stats should not hit LBUG ==================== 15:42:31 (1305672151)
May 17 15:42:31 client-18 xinetd[3390]: EXIT: shell status=0 pid=24195 duration=0(sec)
May 17 15:42:31 client-18 xinetd[3390]: START: shell pid=24230 from=192.168.4.5
May 17 15:42:31 client-18 rshd[24231]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:; lsof -t /mnt/lustre || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount  /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre is still busy, wait one second && sleep 1;     umount  /mnt/lustre; fi fi");echo XXRETCODE:$?'
May 17 15:42:32 client-18 kernel: LustreError: 24272:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway
May 17 15:42:32 client-18 kernel: LustreError: 24272:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Skipped 15 previous similar messages
May 17 15:42:32 client-18 kernel: LustreError: 24272:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108
May 17 15:42:32 client-18 kernel: LustreError: 24272:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) Skipped 15 previous similar messages
May 17 15:42:32 client-18 kernel: Lustre: client ffff810325d3f400 umount complete
May 17 15:42:32 client-18 xinetd[3390]: EXIT: shell status=0 pid=24230 duration=1(sec)
May 17 15:42:42 client-18 xinetd[3390]: START: shell pid=24274 from=192.168.4.5
May 17 15:42:42 client-18 rshd[24275]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c " running=\$(mount | grep -c /mnt/lustre' '); rc=0; if [ \$running -eq 0 ] ; then     mkdir -p /mnt/lustre;     mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre;     rc=\$?; fi; exit \$rc");echo XXRETCODE:$?'
May 17 15:42:42 client-18 kernel: Lustre: 24314:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null
May 17 15:42:42 client-18 kernel: Lustre: 24314:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 6 previous similar messages
May 17 15:42:42 client-18 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'.
May 17 15:42:42 client-18 kernel: Lustre: Increasing default stripe size to min 1048576
May 17 15:42:42 client-18 kernel: Lustre: Client lustre-client has started
May 17 15:42:42 client-18 xinetd[3390]: EXIT: shell status=0 pid=24274 duration=0(sec)
May 17 15:42:42 client-18 xinetd[3390]: START: shell pid=24327 from=192.168.4.5
May 17 15:42:42 client-18 rshd[24328]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "mount | grep -w /mnt/lustre");echo XXRETCODE:$?'
May 17 15:42:42 client-18 xinetd[3390]: EXIT: shell status=0 pid=24327 duration=0(sec)
May 17 15:42:42 client-18 xinetd[3390]: START: shell pid=24364 from=192.168.4.5
May 17 15:42:42 client-18 rshd[24365]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param debug=\"0x33f0406\";         lctl set_param subsystem_debug=\"0xffb7e3ff\";         lctl set_param debug_mb=32;");echo XXRETCODE:$?'
May 17 15:42:42 client-18 kernel: Lustre: 24399:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release.
May 17 15:42:42 client-18 kernel: Lustre: 24399:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message
May 17 15:42:42 client-18 xinetd[3390]: EXIT: shell status=0 pid=24364 duration=0(sec)
May 17 15:42:42 client-18 xinetd[3390]: START: shell pid=24401 from=192.168.4.5
May 17 15:42:42 client-18 rshd[24402]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 15:42:42 client-18 xinetd[3390]: EXIT: shell status=0 pid=24401 duration=0(sec)
May 17 15:42:43 client-18 xinetd[3390]: START: shell pid=24437 from=192.168.4.5
May 17 15:42:43 client-18 rshd[24438]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 15:42:43 client-18 xinetd[3390]: EXIT: shell status=0 pid=24437 duration=0(sec)
May 17 15:42:43 client-18 xinetd[3390]: START: shell pid=24474 from=192.168.4.5
May 17 15:42:43 client-18 rshd[24475]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 87: write replay =============================================================== 15:42:43 \(1305672163\)");echo XXRETCODE:$?'
May 17 15:42:43 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 87: write replay =============================================================== 15:42:43 (1305672163)
May 17 15:42:43 client-18 xinetd[3390]: EXIT: shell status=0 pid=24474 duration=0(sec)
May 17 15:42:43 client-18 xinetd[3390]: START: shell pid=24509 from=192.168.4.5
May 17 15:42:43 client-18 rshd[24510]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 15:42:43 client-18 xinetd[3390]: EXIT: shell status=0 pid=24509 duration=0(sec)
May 17 15:42:59 client-18 xinetd[3390]: START: shell pid=24548 from=192.168.4.5
May 17 15:42:59 client-18 rshd[24549]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 15:43:02 client-18 xinetd[3390]: EXIT: shell status=0 pid=24548 duration=3(sec)
May 17 15:43:02 client-18 xinetd[3390]: START: shell pid=24582 from=192.168.4.5
May 17 15:43:02 client-18 rshd[24583]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 15:43:02 client-18 xinetd[3390]: EXIT: shell status=0 pid=24582 duration=0(sec)
May 17 15:43:02 client-18 xinetd[3390]: START: shell pid=24618 from=192.168.4.5
May 17 15:43:03 client-18 rshd[24619]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 15:43:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=24618 duration=1(sec)
May 17 15:43:03 client-18 xinetd[3390]: START: shell pid=24655 from=192.168.4.5
May 17 15:43:03 client-18 rshd[24656]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 87b: write replay with changed data \(checksum resend\) ========================== 15:43:03 \(1305672183\)");echo XXRETCODE:$?'
May 17 15:43:03 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 87b: write replay with changed data (checksum resend) ========================== 15:43:03 (1305672183)
May 17 15:43:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=24655 duration=0(sec)
May 17 15:43:03 client-18 xinetd[3390]: START: shell pid=24690 from=192.168.4.5
May 17 15:43:03 client-18 rshd[24691]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 15:43:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=24690 duration=0(sec)
May 17 15:43:19 client-18 xinetd[3390]: START: shell pid=24729 from=192.168.4.5
May 17 15:43:19 client-18 rshd[24730]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 15:43:22 client-18 xinetd[3390]: EXIT: shell status=0 pid=24729 duration=3(sec)
May 17 15:43:22 client-18 xinetd[3390]: START: shell pid=24763 from=192.168.4.5
May 17 15:43:22 client-18 rshd[24764]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 15:43:22 client-18 xinetd[3390]: EXIT: shell status=0 pid=24763 duration=0(sec)
May 17 15:43:22 client-18 xinetd[3390]: START: shell pid=24799 from=192.168.4.5
May 17 15:43:22 client-18 rshd[24800]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 15:43:23 client-18 xinetd[3390]: EXIT: shell status=0 pid=24799 duration=1(sec)
May 17 15:43:23 client-18 xinetd[3390]: START: shell pid=24836 from=192.168.4.5
May 17 15:43:23 client-18 rshd[24837]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 88: MDS should not assign same objid to different files ======================== 15:43:23 \(1305672203\)");echo XXRETCODE:$?'
May 17 15:43:23 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 88: MDS should not assign same objid to different files ======================== 15:43:23 (1305672203)
May 17 15:43:23 client-18 xinetd[3390]: EXIT: shell status=0 pid=24836 duration=0(sec)
May 17 15:43:23 client-18 xinetd[3390]: START: shell pid=24871 from=192.168.4.5
May 17 15:43:23 client-18 rshd[24872]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 15:43:23 client-18 xinetd[3390]: EXIT: shell status=0 pid=24871 duration=0(sec)
May 17 15:43:23 client-18 xinetd[3390]: START: shell pid=24910 from=192.168.4.5
May 17 15:43:24 client-18 rshd[24911]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 15:43:24 client-18 xinetd[3390]: EXIT: shell status=0 pid=24910 duration=1(sec)
May 17 15:43:44 client-18 kernel: Lustre: 3860:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74f26af4 to 0x62e5a8d74f27512
May 17 15:43:57 client-18 xinetd[3390]: START: shell pid=24957 from=192.168.4.5
May 17 15:43:57 client-18 rshd[24958]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 15:44:00 client-18 xinetd[3390]: EXIT: shell status=0 pid=24957 duration=3(sec)
May 17 15:44:02 client-18 xinetd[3390]: START: shell pid=24991 from=192.168.4.5
May 17 15:44:03 client-18 rshd[24992]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 15:44:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=24991 duration=1(sec)
May 17 15:44:03 client-18 xinetd[3390]: START: shell pid=25027 from=192.168.4.5
May 17 15:44:03 client-18 rshd[25028]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 15:44:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=25027 duration=0(sec)
May 17 15:44:03 client-18 xinetd[3390]: START: shell pid=25064 from=192.168.4.5
May 17 15:44:03 client-18 rshd[25065]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single test 89: no disk space leak on late ost connection ================================== 15:44:03 \(1305672243\)");echo XXRETCODE:$?'
May 17 15:44:03 client-18 kernel: Lustre: DEBUG MARKER: == replay-single test 89: no disk space leak on late ost connection ================================== 15:44:03 (1305672243)
May 17 15:44:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=25064 duration=0(sec)
May 17 15:44:36 client-18 kernel: Lustre: 3860:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74f27512 to 0x62e5a8d74f28709
May 17 15:46:02 client-18 kernel: Lustre: lustre-OST0000-osc-ffff810313c17000: Connection restored to service lustre-OST0000 using nid 192.168.4.129@o2ib.
May 17 15:46:02 client-18 kernel: Lustre: Skipped 22 previous similar messages
May 17 15:46:07 client-18 xinetd[3390]: START: shell pid=25106 from=192.168.4.5
May 17 15:46:07 client-18 rshd[25107]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 15:46:07 client-18 xinetd[3390]: EXIT: shell status=0 pid=25106 duration=0(sec)
May 17 15:46:07 client-18 xinetd[3390]: START: shell pid=25142 from=192.168.4.5
May 17 15:46:07 client-18 rshd[25143]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 15:46:07 client-18 xinetd[3390]: EXIT: shell status=0 pid=25142 duration=0(sec)
May 17 15:46:07 client-18 xinetd[3390]: START: shell pid=25179 from=192.168.4.5
May 17 15:46:07 client-18 rshd[25180]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-single replay-single.sh test complete, duration 4260 sec =================================== 15:46:07 \(1305672367\)");echo XXRETCODE:$?'
May 17 15:46:07 client-18 kernel: Lustre: DEBUG MARKER: == replay-single replay-single.sh test complete, duration 4260 sec =================================== 15:46:07 (1305672367)
May 17 15:46:07 client-18 xinetd[3390]: EXIT: shell status=0 pid=25179 duration=0(sec)
May 17 15:46:08 client-18 xinetd[3390]: START: shell pid=25214 from=192.168.4.5
May 17 15:46:08 client-18 rshd[25215]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark -----============= acceptance-small: conf-sanity ============----- Tue May 17 15:46:07 PDT 2011");echo XXRETCODE:$?'
May 17 15:46:08 client-18 kernel: Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue May 17 15:46:07 PDT 2011
May 17 15:46:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=25214 duration=0(sec)
May 17 15:46:08 client-18 xinetd[3390]: START: shell pid=25249 from=192.168.4.5
May 17 15:46:08 client-18 rshd[25250]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark excepting tests: ");echo XXRETCODE:$?'
May 17 15:46:08 client-18 kernel: Lustre: DEBUG MARKER: excepting tests:
May 17 15:46:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=25249 duration=0(sec)
May 17 15:46:08 client-18 xinetd[3390]: START: shell pid=25284 from=192.168.4.5
May 17 15:46:08 client-18 rshd[25285]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:; lsof -t /mnt/lustre || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount  /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre is still busy, wait one second && sleep 1;     umount  /mnt/lustre; fi fi");echo XXRETCODE:$?'
May 17 15:46:08 client-18 kernel: LustreError: 25326:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway
May 17 15:46:08 client-18 kernel: LustreError: 25326:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Skipped 1 previous similar message
May 17 15:46:08 client-18 kernel: LustreError: 25326:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108
May 17 15:46:08 client-18 kernel: LustreError: 25326:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) Skipped 1 previous similar message
May 17 15:46:08 client-18 kernel: Lustre: client ffff810313c17000 umount complete
May 17 15:46:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=25284 duration=0(sec)
May 17 15:46:08 client-18 xinetd[3390]: START: shell pid=25328 from=192.168.4.5
May 17 15:46:08 client-18 rshd[25329]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:; lsof -t /mnt/lustre2 || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre2 | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount  /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre2 is still busy, wait one second && sleep 1;     umount  /mnt/lustre2; fi fi");echo XXRETCODE:$?'
May 17 15:46:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=25328 duration=0(sec)
May 17 15:46:45 client-18 xinetd[3390]: START: shell pid=25369 from=192.168.4.5
May 17 15:46:45 client-18 rshd[25370]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark ost now in FULL state");echo XXRETCODE:$?'
May 17 15:46:45 client-18 kernel: Lustre: DEBUG MARKER: ost now in FULL state
May 17 15:46:45 client-18 xinetd[3390]: EXIT: shell status=0 pid=25369 duration=0(sec)
May 17 15:46:57 client-18 xinetd[3390]: START: shell pid=25404 from=192.168.4.5
May 17 15:46:57 client-18 rshd[25405]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == conf-sanity test 0: single mount setup ============================================================ 15:46:57 \(1305672417\)");echo XXRETCODE:$?'
May 17 15:46:57 client-18 kernel: Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ============================================================ 15:46:57 (1305672417)
May 17 15:46:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=25404 duration=0(sec)
May 17 15:47:33 client-18 xinetd[3390]: START: shell pid=25443 from=192.168.4.5
May 17 15:47:33 client-18 rshd[25444]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 15:47:34 client-18 xinetd[3390]: EXIT: shell status=0 pid=25443 duration=1(sec)
May 17 15:47:34 client-18 xinetd[3390]: START: shell pid=25479 from=192.168.4.5
May 17 15:47:34 client-18 rshd[25480]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 15:47:34 client-18 xinetd[3390]: EXIT: shell status=0 pid=25479 duration=0(sec)
May 17 15:47:35 client-18 xinetd[3390]: START: shell pid=25516 from=192.168.4.5
May 17 15:47:35 client-18 rshd[25517]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == conf-sanity test 1: start up ost twice \(should return errors\) ===================================== 15:47:34 \(1305672454\)");echo XXRETCODE:$?'
May 17 15:47:35 client-18 kernel: Lustre: DEBUG MARKER: == conf-sanity test 1: start up ost twice (should return errors) ===================================== 15:47:34 (1305672454)
May 17 15:47:35 client-18 xinetd[3390]: EXIT: shell status=0 pid=25516 duration=0(sec)
May 17 15:48:13 client-18 xinetd[3390]: START: shell pid=25551 from=192.168.4.5
May 17 15:48:13 client-18 rshd[25552]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 15:48:13 client-18 xinetd[3390]: EXIT: shell status=0 pid=25551 duration=0(sec)
May 17 15:48:13 client-18 xinetd[3390]: START: shell pid=25587 from=192.168.4.5
May 17 15:48:13 client-18 rshd[25588]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 15:48:13 client-18 xinetd[3390]: EXIT: shell status=0 pid=25587 duration=0(sec)
May 17 15:48:14 client-18 xinetd[3390]: START: shell pid=25624 from=192.168.4.5
May 17 15:48:14 client-18 rshd[25625]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == conf-sanity test 2: start up mds twice \(should return err\) ======================================== 15:48:13 \(1305672493\)");echo XXRETCODE:$?'
May 17 15:48:14 client-18 kernel: Lustre: DEBUG MARKER: == conf-sanity test 2: start up mds twice (should return err) ======================================== 15:48:13 (1305672493)
May 17 15:48:14 client-18 xinetd[3390]: EXIT: shell status=0 pid=25624 duration=0(sec)
May 17 15:48:47 client-18 xinetd[3390]: START: shell pid=25663 from=192.168.4.5
May 17 15:48:47 client-18 rshd[25664]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 15:48:47 client-18 xinetd[3390]: EXIT: shell status=0 pid=25663 duration=0(sec)
May 17 15:48:47 client-18 xinetd[3390]: START: shell pid=25699 from=192.168.4.5
May 17 15:48:47 client-18 rshd[25700]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 15:48:47 client-18 xinetd[3390]: EXIT: shell status=0 pid=25699 duration=0(sec)
May 17 15:48:48 client-18 xinetd[3390]: START: shell pid=25736 from=192.168.4.5
May 17 15:48:48 client-18 rshd[25737]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == conf-sanity test 3: mount client twice \(should return err\) ======================================== 15:48:47 \(1305672527\)");echo XXRETCODE:$?'
May 17 15:48:48 client-18 kernel: Lustre: DEBUG MARKER: == conf-sanity test 3: mount client twice (should return err) ======================================== 15:48:47 (1305672527)
May 17 15:48:48 client-18 xinetd[3390]: EXIT: shell status=0 pid=25736 duration=0(sec)
May 17 15:49:26 client-18 xinetd[3390]: START: shell pid=25771 from=192.168.4.5
May 17 15:49:26 client-18 rshd[25772]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 15:49:26 client-18 xinetd[3390]: EXIT: shell status=0 pid=25771 duration=0(sec)
May 17 15:49:26 client-18 xinetd[3390]: START: shell pid=25807 from=192.168.4.5
May 17 15:49:26 client-18 rshd[25808]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 15:49:26 client-18 xinetd[3390]: EXIT: shell status=0 pid=25807 duration=0(sec)
May 17 15:49:26 client-18 xinetd[3390]: START: shell pid=25844 from=192.168.4.5
May 17 15:49:27 client-18 rshd[25845]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == conf-sanity test 4: force cleanup ost, then cleanup =============================================== 15:49:26 \(1305672566\)");echo XXRETCODE:$?'
May 17 15:49:27 client-18 kernel: Lustre: DEBUG MARKER: == conf-sanity test 4: force cleanup ost, then cleanup =============================================== 15:49:26 (1305672566)
May 17 15:49:27 client-18 xinetd[3390]: EXIT: shell status=0 pid=25844 duration=1(sec)
May 17 15:50:04 client-18 xinetd[3390]: START: shell pid=25883 from=192.168.4.5
May 17 15:50:04 client-18 rshd[25884]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 15:50:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=25883 duration=0(sec)
May 17 15:50:04 client-18 xinetd[3390]: START: shell pid=25919 from=192.168.4.5
May 17 15:50:04 client-18 rshd[25920]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 15:50:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=25919 duration=0(sec)
May 17 15:50:05 client-18 xinetd[3390]: START: shell pid=25956 from=192.168.4.5
May 17 15:50:05 client-18 rshd[25957]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == conf-sanity test 5a: force cleanup mds, then cleanup ============================================== 15:50:04 \(1305672604\)");echo XXRETCODE:$?'
May 17 15:50:05 client-18 kernel: Lustre: DEBUG MARKER: == conf-sanity test 5a: force cleanup mds, then cleanup ============================================== 15:50:04 (1305672604)
May 17 15:50:05 client-18 xinetd[3390]: EXIT: shell status=0 pid=25956 duration=0(sec)
May 17 15:50:53 client-18 xinetd[3390]: START: shell pid=25991 from=192.168.4.5
May 17 15:50:53 client-18 rshd[25992]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 15:50:53 client-18 xinetd[3390]: EXIT: shell status=0 pid=25991 duration=0(sec)
May 17 15:50:53 client-18 xinetd[3390]: START: shell pid=26027 from=192.168.4.5
May 17 15:50:53 client-18 rshd[26028]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 15:50:53 client-18 xinetd[3390]: EXIT: shell status=0 pid=26027 duration=0(sec)
May 17 15:50:54 client-18 xinetd[3390]: START: shell pid=26064 from=192.168.4.5
May 17 15:50:54 client-18 rshd[26065]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == conf-sanity test 5b: Try to start a client with no MGS \(should return errs\) ======================= 15:50:53 \(1305672653\)");echo XXRETCODE:$?'
May 17 15:50:54 client-18 kernel: Lustre: DEBUG MARKER: == conf-sanity test 5b: Try to start a client with no MGS (should return errs) ======================= 15:50:53 (1305672653)
May 17 15:50:54 client-18 xinetd[3390]: EXIT: shell status=0 pid=26064 duration=0(sec)
May 17 15:52:41 client-18 xinetd[3390]: START: shell pid=26107 from=192.168.4.5
May 17 15:52:41 client-18 rshd[26108]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 15:52:41 client-18 xinetd[3390]: EXIT: shell status=0 pid=26107 duration=0(sec)
May 17 15:52:41 client-18 xinetd[3390]: START: shell pid=26143 from=192.168.4.5
May 17 15:52:41 client-18 rshd[26144]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 15:52:41 client-18 xinetd[3390]: EXIT: shell status=0 pid=26143 duration=0(sec)
May 17 15:52:42 client-18 xinetd[3390]: START: shell pid=26180 from=192.168.4.5
May 17 15:52:42 client-18 rshd[26181]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == conf-sanity test 5c: cleanup after failed mount \(bug 2712\) \(should return errs\) =================== 15:52:41 \(1305672761\)");echo XXRETCODE:$?'
May 17 15:52:42 client-18 kernel: Lustre: DEBUG MARKER: == conf-sanity test 5c: cleanup after failed mount (bug 2712) (should return errs) =================== 15:52:41 (1305672761)
May 17 15:52:42 client-18 xinetd[3390]: EXIT: shell status=0 pid=26180 duration=0(sec)
May 17 15:53:03 client-18 xinetd[3390]: START: shell pid=26215 from=192.168.4.5
May 17 15:53:03 client-18 rshd[26216]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 15:53:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=26215 duration=1(sec)
May 17 15:53:04 client-18 xinetd[3390]: START: shell pid=26251 from=192.168.4.5
May 17 15:53:04 client-18 rshd[26252]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 15:53:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=26251 duration=0(sec)
May 17 15:53:04 client-18 xinetd[3390]: START: shell pid=26288 from=192.168.4.5
May 17 15:53:04 client-18 rshd[26289]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == conf-sanity test 5d: mount with ost down ========================================================== 15:53:04 \(1305672784\)");echo XXRETCODE:$?'
May 17 15:53:04 client-18 kernel: Lustre: DEBUG MARKER: == conf-sanity test 5d: mount with ost down ========================================================== 15:53:04 (1305672784)
May 17 15:53:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=26288 duration=0(sec)
May 17 15:54:00 client-18 xinetd[3390]: START: shell pid=26327 from=192.168.4.5
May 17 15:54:00 client-18 rshd[26328]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 15:54:00 client-18 xinetd[3390]: EXIT: shell status=0 pid=26327 duration=0(sec)
May 17 15:54:00 client-18 xinetd[3390]: START: shell pid=26363 from=192.168.4.5
May 17 15:54:00 client-18 rshd[26364]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 15:54:00 client-18 xinetd[3390]: EXIT: shell status=0 pid=26363 duration=0(sec)
May 17 15:54:01 client-18 xinetd[3390]: START: shell pid=26400 from=192.168.4.5
May 17 15:54:01 client-18 rshd[26401]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == conf-sanity test 5e: delayed connect, don\'t crash \(bug 10268\) ===================================== 15:54:00 \(1305672840\)");echo XXRETCODE:$?'
May 17 15:54:01 client-18 kernel: Lustre: DEBUG MARKER: == conf-sanity test 5e: delayed connect, don't crash (bug 10268) ===================================== 15:54:00 (1305672840)
May 17 15:54:01 client-18 xinetd[3390]: EXIT: shell status=0 pid=26400 duration=0(sec)
May 17 15:54:37 client-18 xinetd[3390]: START: shell pid=26435 from=192.168.4.5
May 17 15:54:37 client-18 rshd[26436]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 15:54:37 client-18 xinetd[3390]: EXIT: shell status=0 pid=26435 duration=0(sec)
May 17 15:54:37 client-18 xinetd[3390]: START: shell pid=26471 from=192.168.4.5
May 17 15:54:37 client-18 rshd[26472]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 15:54:37 client-18 xinetd[3390]: EXIT: shell status=0 pid=26471 duration=0(sec)
May 17 15:54:38 client-18 xinetd[3390]: START: shell pid=26508 from=192.168.4.5
May 17 15:54:38 client-18 rshd[26509]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == conf-sanity test 5f: mds down, cleanup after failed mount \(bug 2712\) ============================== 15:54:37 \(1305672877\)");echo XXRETCODE:$?'
May 17 15:54:38 client-18 kernel: Lustre: DEBUG MARKER: == conf-sanity test 5f: mds down, cleanup after failed mount (bug 2712) ============================== 15:54:37 (1305672877)
May 17 15:54:38 client-18 xinetd[3390]: EXIT: shell status=0 pid=26508 duration=0(sec)
May 17 15:54:38 client-18 xinetd[3390]: START: shell pid=26543 from=192.168.4.5
May 17 15:54:38 client-18 rshd[26544]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark  SKIP: conf-sanity test_5f combined mgs and mds");echo XXRETCODE:$?'
May 17 15:54:38 client-18 kernel: Lustre: DEBUG MARKER: SKIP: conf-sanity test_5f combined mgs and mds
May 17 15:54:38 client-18 xinetd[3390]: EXIT: shell status=0 pid=26543 duration=0(sec)
May 17 15:54:38 client-18 xinetd[3390]: START: shell pid=26578 from=192.168.4.5
May 17 15:54:38 client-18 rshd[26579]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 15:54:38 client-18 xinetd[3390]: EXIT: shell status=0 pid=26578 duration=0(sec)
May 17 15:54:38 client-18 xinetd[3390]: START: shell pid=26614 from=192.168.4.5
May 17 15:54:38 client-18 rshd[26615]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 15:54:38 client-18 xinetd[3390]: EXIT: shell status=0 pid=26614 duration=0(sec)
May 17 15:54:38 client-18 xinetd[3390]: START: shell pid=26651 from=192.168.4.5
May 17 15:54:38 client-18 rshd[26652]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == conf-sanity test 6: manual umount, then mount again =============================================== 15:54:38 \(1305672878\)");echo XXRETCODE:$?'
May 17 15:54:38 client-18 kernel: Lustre: DEBUG MARKER: == conf-sanity test 6: manual umount, then mount again =============================================== 15:54:38 (1305672878)
May 17 15:54:38 client-18 xinetd[3390]: EXIT: shell status=0 pid=26651 duration=0(sec)
May 17 15:55:16 client-18 xinetd[3390]: START: shell pid=26690 from=192.168.4.5
May 17 15:55:16 client-18 rshd[26691]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 15:55:16 client-18 xinetd[3390]: EXIT: shell status=0 pid=26690 duration=0(sec)
May 17 15:55:17 client-18 xinetd[3390]: START: shell pid=26726 from=192.168.4.5
May 17 15:55:17 client-18 rshd[26727]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 15:55:17 client-18 xinetd[3390]: EXIT: shell status=0 pid=26726 duration=0(sec)
May 17 15:55:17 client-18 xinetd[3390]: START: shell pid=26763 from=192.168.4.5
May 17 15:55:17 client-18 rshd[26764]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == conf-sanity test 7: manual umount, then cleanup =================================================== 15:55:17 \(1305672917\)");echo XXRETCODE:$?'
May 17 15:55:17 client-18 kernel: Lustre: DEBUG MARKER: == conf-sanity test 7: manual umount, then cleanup =================================================== 15:55:17 (1305672917)
May 17 15:55:17 client-18 xinetd[3390]: EXIT: shell status=0 pid=26763 duration=0(sec)
May 17 15:55:52 client-18 xinetd[3390]: START: shell pid=26798 from=192.168.4.5
May 17 15:55:52 client-18 rshd[26799]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 15:55:52 client-18 xinetd[3390]: EXIT: shell status=0 pid=26798 duration=0(sec)
May 17 15:55:52 client-18 xinetd[3390]: START: shell pid=26834 from=192.168.4.5
May 17 15:55:52 client-18 rshd[26835]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 15:55:52 client-18 xinetd[3390]: EXIT: shell status=0 pid=26834 duration=0(sec)
May 17 15:55:53 client-18 xinetd[3390]: START: shell pid=26871 from=192.168.4.5
May 17 15:55:53 client-18 rshd[26872]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == conf-sanity test 8: double mount setup ============================================================ 15:55:52 \(1305672952\)");echo XXRETCODE:$?'
May 17 15:55:53 client-18 kernel: Lustre: DEBUG MARKER: == conf-sanity test 8: double mount setup ============================================================ 15:55:52 (1305672952)
May 17 15:55:53 client-18 xinetd[3390]: EXIT: shell status=0 pid=26871 duration=0(sec)
May 17 15:56:31 client-18 xinetd[3390]: START: shell pid=26910 from=192.168.4.5
May 17 15:56:31 client-18 rshd[26911]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 15:56:31 client-18 xinetd[3390]: EXIT: shell status=0 pid=26910 duration=0(sec)
May 17 15:56:31 client-18 xinetd[3390]: START: shell pid=26946 from=192.168.4.5
May 17 15:56:31 client-18 rshd[26947]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 15:56:31 client-18 xinetd[3390]: EXIT: shell status=0 pid=26946 duration=0(sec)
May 17 15:56:32 client-18 xinetd[3390]: START: shell pid=26983 from=192.168.4.5
May 17 15:56:32 client-18 rshd[26984]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == conf-sanity test 9: test ptldebug and subsystem for mkfs ========================================== 15:56:31 \(1305672991\)");echo XXRETCODE:$?'
May 17 15:56:32 client-18 kernel: Lustre: DEBUG MARKER: == conf-sanity test 9: test ptldebug and subsystem for mkfs ========================================== 15:56:31 (1305672991)
May 17 15:56:32 client-18 xinetd[3390]: EXIT: shell status=0 pid=26983 duration=0(sec)
May 17 15:56:57 client-18 xinetd[3390]: START: shell pid=27018 from=192.168.4.5
May 17 15:56:57 client-18 rshd[27019]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 15:56:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=27018 duration=0(sec)
May 17 15:56:57 client-18 xinetd[3390]: START: shell pid=27054 from=192.168.4.5
May 17 15:56:57 client-18 rshd[27055]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 15:56:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=27054 duration=0(sec)
May 17 15:56:57 client-18 xinetd[3390]: START: shell pid=27091 from=192.168.4.5
May 17 15:56:57 client-18 rshd[27092]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == conf-sanity test 17: Verify failed mds_postsetup won\'t fail assertion \(2936\) \(should return errs\) ====================================================================================================== 15:56:57 \(1305673017\)");echo XXRETCODE:$?'
May 17 15:56:58 client-18 kernel: Lustre: DEBUG MARKER: == conf-sanity test 17: Verify failed mds_postsetup won't fail assertion (2936) (should return errs) ====================================================================================================== 15:56:57 (1305673017)
May 17 15:56:58 client-18 xinetd[3390]: EXIT: shell status=0 pid=27091 duration=1(sec)
May 17 15:58:03 client-18 xinetd[3390]: START: shell pid=27130 from=192.168.4.5
May 17 15:58:03 client-18 rshd[27131]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:; lsof -t /mnt/lustre || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount  /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre is still busy, wait one second && sleep 1;     umount  /mnt/lustre; fi fi");echo XXRETCODE:$?'
May 17 15:58:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=27130 duration=1(sec)
May 17 15:58:04 client-18 xinetd[3390]: START: shell pid=27167 from=192.168.4.5
May 17 15:58:04 client-18 rshd[27168]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:; lsof -t /mnt/lustre2 || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre2 | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount  /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre2 is still busy, wait one second && sleep 1;     umount  /mnt/lustre2; fi fi");echo XXRETCODE:$?'
May 17 15:58:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=27167 duration=0(sec)
May 17 15:58:23 client-18 xinetd[3390]: START: shell pid=27204 from=192.168.4.5
May 17 15:58:23 client-18 rshd[27205]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark ost now in FULL state");echo XXRETCODE:$?'
May 17 15:58:23 client-18 kernel: Lustre: DEBUG MARKER: ost now in FULL state
May 17 15:58:23 client-18 xinetd[3390]: EXIT: shell status=0 pid=27204 duration=0(sec)
May 17 15:58:34 client-18 xinetd[3390]: START: shell pid=27239 from=192.168.4.5
May 17 15:58:34 client-18 rshd[27240]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 15:58:35 client-18 xinetd[3390]: EXIT: shell status=0 pid=27239 duration=1(sec)
May 17 15:58:35 client-18 xinetd[3390]: START: shell pid=27275 from=192.168.4.5
May 17 15:58:35 client-18 rshd[27276]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 15:58:35 client-18 xinetd[3390]: EXIT: shell status=0 pid=27275 duration=0(sec)
May 17 15:58:35 client-18 xinetd[3390]: START: shell pid=27312 from=192.168.4.5
May 17 15:58:35 client-18 rshd[27313]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == conf-sanity test 18: check mkfs creates large journals ============================================ 15:58:35 \(1305673115\)");echo XXRETCODE:$?'
May 17 15:58:35 client-18 kernel: Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ============================================ 15:58:35 (1305673115)
May 17 15:58:35 client-18 xinetd[3390]: EXIT: shell status=0 pid=27312 duration=0(sec)
May 17 15:58:35 client-18 xinetd[3390]: START: shell pid=27347 from=192.168.4.5
May 17 15:58:35 client-18 rshd[27348]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark use device \/dev\/sdb with MIN=2000000");echo XXRETCODE:$?'
May 17 15:58:35 client-18 kernel: Lustre: DEBUG MARKER: use device /dev/sdb with MIN=2000000
May 17 15:58:35 client-18 xinetd[3390]: EXIT: shell status=0 pid=27347 duration=0(sec)
May 17 15:58:35 client-18 xinetd[3390]: START: shell pid=27382 from=192.168.4.5
May 17 15:58:35 client-18 rshd[27383]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:; lsof -t /mnt/lustre || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount  /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre is still busy, wait one second && sleep 1;     umount  /mnt/lustre; fi fi");echo XXRETCODE:$?'
May 17 15:58:35 client-18 xinetd[3390]: EXIT: shell status=0 pid=27382 duration=0(sec)
May 17 15:58:35 client-18 xinetd[3390]: START: shell pid=27419 from=192.168.4.5
May 17 15:58:35 client-18 rshd[27420]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:; lsof -t /mnt/lustre2 || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre2 | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount  /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre2 is still busy, wait one second && sleep 1;     umount  /mnt/lustre2; fi fi");echo XXRETCODE:$?'
May 17 15:58:35 client-18 xinetd[3390]: EXIT: shell status=0 pid=27419 duration=0(sec)
May 17 15:58:50 client-18 xinetd[3390]: START: shell pid=27460 from=192.168.4.5
May 17 15:58:50 client-18 rshd[27461]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark ost now in FULL state");echo XXRETCODE:$?'
May 17 15:58:50 client-18 kernel: Lustre: DEBUG MARKER: ost now in FULL state
May 17 15:58:50 client-18 xinetd[3390]: EXIT: shell status=0 pid=27460 duration=0(sec)
May 17 15:59:14 client-18 xinetd[3390]: START: shell pid=27495 from=192.168.4.5
May 17 15:59:14 client-18 rshd[27496]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark Success: mkfs creates large journals. Size: 76M");echo XXRETCODE:$?'
May 17 15:59:14 client-18 kernel: Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 76M
May 17 15:59:14 client-18 xinetd[3390]: EXIT: shell status=0 pid=27495 duration=0(sec)
May 17 15:59:35 client-18 xinetd[3390]: START: shell pid=27530 from=192.168.4.5
May 17 15:59:36 client-18 rshd[27531]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:; lsof -t /mnt/lustre || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount  /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre is still busy, wait one second && sleep 1;     umount  /mnt/lustre; fi fi");echo XXRETCODE:$?'
May 17 15:59:36 client-18 xinetd[3390]: EXIT: shell status=0 pid=27530 duration=1(sec)
May 17 15:59:36 client-18 xinetd[3390]: START: shell pid=27567 from=192.168.4.5
May 17 15:59:36 client-18 rshd[27568]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:; lsof -t /mnt/lustre2 || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre2 | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount  /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre2 is still busy, wait one second && sleep 1;     umount  /mnt/lustre2; fi fi");echo XXRETCODE:$?'
May 17 15:59:36 client-18 xinetd[3390]: EXIT: shell status=0 pid=27567 duration=0(sec)
May 17 15:59:49 client-18 xinetd[3390]: START: shell pid=27604 from=192.168.4.5
May 17 15:59:49 client-18 rshd[27605]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark ost now in FULL state");echo XXRETCODE:$?'
May 17 15:59:49 client-18 kernel: Lustre: DEBUG MARKER: ost now in FULL state
May 17 15:59:49 client-18 xinetd[3390]: EXIT: shell status=0 pid=27604 duration=0(sec)
May 17 16:00:01 client-18 xinetd[3390]: START: shell pid=27643 from=192.168.4.5
May 17 16:00:01 client-18 rshd[27644]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 16:00:01 client-18 xinetd[3390]: EXIT: shell status=0 pid=27643 duration=0(sec)
May 17 16:00:01 client-18 xinetd[3390]: START: shell pid=27679 from=192.168.4.5
May 17 16:00:01 client-18 rshd[27680]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 16:00:01 client-18 xinetd[3390]: EXIT: shell status=0 pid=27679 duration=0(sec)
May 17 16:00:01 client-18 xinetd[3390]: START: shell pid=27716 from=192.168.4.5
May 17 16:00:01 client-18 rshd[27717]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == conf-sanity test 19a: start\/stop MDS without OSTs ================================================= 16:00:01 \(1305673201\)");echo XXRETCODE:$?'
May 17 16:00:01 client-18 kernel: Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ================================================= 16:00:01 (1305673201)
May 17 16:00:01 client-18 xinetd[3390]: EXIT: shell status=0 pid=27716 duration=0(sec)
May 17 16:00:09 client-18 xinetd[3390]: START: shell pid=27751 from=192.168.4.5
May 17 16:00:09 client-18 rshd[27752]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 16:00:09 client-18 xinetd[3390]: EXIT: shell status=0 pid=27751 duration=0(sec)
May 17 16:00:09 client-18 xinetd[3390]: START: shell pid=27787 from=192.168.4.5
May 17 16:00:09 client-18 rshd[27788]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 16:00:09 client-18 xinetd[3390]: EXIT: shell status=0 pid=27787 duration=0(sec)
May 17 16:00:09 client-18 xinetd[3390]: START: shell pid=27824 from=192.168.4.5
May 17 16:00:09 client-18 rshd[27825]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == conf-sanity test 19b: start\/stop OSTs without MDS ================================================= 16:00:09 \(1305673209\)");echo XXRETCODE:$?'
May 17 16:00:09 client-18 kernel: Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ================================================= 16:00:09 (1305673209)
May 17 16:00:09 client-18 xinetd[3390]: EXIT: shell status=0 pid=27824 duration=0(sec)
May 17 16:00:34 client-18 xinetd[3390]: START: shell pid=27859 from=192.168.4.5
May 17 16:00:34 client-18 rshd[27860]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 16:00:34 client-18 xinetd[3390]: EXIT: shell status=0 pid=27859 duration=0(sec)
May 17 16:00:34 client-18 xinetd[3390]: START: shell pid=27895 from=192.168.4.5
May 17 16:00:34 client-18 rshd[27896]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 16:00:34 client-18 xinetd[3390]: EXIT: shell status=0 pid=27895 duration=0(sec)
May 17 16:00:34 client-18 xinetd[3390]: START: shell pid=27932 from=192.168.4.5
May 17 16:00:34 client-18 rshd[27933]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == conf-sanity test 20: remount ro,rw mounts work and doesn\'t break \/etc\/mtab ======================== 16:00:34 \(1305673234\)");echo XXRETCODE:$?'
May 17 16:00:34 client-18 kernel: Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ======================== 16:00:34 (1305673234)
May 17 16:00:34 client-18 xinetd[3390]: EXIT: shell status=0 pid=27932 duration=0(sec)
May 17 16:01:07 client-18 xinetd[3390]: START: shell pid=27973 from=192.168.4.5
May 17 16:01:07 client-18 rshd[27974]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 16:01:07 client-18 xinetd[3390]: EXIT: shell status=0 pid=27973 duration=0(sec)
May 17 16:01:07 client-18 xinetd[3390]: START: shell pid=28009 from=192.168.4.5
May 17 16:01:07 client-18 rshd[28010]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 16:01:07 client-18 xinetd[3390]: EXIT: shell status=0 pid=28009 duration=0(sec)
May 17 16:01:07 client-18 xinetd[3390]: START: shell pid=28046 from=192.168.4.5
May 17 16:01:07 client-18 rshd[28047]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == conf-sanity test 21a: start mds before ost, stop ost first ======================================== 16:01:07 \(1305673267\)");echo XXRETCODE:$?'
May 17 16:01:07 client-18 kernel: Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ======================================== 16:01:07 (1305673267)
May 17 16:01:07 client-18 xinetd[3390]: EXIT: shell status=0 pid=28046 duration=0(sec)
May 17 16:01:18 client-18 xinetd[3390]: START: shell pid=28085 from=192.168.4.5
May 17 16:01:18 client-18 rshd[28086]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark ost now in FULL state");echo XXRETCODE:$?'
May 17 16:01:18 client-18 kernel: Lustre: DEBUG MARKER: ost now in FULL state
May 17 16:01:18 client-18 xinetd[3390]: EXIT: shell status=0 pid=28085 duration=0(sec)
May 17 16:01:33 client-18 xinetd[3390]: START: shell pid=28120 from=192.168.4.5
May 17 16:01:33 client-18 rshd[28121]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 16:01:33 client-18 xinetd[3390]: EXIT: shell status=0 pid=28120 duration=0(sec)
May 17 16:01:33 client-18 xinetd[3390]: START: shell pid=28156 from=192.168.4.5
May 17 16:01:33 client-18 rshd[28157]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 16:01:33 client-18 xinetd[3390]: EXIT: shell status=0 pid=28156 duration=0(sec)
May 17 16:01:33 client-18 xinetd[3390]: START: shell pid=28193 from=192.168.4.5
May 17 16:01:33 client-18 rshd[28194]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == conf-sanity test 21b: start ost before mds, stop mds first ======================================== 16:01:33 \(1305673293\)");echo XXRETCODE:$?'
May 17 16:01:33 client-18 kernel: Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ======================================== 16:01:33 (1305673293)
May 17 16:01:33 client-18 xinetd[3390]: EXIT: shell status=0 pid=28193 duration=0(sec)
May 17 16:01:57 client-18 xinetd[3390]: START: shell pid=28228 from=192.168.4.5
May 17 16:01:57 client-18 rshd[28229]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark ost now in FULL state");echo XXRETCODE:$?'
May 17 16:01:57 client-18 kernel: Lustre: DEBUG MARKER: ost now in FULL state
May 17 16:01:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=28228 duration=0(sec)
May 17 16:02:14 client-18 xinetd[3390]: START: shell pid=28263 from=192.168.4.5
May 17 16:02:14 client-18 rshd[28264]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 16:02:14 client-18 xinetd[3390]: EXIT: shell status=0 pid=28263 duration=0(sec)
May 17 16:02:14 client-18 xinetd[3390]: START: shell pid=28299 from=192.168.4.5
May 17 16:02:14 client-18 rshd[28300]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 16:02:14 client-18 xinetd[3390]: EXIT: shell status=0 pid=28299 duration=0(sec)
May 17 16:02:14 client-18 xinetd[3390]: START: shell pid=28336 from=192.168.4.5
May 17 16:02:14 client-18 rshd[28337]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == conf-sanity test 21c: start mds between two osts, stop mds last =================================== 16:02:14 \(1305673334\)");echo XXRETCODE:$?'
May 17 16:02:14 client-18 kernel: Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last =================================== 16:02:14 (1305673334)
May 17 16:02:14 client-18 xinetd[3390]: EXIT: shell status=0 pid=28336 duration=0(sec)
May 17 16:02:42 client-18 xinetd[3390]: START: shell pid=28375 from=192.168.4.5
May 17 16:02:42 client-18 rshd[28376]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark ost2 now in FULL state");echo XXRETCODE:$?'
May 17 16:02:42 client-18 kernel: Lustre: DEBUG MARKER: ost2 now in FULL state
May 17 16:02:42 client-18 xinetd[3390]: EXIT: shell status=0 pid=28375 duration=0(sec)
May 17 16:02:57 client-18 xinetd[3390]: START: shell pid=28410 from=192.168.4.5
May 17 16:02:57 client-18 rshd[28411]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 16:02:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=28410 duration=0(sec)
May 17 16:02:57 client-18 xinetd[3390]: START: shell pid=28446 from=192.168.4.5
May 17 16:02:57 client-18 rshd[28447]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 16:02:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=28446 duration=0(sec)
May 17 16:02:57 client-18 xinetd[3390]: START: shell pid=28483 from=192.168.4.5
May 17 16:02:57 client-18 rshd[28484]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == conf-sanity test 21d: start mgs then ost and then mds ============================================= 16:02:57 \(1305673377\)");echo XXRETCODE:$?'
May 17 16:02:57 client-18 kernel: Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ============================================= 16:02:57 (1305673377)
May 17 16:02:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=28483 duration=0(sec)
May 17 16:02:57 client-18 xinetd[3390]: START: shell pid=28518 from=192.168.4.5
May 17 16:02:57 client-18 rshd[28519]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark  SKIP: conf-sanity test_21d need separate mgs device");echo XXRETCODE:$?'
May 17 16:02:58 client-18 kernel: Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device
May 17 16:02:58 client-18 xinetd[3390]: EXIT: shell status=0 pid=28518 duration=1(sec)
May 17 16:02:58 client-18 xinetd[3390]: START: shell pid=28553 from=192.168.4.5
May 17 16:02:58 client-18 rshd[28554]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 16:02:58 client-18 xinetd[3390]: EXIT: shell status=0 pid=28553 duration=0(sec)
May 17 16:02:58 client-18 xinetd[3390]: START: shell pid=28589 from=192.168.4.5
May 17 16:02:58 client-18 rshd[28590]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 16:02:58 client-18 xinetd[3390]: EXIT: shell status=0 pid=28589 duration=0(sec)
May 17 16:02:58 client-18 xinetd[3390]: START: shell pid=28626 from=192.168.4.5
May 17 16:02:58 client-18 rshd[28627]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == conf-sanity test 22: start a client before osts \(should return errs\) ============================== 16:02:58 \(1305673378\)");echo XXRETCODE:$?'
May 17 16:02:58 client-18 kernel: Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ============================== 16:02:58 (1305673378)
May 17 16:02:58 client-18 xinetd[3390]: EXIT: shell status=0 pid=28626 duration=0(sec)
May 17 16:03:05 client-18 xinetd[3390]: START: shell pid=28661 from=192.168.4.5
May 17 16:03:05 client-18 rshd[28662]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark ost now in FULL state");echo XXRETCODE:$?'
May 17 16:03:05 client-18 kernel: Lustre: DEBUG MARKER: ost now in FULL state
May 17 16:03:05 client-18 xinetd[3390]: EXIT: shell status=0 pid=28661 duration=0(sec)
May 17 16:03:37 client-18 xinetd[3390]: START: shell pid=28696 from=192.168.4.5
May 17 16:03:37 client-18 rshd[28697]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 16:03:37 client-18 xinetd[3390]: EXIT: shell status=0 pid=28696 duration=0(sec)
May 17 16:03:37 client-18 xinetd[3390]: START: shell pid=28732 from=192.168.4.5
May 17 16:03:37 client-18 rshd[28733]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 16:03:37 client-18 xinetd[3390]: EXIT: shell status=0 pid=28732 duration=0(sec)
May 17 16:03:38 client-18 xinetd[3390]: START: shell pid=28769 from=192.168.4.5
May 17 16:03:38 client-18 rshd[28770]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == conf-sanity test 23a: interrupt client during recovery mount delay ================================ 16:03:38 \(1305673418\)");echo XXRETCODE:$?'
May 17 16:03:38 client-18 kernel: Lustre: DEBUG MARKER: == conf-sanity test 23a: interrupt client during recovery mount delay ================================ 16:03:38 (1305673418)
May 17 16:03:38 client-18 xinetd[3390]: EXIT: shell status=0 pid=28769 duration=0(sec)
May 17 16:04:22 client-18 xinetd[3390]: START: shell pid=28808 from=192.168.4.5
May 17 16:04:23 client-18 rshd[28809]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 16:04:23 client-18 xinetd[3390]: EXIT: shell status=0 pid=28808 duration=1(sec)
May 17 16:04:23 client-18 xinetd[3390]: START: shell pid=28844 from=192.168.4.5
May 17 16:04:23 client-18 rshd[28845]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 16:04:23 client-18 xinetd[3390]: EXIT: shell status=0 pid=28844 duration=0(sec)
May 17 16:04:34 client-18 xinetd[3390]: START: shell pid=28881 from=192.168.4.5
May 17 16:04:34 client-18 rshd[28882]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == conf-sanity test 23b: Simulate -EINTR during mount ================================================ 16:04:34 \(1305673474\)");echo XXRETCODE:$?'
May 17 16:04:34 client-18 kernel: Lustre: DEBUG MARKER: == conf-sanity test 23b: Simulate -EINTR during mount ================================================ 16:04:34 (1305673474)
May 17 16:04:34 client-18 xinetd[3390]: EXIT: shell status=0 pid=28881 duration=0(sec)
May 17 16:05:08 client-18 xinetd[3390]: START: shell pid=28920 from=192.168.4.5
May 17 16:05:08 client-18 rshd[28921]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 16:05:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=28920 duration=0(sec)
May 17 16:05:08 client-18 xinetd[3390]: START: shell pid=28956 from=192.168.4.5
May 17 16:05:08 client-18 rshd[28957]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 16:05:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=28956 duration=0(sec)
May 17 16:05:09 client-18 xinetd[3390]: START: shell pid=28993 from=192.168.4.5
May 17 16:05:09 client-18 rshd[28994]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == conf-sanity test 24a: Multiple MDTs on a single node ============================================== 16:05:08 \(1305673508\)");echo XXRETCODE:$?'
May 17 16:05:09 client-18 kernel: Lustre: DEBUG MARKER: == conf-sanity test 24a: Multiple MDTs on a single node ============================================== 16:05:08 (1305673508)
May 17 16:05:09 client-18 xinetd[3390]: EXIT: shell status=0 pid=28993 duration=0(sec)
May 17 16:05:09 client-18 xinetd[3390]: START: shell pid=29028 from=192.168.4.5
May 17 16:05:09 client-18 rshd[29029]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark  SKIP: conf-sanity test_24a mixed loopback and real device not working");echo XXRETCODE:$?'
May 17 16:05:09 client-18 kernel: Lustre: DEBUG MARKER: SKIP: conf-sanity test_24a mixed loopback and real device not working
May 17 16:05:09 client-18 xinetd[3390]: EXIT: shell status=0 pid=29028 duration=0(sec)
May 17 16:05:09 client-18 xinetd[3390]: START: shell pid=29063 from=192.168.4.5
May 17 16:05:09 client-18 rshd[29064]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 16:05:09 client-18 xinetd[3390]: EXIT: shell status=0 pid=29063 duration=0(sec)
May 17 16:05:09 client-18 xinetd[3390]: START: shell pid=29099 from=192.168.4.5
May 17 16:05:09 client-18 rshd[29100]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 16:05:10 client-18 xinetd[3390]: EXIT: shell status=0 pid=29099 duration=1(sec)
May 17 16:05:10 client-18 xinetd[3390]: START: shell pid=29136 from=192.168.4.5
May 17 16:05:10 client-18 rshd[29137]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == conf-sanity test 24b: Multiple MGSs on a single node \(should return err\) ========================== 16:05:10 \(1305673510\)");echo XXRETCODE:$?'
May 17 16:05:10 client-18 kernel: Lustre: DEBUG MARKER: == conf-sanity test 24b: Multiple MGSs on a single node (should return err) ========================== 16:05:10 (1305673510)
May 17 16:05:10 client-18 xinetd[3390]: EXIT: shell status=0 pid=29136 duration=0(sec)
May 17 16:05:10 client-18 xinetd[3390]: START: shell pid=29171 from=192.168.4.5
May 17 16:05:10 client-18 rshd[29172]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark  SKIP: conf-sanity test_24b mixed loopback and real device not working");echo XXRETCODE:$?'
May 17 16:05:10 client-18 kernel: Lustre: DEBUG MARKER: SKIP: conf-sanity test_24b mixed loopback and real device not working
May 17 16:05:10 client-18 xinetd[3390]: EXIT: shell status=0 pid=29171 duration=0(sec)
May 17 16:05:10 client-18 xinetd[3390]: START: shell pid=29206 from=192.168.4.5
May 17 16:05:10 client-18 rshd[29207]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 16:05:10 client-18 xinetd[3390]: EXIT: shell status=0 pid=29206 duration=0(sec)
May 17 16:05:10 client-18 xinetd[3390]: START: shell pid=29242 from=192.168.4.5
May 17 16:05:10 client-18 rshd[29243]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 16:05:10 client-18 xinetd[3390]: EXIT: shell status=0 pid=29242 duration=0(sec)
May 17 16:05:10 client-18 xinetd[3390]: START: shell pid=29279 from=192.168.4.5
May 17 16:05:10 client-18 rshd[29280]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == conf-sanity test 25: Verify modules are referenced ================================================ 16:05:10 \(1305673510\)");echo XXRETCODE:$?'
May 17 16:05:10 client-18 kernel: Lustre: DEBUG MARKER: == conf-sanity test 25: Verify modules are referenced ================================================ 16:05:10 (1305673510)
May 17 16:05:10 client-18 xinetd[3390]: EXIT: shell status=0 pid=29279 duration=0(sec)
May 17 16:05:48 client-18 xinetd[3390]: START: shell pid=29314 from=192.168.4.5
May 17 16:05:48 client-18 rshd[29315]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 16:05:48 client-18 xinetd[3390]: EXIT: shell status=0 pid=29314 duration=0(sec)
May 17 16:05:48 client-18 xinetd[3390]: START: shell pid=29350 from=192.168.4.5
May 17 16:05:48 client-18 rshd[29351]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 16:05:48 client-18 xinetd[3390]: EXIT: shell status=0 pid=29350 duration=0(sec)
May 17 16:05:49 client-18 xinetd[3390]: START: shell pid=29387 from=192.168.4.5
May 17 16:05:49 client-18 rshd[29388]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == conf-sanity test 26: MDT startup failure cleans LOV \(should return errs\) ========================== 16:05:48 \(1305673548\)");echo XXRETCODE:$?'
May 17 16:05:49 client-18 kernel: Lustre: DEBUG MARKER: == conf-sanity test 26: MDT startup failure cleans LOV (should return errs) ========================== 16:05:48 (1305673548)
May 17 16:05:49 client-18 xinetd[3390]: EXIT: shell status=0 pid=29387 duration=0(sec)
May 17 16:06:07 client-18 xinetd[3390]: START: shell pid=29422 from=192.168.4.5
May 17 16:06:07 client-18 rshd[29423]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 16:06:07 client-18 xinetd[3390]: EXIT: shell status=0 pid=29422 duration=0(sec)
May 17 16:06:07 client-18 xinetd[3390]: START: shell pid=29458 from=192.168.4.5
May 17 16:06:07 client-18 rshd[29459]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 16:06:07 client-18 xinetd[3390]: EXIT: shell status=0 pid=29458 duration=0(sec)
May 17 16:06:08 client-18 xinetd[3390]: START: shell pid=29495 from=192.168.4.5
May 17 16:06:08 client-18 rshd[29496]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == conf-sanity test 27a: Reacquire MGS lock if OST started first ===================================== 16:06:07 \(1305673567\)");echo XXRETCODE:$?'
May 17 16:06:08 client-18 kernel: Lustre: DEBUG MARKER: == conf-sanity test 27a: Reacquire MGS lock if OST started first ===================================== 16:06:07 (1305673567)
May 17 16:06:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=29495 duration=0(sec)
May 17 16:07:56 client-18 xinetd[3390]: START: shell pid=29538 from=192.168.4.5
May 17 16:07:56 client-18 rshd[29539]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 16:07:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=29538 duration=1(sec)
May 17 16:07:57 client-18 xinetd[3390]: START: shell pid=29574 from=192.168.4.5
May 17 16:07:57 client-18 rshd[29575]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 16:07:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=29574 duration=0(sec)
May 17 16:07:57 client-18 xinetd[3390]: START: shell pid=29611 from=192.168.4.5
May 17 16:07:57 client-18 rshd[29612]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == conf-sanity test 27b: Reacquire MGS lock after failover =========================================== 16:07:57 \(1305673677\)");echo XXRETCODE:$?'
May 17 16:07:57 client-18 kernel: Lustre: DEBUG MARKER: == conf-sanity test 27b: Reacquire MGS lock after failover =========================================== 16:07:57 (1305673677)
May 17 16:07:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=29611 duration=0(sec)
May 17 16:09:03 client-18 xinetd[3390]: START: shell pid=29650 from=192.168.4.5
May 17 16:09:03 client-18 rshd[29651]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 16:09:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=29650 duration=0(sec)
May 17 16:09:03 client-18 xinetd[3390]: START: shell pid=29686 from=192.168.4.5
May 17 16:09:04 client-18 rshd[29687]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 16:09:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=29686 duration=1(sec)
May 17 16:09:04 client-18 xinetd[3390]: START: shell pid=29723 from=192.168.4.5
May 17 16:09:04 client-18 rshd[29724]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == conf-sanity test 28: permanent parameter setting ================================================== 16:09:04 \(1305673744\)");echo XXRETCODE:$?'
May 17 16:09:04 client-18 kernel: Lustre: DEBUG MARKER: == conf-sanity test 28: permanent parameter setting ================================================== 16:09:04 (1305673744)
May 17 16:09:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=29723 duration=0(sec)
May 17 16:09:52 client-18 xinetd[3390]: START: shell pid=29758 from=192.168.4.5
May 17 16:09:52 client-18 rshd[29759]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 16:09:52 client-18 xinetd[3390]: EXIT: shell status=0 pid=29758 duration=0(sec)
May 17 16:09:52 client-18 xinetd[3390]: START: shell pid=29794 from=192.168.4.5
May 17 16:09:52 client-18 rshd[29795]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 16:09:52 client-18 xinetd[3390]: EXIT: shell status=0 pid=29794 duration=0(sec)
May 17 16:09:52 client-18 xinetd[3390]: START: shell pid=29831 from=192.168.4.5
May 17 16:09:53 client-18 rshd[29832]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == conf-sanity test 29: permanently remove an OST ==================================================== 16:09:52 \(1305673792\)");echo XXRETCODE:$?'
May 17 16:09:53 client-18 kernel: Lustre: DEBUG MARKER: == conf-sanity test 29: permanently remove an OST ==================================================== 16:09:52 (1305673792)
May 17 16:09:53 client-18 xinetd[3390]: EXIT: shell status=0 pid=29831 duration=1(sec)
May 17 16:10:54 client-18 xinetd[3390]: START: shell pid=29870 from=192.168.4.5
May 17 16:10:54 client-18 rshd[29871]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 16:10:54 client-18 xinetd[3390]: EXIT: shell status=0 pid=29870 duration=0(sec)
May 17 16:10:54 client-18 xinetd[3390]: START: shell pid=29906 from=192.168.4.5
May 17 16:10:54 client-18 rshd[29907]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 16:10:54 client-18 xinetd[3390]: EXIT: shell status=0 pid=29906 duration=0(sec)
May 17 16:10:54 client-18 xinetd[3390]: START: shell pid=29943 from=192.168.4.5
May 17 16:10:54 client-18 rshd[29944]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == conf-sanity test 30a: Big config llog and conf_param deletion ===================================== 16:10:54 \(1305673854\)");echo XXRETCODE:$?'
May 17 16:10:54 client-18 kernel: Lustre: DEBUG MARKER: == conf-sanity test 30a: Big config llog and conf_param deletion ===================================== 16:10:54 (1305673854)
May 17 16:10:54 client-18 xinetd[3390]: EXIT: shell status=0 pid=29943 duration=0(sec)
May 17 16:13:11 client-18 xinetd[3390]: START: shell pid=29989 from=192.168.4.5
May 17 16:13:11 client-18 rshd[29990]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 16:13:12 client-18 xinetd[3390]: EXIT: shell status=0 pid=29989 duration=1(sec)
May 17 16:13:12 client-18 xinetd[3390]: START: shell pid=30025 from=192.168.4.5
May 17 16:13:12 client-18 rshd[30026]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 16:13:12 client-18 xinetd[3390]: EXIT: shell status=0 pid=30025 duration=0(sec)
May 17 16:13:12 client-18 xinetd[3390]: START: shell pid=30062 from=192.168.4.5
May 17 16:13:12 client-18 rshd[30063]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == conf-sanity test 30b: Remove failover nids ======================================================== 16:13:12 \(1305673992\)");echo XXRETCODE:$?'
May 17 16:13:12 client-18 kernel: Lustre: DEBUG MARKER: == conf-sanity test 30b: Remove failover nids ======================================================== 16:13:12 (1305673992)
May 17 16:13:12 client-18 xinetd[3390]: EXIT: shell status=0 pid=30062 duration=0(sec)
May 17 16:13:54 client-18 xinetd[3390]: START: shell pid=30101 from=192.168.4.5
May 17 16:13:54 client-18 rshd[30102]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 16:13:54 client-18 xinetd[3390]: EXIT: shell status=0 pid=30101 duration=0(sec)
May 17 16:13:54 client-18 xinetd[3390]: START: shell pid=30137 from=192.168.4.5
May 17 16:13:54 client-18 rshd[30138]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 16:13:54 client-18 xinetd[3390]: EXIT: shell status=0 pid=30137 duration=0(sec)
May 17 16:13:55 client-18 xinetd[3390]: START: shell pid=30174 from=192.168.4.5
May 17 16:13:55 client-18 rshd[30175]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == conf-sanity test 31: Connect to non-existent node \(shouldn\'t crash\) =============================== 16:13:54 \(1305674034\)");echo XXRETCODE:$?'
May 17 16:13:55 client-18 kernel: Lustre: DEBUG MARKER: == conf-sanity test 31: Connect to non-existent node (shouldn't crash) =============================== 16:13:54 (1305674034)
May 17 16:13:55 client-18 xinetd[3390]: EXIT: shell status=0 pid=30174 duration=0(sec)
May 17 16:17:06 client-18 xinetd[3390]: START: shell pid=30217 from=192.168.4.5
May 17 16:17:06 client-18 rshd[30218]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 16:17:06 client-18 xinetd[3390]: EXIT: shell status=0 pid=30217 duration=0(sec)
May 17 16:17:06 client-18 xinetd[3390]: START: shell pid=30253 from=192.168.4.5
May 17 16:17:06 client-18 rshd[30254]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 16:17:06 client-18 xinetd[3390]: EXIT: shell status=0 pid=30253 duration=0(sec)
May 17 16:17:07 client-18 xinetd[3390]: START: shell pid=30290 from=192.168.4.5
May 17 16:17:07 client-18 rshd[30291]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == conf-sanity test 32a: Upgrade from 1.8 \(not live\) ================================================= 16:17:06 \(1305674226\)");echo XXRETCODE:$?'
May 17 16:17:07 client-18 kernel: Lustre: DEBUG MARKER: == conf-sanity test 32a: Upgrade from 1.8 (not live) ================================================= 16:17:06 (1305674226)
May 17 16:17:07 client-18 xinetd[3390]: EXIT: shell status=0 pid=30290 duration=0(sec)
May 17 16:17:07 client-18 xinetd[3390]: START: shell pid=30325 from=192.168.4.5
May 17 16:17:07 client-18 rshd[30326]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark  SKIP: conf-sanity test_32a client only testing");echo XXRETCODE:$?'
May 17 16:17:07 client-18 kernel: Lustre: DEBUG MARKER: SKIP: conf-sanity test_32a client only testing
May 17 16:17:07 client-18 xinetd[3390]: EXIT: shell status=0 pid=30325 duration=0(sec)
May 17 16:17:07 client-18 xinetd[3390]: START: shell pid=30360 from=192.168.4.5
May 17 16:17:07 client-18 rshd[30361]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 16:17:07 client-18 xinetd[3390]: EXIT: shell status=0 pid=30360 duration=0(sec)
May 17 16:17:07 client-18 xinetd[3390]: START: shell pid=30396 from=192.168.4.5
May 17 16:17:07 client-18 rshd[30397]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 16:17:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=30396 duration=1(sec)
May 17 16:17:08 client-18 xinetd[3390]: START: shell pid=30433 from=192.168.4.5
May 17 16:17:08 client-18 rshd[30434]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == conf-sanity test 32b: Upgrade from 1.8 with writeconf ============================================= 16:17:08 \(1305674228\)");echo XXRETCODE:$?'
May 17 16:17:08 client-18 kernel: Lustre: DEBUG MARKER: == conf-sanity test 32b: Upgrade from 1.8 with writeconf ============================================= 16:17:08 (1305674228)
May 17 16:17:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=30433 duration=0(sec)
May 17 16:17:08 client-18 xinetd[3390]: START: shell pid=30468 from=192.168.4.5
May 17 16:17:08 client-18 rshd[30469]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark  SKIP: conf-sanity test_32b client only testing");echo XXRETCODE:$?'
May 17 16:17:08 client-18 kernel: Lustre: DEBUG MARKER: SKIP: conf-sanity test_32b client only testing
May 17 16:17:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=30468 duration=0(sec)
May 17 16:17:08 client-18 xinetd[3390]: START: shell pid=30503 from=192.168.4.5
May 17 16:17:08 client-18 rshd[30504]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 16:17:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=30503 duration=0(sec)
May 17 16:17:08 client-18 xinetd[3390]: START: shell pid=30539 from=192.168.4.5
May 17 16:17:08 client-18 rshd[30540]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 16:17:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=30539 duration=0(sec)
May 17 16:17:08 client-18 xinetd[3390]: START: shell pid=30576 from=192.168.4.5
May 17 16:17:08 client-18 rshd[30577]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == conf-sanity test 33a: Mount ost with a large index number ========================================= 16:17:08 \(1305674228\)");echo XXRETCODE:$?'
May 17 16:17:08 client-18 kernel: Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================= 16:17:08 (1305674228)
May 17 16:17:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=30576 duration=0(sec)
May 17 16:17:09 client-18 xinetd[3390]: START: shell pid=30611 from=192.168.4.5
May 17 16:17:09 client-18 rshd[30612]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark  SKIP: conf-sanity test_33a mixed loopback and real device not working");echo XXRETCODE:$?'
May 17 16:17:09 client-18 kernel: Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working
May 17 16:17:09 client-18 xinetd[3390]: EXIT: shell status=0 pid=30611 duration=0(sec)
May 17 16:17:09 client-18 xinetd[3390]: START: shell pid=30646 from=192.168.4.5
May 17 16:17:09 client-18 rshd[30647]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 16:17:09 client-18 xinetd[3390]: EXIT: shell status=0 pid=30646 duration=0(sec)
May 17 16:17:09 client-18 xinetd[3390]: START: shell pid=30682 from=192.168.4.5
May 17 16:17:09 client-18 rshd[30683]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 16:17:09 client-18 xinetd[3390]: EXIT: shell status=0 pid=30682 duration=0(sec)
May 17 16:17:09 client-18 xinetd[3390]: START: shell pid=30719 from=192.168.4.5
May 17 16:17:09 client-18 rshd[30720]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == conf-sanity test 33b: Drop cancel during umount =================================================== 16:17:09 \(1305674229\)");echo XXRETCODE:$?'
May 17 16:17:09 client-18 kernel: Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount =================================================== 16:17:09 (1305674229)
May 17 16:17:09 client-18 xinetd[3390]: EXIT: shell status=0 pid=30719 duration=0(sec)
May 17 16:17:46 client-18 xinetd[3390]: START: shell pid=30758 from=192.168.4.5
May 17 16:17:46 client-18 rshd[30759]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 16:17:46 client-18 xinetd[3390]: EXIT: shell status=0 pid=30758 duration=0(sec)
May 17 16:17:46 client-18 xinetd[3390]: START: shell pid=30794 from=192.168.4.5
May 17 16:17:46 client-18 rshd[30795]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 16:17:46 client-18 xinetd[3390]: EXIT: shell status=0 pid=30794 duration=0(sec)
May 17 16:17:47 client-18 xinetd[3390]: START: shell pid=30831 from=192.168.4.5
May 17 16:17:47 client-18 rshd[30832]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == conf-sanity test 34a: umount with opened file should be fail ====================================== 16:17:46 \(1305674266\)");echo XXRETCODE:$?'
May 17 16:17:47 client-18 kernel: Lustre: DEBUG MARKER: == conf-sanity test 34a: umount with opened file should be fail ====================================== 16:17:46 (1305674266)
May 17 16:17:47 client-18 xinetd[3390]: EXIT: shell status=0 pid=30831 duration=0(sec)
May 17 16:18:23 client-18 xinetd[3390]: START: shell pid=30866 from=192.168.4.5
May 17 16:18:23 client-18 rshd[30867]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 16:18:23 client-18 xinetd[3390]: EXIT: shell status=0 pid=30866 duration=0(sec)
May 17 16:18:23 client-18 xinetd[3390]: START: shell pid=30902 from=192.168.4.5
May 17 16:18:23 client-18 rshd[30903]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 16:18:23 client-18 xinetd[3390]: EXIT: shell status=0 pid=30902 duration=0(sec)
May 17 16:18:23 client-18 xinetd[3390]: START: shell pid=30939 from=192.168.4.5
May 17 16:18:24 client-18 rshd[30940]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == conf-sanity test 34b: force umount with failed mds should be normal =============================== 16:18:23 \(1305674303\)");echo XXRETCODE:$?'
May 17 16:18:24 client-18 kernel: Lustre: DEBUG MARKER: == conf-sanity test 34b: force umount with failed mds should be normal =============================== 16:18:23 (1305674303)
May 17 16:18:24 client-18 xinetd[3390]: EXIT: shell status=0 pid=30939 duration=1(sec)
May 17 16:19:09 client-18 xinetd[3390]: START: shell pid=30978 from=192.168.4.5
May 17 16:19:09 client-18 rshd[30979]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 16:19:10 client-18 xinetd[3390]: EXIT: shell status=0 pid=30978 duration=1(sec)
May 17 16:19:10 client-18 xinetd[3390]: START: shell pid=31014 from=192.168.4.5
May 17 16:19:10 client-18 rshd[31015]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 16:19:10 client-18 xinetd[3390]: EXIT: shell status=0 pid=31014 duration=0(sec)
May 17 16:19:10 client-18 xinetd[3390]: START: shell pid=31051 from=192.168.4.5
May 17 16:19:10 client-18 rshd[31052]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == conf-sanity test 34c: force umount with failed ost should be normal =============================== 16:19:10 \(1305674350\)");echo XXRETCODE:$?'
May 17 16:19:10 client-18 kernel: Lustre: DEBUG MARKER: == conf-sanity test 34c: force umount with failed ost should be normal =============================== 16:19:10 (1305674350)
May 17 16:19:10 client-18 xinetd[3390]: EXIT: shell status=0 pid=31051 duration=0(sec)
May 17 16:19:48 client-18 xinetd[3390]: START: shell pid=31086 from=192.168.4.5
May 17 16:19:48 client-18 rshd[31087]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 16:19:48 client-18 xinetd[3390]: EXIT: shell status=0 pid=31086 duration=0(sec)
May 17 16:19:48 client-18 xinetd[3390]: START: shell pid=31122 from=192.168.4.5
May 17 16:19:48 client-18 rshd[31123]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 16:19:48 client-18 xinetd[3390]: EXIT: shell status=0 pid=31122 duration=0(sec)
May 17 16:19:49 client-18 xinetd[3390]: START: shell pid=31159 from=192.168.4.5
May 17 16:19:49 client-18 rshd[31160]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == conf-sanity test 35a: Reconnect to the last active server first =================================== 16:19:48 \(1305674388\)");echo XXRETCODE:$?'
May 17 16:19:49 client-18 kernel: Lustre: DEBUG MARKER: == conf-sanity test 35a: Reconnect to the last active server first =================================== 16:19:48 (1305674388)
May 17 16:19:49 client-18 xinetd[3390]: EXIT: shell status=0 pid=31159 duration=0(sec)
May 17 16:20:01 client-18 xinetd[3390]: START: shell pid=31198 from=192.168.4.5
May 17 16:20:01 client-18 rshd[31199]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark Set up a fake failnode for the MDS");echo XXRETCODE:$?'
May 17 16:20:01 client-18 kernel: Lustre: DEBUG MARKER: Set up a fake failnode for the MDS
May 17 16:20:01 client-18 xinetd[3390]: EXIT: shell status=0 pid=31198 duration=0(sec)
May 17 16:20:01 client-18 xinetd[3390]: START: shell pid=31233 from=192.168.4.5
May 17 16:20:01 client-18 rshd[31234]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark Wait for RECONNECT_INTERVAL seconds \(10s\)");echo XXRETCODE:$?'
May 17 16:20:01 client-18 kernel: Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s)
May 17 16:20:01 client-18 xinetd[3390]: EXIT: shell status=0 pid=31233 duration=0(sec)
May 17 16:20:11 client-18 xinetd[3390]: START: shell pid=31268 from=192.168.4.5
May 17 16:20:11 client-18 rshd[31269]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark conf-sanity.sh test_35a 2011-05-1716h20m11s");echo XXRETCODE:$?'
May 17 16:20:11 client-18 kernel: Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2011-05-1716h20m11s
May 17 16:20:11 client-18 xinetd[3390]: EXIT: shell status=0 pid=31268 duration=0(sec)
May 17 16:20:11 client-18 xinetd[3390]: START: shell pid=31303 from=192.168.4.5
May 17 16:20:11 client-18 rshd[31304]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark Stopping the MDT:");echo XXRETCODE:$?'
May 17 16:20:11 client-18 kernel: Lustre: DEBUG MARKER: Stopping the MDT:
May 17 16:20:12 client-18 xinetd[3390]: EXIT: shell status=0 pid=31303 duration=1(sec)
May 17 16:20:18 client-18 xinetd[3390]: START: shell pid=31338 from=192.168.4.5
May 17 16:20:18 client-18 rshd[31339]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark Restarting the MDT:");echo XXRETCODE:$?'
May 17 16:20:18 client-18 kernel: Lustre: DEBUG MARKER: Restarting the MDT:
May 17 16:20:18 client-18 xinetd[3390]: EXIT: shell status=0 pid=31338 duration=0(sec)
May 17 16:20:19 client-18 xinetd[3390]: START: shell pid=31373 from=192.168.4.5
May 17 16:20:19 client-18 rshd[31374]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark Wait for df \(10695\) ... ");echo XXRETCODE:$?'
May 17 16:20:19 client-18 kernel: Lustre: DEBUG MARKER: Wait for df (10695) ...
May 17 16:20:19 client-18 xinetd[3390]: EXIT: shell status=0 pid=31373 duration=0(sec)
May 17 16:20:23 client-18 xinetd[3390]: START: shell pid=31408 from=192.168.4.5
May 17 16:20:23 client-18 rshd[31409]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark done");echo XXRETCODE:$?'
May 17 16:20:23 client-18 kernel: Lustre: DEBUG MARKER: done
May 17 16:20:23 client-18 xinetd[3390]: EXIT: shell status=0 pid=31408 duration=0(sec)
May 17 16:20:45 client-18 xinetd[3390]: START: shell pid=31443 from=192.168.4.5
May 17 16:20:45 client-18 rshd[31444]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 16:20:45 client-18 xinetd[3390]: EXIT: shell status=0 pid=31443 duration=0(sec)
May 17 16:20:45 client-18 xinetd[3390]: START: shell pid=31479 from=192.168.4.5
May 17 16:20:45 client-18 rshd[31480]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 16:20:45 client-18 xinetd[3390]: EXIT: shell status=0 pid=31479 duration=0(sec)
May 17 16:20:45 client-18 xinetd[3390]: START: shell pid=31516 from=192.168.4.5
May 17 16:20:45 client-18 rshd[31517]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ================= 16:20:45 \(1305674445\)");echo XXRETCODE:$?'
May 17 16:20:45 client-18 kernel: Lustre: DEBUG MARKER: == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ================= 16:20:45 (1305674445)
May 17 16:20:45 client-18 xinetd[3390]: EXIT: shell status=0 pid=31516 duration=0(sec)
May 17 16:20:57 client-18 xinetd[3390]: START: shell pid=31551 from=192.168.4.5
May 17 16:20:57 client-18 rshd[31552]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark conf-sanity.sh test_35b 2011-05-1716h20m56s");echo XXRETCODE:$?'
May 17 16:20:57 client-18 kernel: Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2011-05-1716h20m56s
May 17 16:20:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=31551 duration=0(sec)
May 17 16:20:57 client-18 xinetd[3390]: START: shell pid=31586 from=192.168.4.5
May 17 16:20:57 client-18 rshd[31587]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark Set up a fake failnode for the MDS");echo XXRETCODE:$?'
May 17 16:20:57 client-18 kernel: Lustre: DEBUG MARKER: Set up a fake failnode for the MDS
May 17 16:20:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=31586 duration=0(sec)
May 17 16:20:57 client-18 xinetd[3390]: START: shell pid=31621 from=192.168.4.5
May 17 16:20:57 client-18 rshd[31622]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark Injecting EBUSY on MDS");echo XXRETCODE:$?'
May 17 16:20:57 client-18 kernel: Lustre: DEBUG MARKER: Injecting EBUSY on MDS
May 17 16:20:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=31621 duration=0(sec)
May 17 16:20:58 client-18 xinetd[3390]: START: shell pid=31656 from=192.168.4.5
May 17 16:20:58 client-18 rshd[31657]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark Stat on a test file");echo XXRETCODE:$?'
May 17 16:20:58 client-18 kernel: Lustre: DEBUG MARKER: Stat on a test file
May 17 16:20:58 client-18 xinetd[3390]: EXIT: shell status=0 pid=31656 duration=0(sec)
May 17 16:21:38 client-18 xinetd[3390]: START: shell pid=31695 from=192.168.4.5
May 17 16:21:38 client-18 rshd[31696]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark Stop injecting EBUSY on MDS");echo XXRETCODE:$?'
May 17 16:21:38 client-18 kernel: Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS
May 17 16:21:38 client-18 xinetd[3390]: EXIT: shell status=0 pid=31695 duration=0(sec)
May 17 16:21:38 client-18 xinetd[3390]: START: shell pid=31730 from=192.168.4.5
May 17 16:21:38 client-18 rshd[31731]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark done");echo XXRETCODE:$?'
May 17 16:21:38 client-18 kernel: Lustre: DEBUG MARKER: done
May 17 16:21:38 client-18 xinetd[3390]: EXIT: shell status=0 pid=31730 duration=0(sec)
May 17 16:22:04 client-18 xinetd[3390]: START: shell pid=31765 from=192.168.4.5
May 17 16:22:04 client-18 rshd[31766]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 16:22:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=31765 duration=0(sec)
May 17 16:22:04 client-18 xinetd[3390]: START: shell pid=31801 from=192.168.4.5
May 17 16:22:04 client-18 rshd[31802]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 16:22:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=31801 duration=0(sec)
May 17 16:22:04 client-18 xinetd[3390]: START: shell pid=31838 from=192.168.4.5
May 17 16:22:04 client-18 rshd[31839]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == conf-sanity test 36: df report consistency on OSTs with different block size ====================== 16:22:04 \(1305674524\)");echo XXRETCODE:$?'
May 17 16:22:04 client-18 kernel: Lustre: DEBUG MARKER: == conf-sanity test 36: df report consistency on OSTs with different block size ====================== 16:22:04 (1305674524)
May 17 16:22:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=31838 duration=0(sec)
May 17 16:22:04 client-18 xinetd[3390]: START: shell pid=31873 from=192.168.4.5
May 17 16:22:04 client-18 rshd[31874]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark  SKIP: conf-sanity test_36 remote OST");echo XXRETCODE:$?'
May 17 16:22:04 client-18 kernel: Lustre: DEBUG MARKER: SKIP: conf-sanity test_36 remote OST
May 17 16:22:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=31873 duration=0(sec)
May 17 16:22:04 client-18 xinetd[3390]: START: shell pid=31908 from=192.168.4.5
May 17 16:22:04 client-18 rshd[31909]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 16:22:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=31908 duration=0(sec)
May 17 16:22:04 client-18 xinetd[3390]: START: shell pid=31944 from=192.168.4.5
May 17 16:22:04 client-18 rshd[31945]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 16:22:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=31944 duration=0(sec)
May 17 16:22:04 client-18 xinetd[3390]: START: shell pid=31981 from=192.168.4.5
May 17 16:22:04 client-18 rshd[31982]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == conf-sanity test 37: verify set tunables works for symlink device ================================= 16:22:04 \(1305674524\)");echo XXRETCODE:$?'
May 17 16:22:05 client-18 kernel: Lustre: DEBUG MARKER: == conf-sanity test 37: verify set tunables works for symlink device ================================= 16:22:04 (1305674524)
May 17 16:22:05 client-18 xinetd[3390]: EXIT: shell status=0 pid=31981 duration=1(sec)
May 17 16:22:12 client-18 xinetd[3390]: START: shell pid=32016 from=192.168.4.5
May 17 16:22:12 client-18 rshd[32017]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 16:22:12 client-18 xinetd[3390]: EXIT: shell status=0 pid=32016 duration=0(sec)
May 17 16:22:12 client-18 xinetd[3390]: START: shell pid=32052 from=192.168.4.5
May 17 16:22:12 client-18 rshd[32053]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 16:22:12 client-18 xinetd[3390]: EXIT: shell status=0 pid=32052 duration=0(sec)
May 17 16:22:12 client-18 xinetd[3390]: START: shell pid=32089 from=192.168.4.5
May 17 16:22:12 client-18 rshd[32090]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == conf-sanity test 38: MDS recreates missing lov_objid file from OST data =========================== 16:22:12 \(1305674532\)");echo XXRETCODE:$?'
May 17 16:22:12 client-18 kernel: Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data =========================== 16:22:12 (1305674532)
May 17 16:22:12 client-18 xinetd[3390]: EXIT: shell status=0 pid=32089 duration=0(sec)
May 17 16:22:19 client-18 xinetd[3390]: START: shell pid=32124 from=192.168.4.5
May 17 16:22:19 client-18 rshd[32125]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark copying 10 files to \/mnt\/lustre\/d0.conf-sanity\/d38");echo XXRETCODE:$?'
May 17 16:22:19 client-18 kernel: Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d0.conf-sanity/d38
May 17 16:22:19 client-18 xinetd[3390]: EXIT: shell status=0 pid=32124 duration=0(sec)
May 17 16:22:27 client-18 xinetd[3390]: START: shell pid=32163 from=192.168.4.5
May 17 16:22:27 client-18 rshd[32164]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark rename lov_objid file on MDS");echo XXRETCODE:$?'
May 17 16:22:27 client-18 kernel: Lustre: DEBUG MARKER: rename lov_objid file on MDS
May 17 16:22:27 client-18 xinetd[3390]: EXIT: shell status=0 pid=32163 duration=0(sec)
May 17 16:22:44 client-18 xinetd[3390]: START: shell pid=32198 from=192.168.4.5
May 17 16:22:44 client-18 rshd[32199]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark files compared the same");echo XXRETCODE:$?'
May 17 16:22:44 client-18 kernel: Lustre: DEBUG MARKER: files compared the same
May 17 16:22:44 client-18 xinetd[3390]: EXIT: shell status=0 pid=32198 duration=0(sec)
May 17 16:22:58 client-18 xinetd[3390]: START: shell pid=32233 from=192.168.4.5
May 17 16:22:58 client-18 rshd[32234]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 16:22:58 client-18 xinetd[3390]: EXIT: shell status=0 pid=32233 duration=0(sec)
May 17 16:22:58 client-18 xinetd[3390]: START: shell pid=32269 from=192.168.4.5
May 17 16:22:58 client-18 rshd[32270]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 16:22:58 client-18 xinetd[3390]: EXIT: shell status=0 pid=32269 duration=0(sec)
May 17 16:22:58 client-18 xinetd[3390]: START: shell pid=32306 from=192.168.4.5
May 17 16:22:58 client-18 rshd[32307]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == conf-sanity test 39: leak_finder recognizes both LUSTRE and LNET malloc messages ================== 16:22:58 \(1305674578\)");echo XXRETCODE:$?'
May 17 16:22:59 client-18 kernel: Lustre: DEBUG MARKER: == conf-sanity test 39: leak_finder recognizes both LUSTRE and LNET malloc messages ================== 16:22:58 (1305674578)
May 17 16:22:59 client-18 xinetd[3390]: EXIT: shell status=0 pid=32306 duration=1(sec)
May 17 16:23:34 client-18 xinetd[3390]: START: shell pid=32341 from=192.168.4.5
May 17 16:23:34 client-18 rshd[32342]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 16:23:34 client-18 xinetd[3390]: EXIT: shell status=0 pid=32341 duration=0(sec)
May 17 16:23:34 client-18 xinetd[3390]: START: shell pid=32377 from=192.168.4.5
May 17 16:23:34 client-18 rshd[32378]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 16:23:34 client-18 xinetd[3390]: EXIT: shell status=0 pid=32377 duration=0(sec)
May 17 16:23:34 client-18 xinetd[3390]: START: shell pid=32414 from=192.168.4.5
May 17 16:23:34 client-18 rshd[32415]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == conf-sanity test 40: race during service thread startup =========================================== 16:23:34 \(1305674614\)");echo XXRETCODE:$?'
May 17 16:23:35 client-18 kernel: Lustre: DEBUG MARKER: == conf-sanity test 40: race during service thread startup =========================================== 16:23:34 (1305674614)
May 17 16:23:35 client-18 xinetd[3390]: EXIT: shell status=0 pid=32414 duration=1(sec)
May 17 16:24:24 client-18 xinetd[3390]: START: shell pid=32453 from=192.168.4.5
May 17 16:24:24 client-18 rshd[32454]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 16:24:24 client-18 xinetd[3390]: EXIT: shell status=0 pid=32453 duration=0(sec)
May 17 16:24:24 client-18 xinetd[3390]: START: shell pid=32489 from=192.168.4.5
May 17 16:24:24 client-18 rshd[32490]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 16:24:24 client-18 xinetd[3390]: EXIT: shell status=0 pid=32489 duration=0(sec)
May 17 16:24:25 client-18 xinetd[3390]: START: shell pid=32526 from=192.168.4.5
May 17 16:24:25 client-18 rshd[32527]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == conf-sanity test 41a: mount mds with --nosvc and --nomgs ========================================== 16:24:24 \(1305674664\)");echo XXRETCODE:$?'
May 17 16:24:25 client-18 kernel: Lustre: DEBUG MARKER: == conf-sanity test 41a: mount mds with --nosvc and --nomgs ========================================== 16:24:24 (1305674664)
May 17 16:24:25 client-18 xinetd[3390]: EXIT: shell status=0 pid=32526 duration=0(sec)
May 17 16:24:57 client-18 xinetd[3390]: START: shell pid=32565 from=192.168.4.5
May 17 16:24:57 client-18 rshd[32566]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 16:24:58 client-18 xinetd[3390]: EXIT: shell status=0 pid=32565 duration=1(sec)
May 17 16:24:58 client-18 xinetd[3390]: START: shell pid=32601 from=192.168.4.5
May 17 16:24:58 client-18 rshd[32602]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 16:24:58 client-18 xinetd[3390]: EXIT: shell status=0 pid=32601 duration=0(sec)
May 17 16:24:58 client-18 xinetd[3390]: START: shell pid=32638 from=192.168.4.5
May 17 16:24:58 client-18 rshd[32639]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount =========================== 16:24:58 \(1305674698\)");echo XXRETCODE:$?'
May 17 16:24:58 client-18 kernel: Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount =========================== 16:24:58 (1305674698)
May 17 16:24:58 client-18 xinetd[3390]: EXIT: shell status=0 pid=32638 duration=0(sec)
May 17 16:24:58 client-18 xinetd[3390]: START: shell pid=32673 from=192.168.4.5
May 17 16:24:58 client-18 rshd[32674]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:; lsof -t /mnt/lustre || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount  /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre is still busy, wait one second && sleep 1;     umount  /mnt/lustre; fi fi");echo XXRETCODE:$?'
May 17 16:24:58 client-18 xinetd[3390]: EXIT: shell status=0 pid=32673 duration=0(sec)
May 17 16:24:58 client-18 xinetd[3390]: START: shell pid=32710 from=192.168.4.5
May 17 16:24:58 client-18 rshd[32711]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:; lsof -t /mnt/lustre2 || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre2 | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount  /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre2 is still busy, wait one second && sleep 1;     umount  /mnt/lustre2; fi fi");echo XXRETCODE:$?'
May 17 16:24:59 client-18 xinetd[3390]: EXIT: shell status=0 pid=32710 duration=1(sec)
May 17 16:25:00 client-18 xinetd[3390]: START: shell pid=32747 from=192.168.4.5
May 17 16:25:00 client-18 rshd[32748]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:; lsof -t /mnt/lustre || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount  /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre is still busy, wait one second && sleep 1;     umount  /mnt/lustre; fi fi");echo XXRETCODE:$?'
May 17 16:25:00 client-18 xinetd[3390]: EXIT: shell status=0 pid=32747 duration=0(sec)
May 17 16:25:00 client-18 xinetd[3390]: START: shell pid=318 from=192.168.4.5
May 17 16:25:00 client-18 rshd[319]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:; lsof -t /mnt/lustre2 || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre2 | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount  /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre2 is still busy, wait one second && sleep 1;     umount  /mnt/lustre2; fi fi");echo XXRETCODE:$?'
May 17 16:25:00 client-18 xinetd[3390]: EXIT: shell status=0 pid=318 duration=0(sec)
May 17 16:25:29 client-18 xinetd[3390]: START: shell pid=355 from=192.168.4.5
May 17 16:25:29 client-18 rshd[356]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 16:25:29 client-18 xinetd[3390]: EXIT: shell status=0 pid=355 duration=0(sec)
May 17 16:25:29 client-18 xinetd[3390]: START: shell pid=391 from=192.168.4.5
May 17 16:25:29 client-18 rshd[392]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 16:25:29 client-18 xinetd[3390]: EXIT: shell status=0 pid=391 duration=0(sec)
May 17 16:25:29 client-18 xinetd[3390]: START: shell pid=436 from=192.168.4.5
May 17 16:25:29 client-18 rshd[437]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == conf-sanity test 42: invalid config param should not prevent client from mounting ================= 16:25:29 \(1305674729\)");echo XXRETCODE:$?'
May 17 16:25:29 client-18 kernel: Lustre: DEBUG MARKER: == conf-sanity test 42: invalid config param should not prevent client from mounting ================= 16:25:29 (1305674729)
May 17 16:25:29 client-18 xinetd[3390]: EXIT: shell status=0 pid=436 duration=0(sec)
May 17 16:26:03 client-18 xinetd[3390]: START: shell pid=471 from=192.168.4.5
May 17 16:26:03 client-18 rshd[472]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 16:26:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=471 duration=0(sec)
May 17 16:26:03 client-18 xinetd[3390]: START: shell pid=507 from=192.168.4.5
May 17 16:26:03 client-18 rshd[508]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 16:26:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=507 duration=0(sec)
May 17 16:26:03 client-18 xinetd[3390]: START: shell pid=544 from=192.168.4.5
May 17 16:26:03 client-18 rshd[545]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == conf-sanity test 43: check root_squash and nosquash_nids ========================================== 16:26:03 \(1305674763\)");echo XXRETCODE:$?'
May 17 16:26:04 client-18 kernel: Lustre: DEBUG MARKER: == conf-sanity test 43: check root_squash and nosquash_nids ========================================== 16:26:03 (1305674763)
May 17 16:26:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=544 duration=1(sec)
May 17 16:26:27 client-18 xinetd[3390]: START: shell pid=584 from=192.168.4.5
May 17 16:26:27 client-18 rshd[585]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 16:26:27 client-18 xinetd[3390]: EXIT: shell status=0 pid=584 duration=0(sec)
May 17 16:26:27 client-18 xinetd[3390]: START: shell pid=632 from=192.168.4.5
May 17 16:26:27 client-18 rshd[633]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 16:26:27 client-18 xinetd[3390]: EXIT: shell status=0 pid=632 duration=0(sec)
May 17 16:26:49 client-18 xinetd[3390]: START: shell pid=671 from=192.168.4.5
May 17 16:26:49 client-18 rshd[672]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == conf-sanity test 44: mounted client proc entry exists ============================================= 16:26:48 \(1305674808\)");echo XXRETCODE:$?'
May 17 16:26:49 client-18 kernel: Lustre: DEBUG MARKER: == conf-sanity test 44: mounted client proc entry exists ============================================= 16:26:48 (1305674808)
May 17 16:26:49 client-18 xinetd[3390]: EXIT: shell status=0 pid=671 duration=0(sec)
May 17 16:27:28 client-18 xinetd[3390]: START: shell pid=711 from=192.168.4.5
May 17 16:27:28 client-18 rshd[712]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 16:27:28 client-18 xinetd[3390]: EXIT: shell status=0 pid=711 duration=0(sec)
May 17 16:27:28 client-18 xinetd[3390]: START: shell pid=747 from=192.168.4.5
May 17 16:27:28 client-18 rshd[748]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 16:27:28 client-18 xinetd[3390]: EXIT: shell status=0 pid=747 duration=0(sec)
May 17 16:27:28 client-18 xinetd[3390]: START: shell pid=784 from=192.168.4.5
May 17 16:27:28 client-18 rshd[785]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == conf-sanity test 45: long unlink handling in ptlrpcd ============================================== 16:27:28 \(1305674848\)");echo XXRETCODE:$?'
May 17 16:27:29 client-18 kernel: Lustre: DEBUG MARKER: == conf-sanity test 45: long unlink handling in ptlrpcd ============================================== 16:27:28 (1305674848)
May 17 16:27:29 client-18 xinetd[3390]: EXIT: shell status=0 pid=784 duration=1(sec)
May 17 16:27:53 client-18 xinetd[3390]: START: shell pid=819 from=192.168.4.5
May 17 16:27:53 client-18 rshd[820]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark sleep 60 sec");echo XXRETCODE:$?'
May 17 16:27:53 client-18 kernel: Lustre: DEBUG MARKER: sleep 60 sec
May 17 16:27:53 client-18 xinetd[3390]: EXIT: shell status=0 pid=819 duration=0(sec)
May 17 16:28:53 client-18 xinetd[3390]: START: shell pid=858 from=192.168.4.5
May 17 16:28:53 client-18 rshd[859]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark sleep 10 sec");echo XXRETCODE:$?'
May 17 16:28:53 client-18 kernel: Lustre: DEBUG MARKER: sleep 10 sec
May 17 16:28:53 client-18 xinetd[3390]: EXIT: shell status=0 pid=858 duration=0(sec)
May 17 16:34:20 client-18 xinetd[3390]: START: shell pid=909 from=192.168.4.5
May 17 16:34:20 client-18 rshd[910]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 16:34:20 client-18 xinetd[3390]: EXIT: shell status=0 pid=909 duration=0(sec)
May 17 16:34:20 client-18 xinetd[3390]: START: shell pid=945 from=192.168.4.5
May 17 16:34:20 client-18 rshd[946]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 16:34:20 client-18 xinetd[3390]: EXIT: shell status=0 pid=945 duration=0(sec)
May 17 16:34:20 client-18 xinetd[3390]: START: shell pid=982 from=192.168.4.5
May 17 16:34:20 client-18 rshd[983]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == conf-sanity test 46a: handle ost additional - wide striped file =================================== 16:34:20 \(1305675260\)");echo XXRETCODE:$?'
May 17 16:34:21 client-18 kernel: Lustre: DEBUG MARKER: == conf-sanity test 46a: handle ost additional - wide striped file =================================== 16:34:20 (1305675260)
May 17 16:34:21 client-18 xinetd[3390]: EXIT: shell status=0 pid=982 duration=1(sec)
May 17 16:34:21 client-18 xinetd[3390]: START: shell pid=1017 from=192.168.4.5
May 17 16:34:21 client-18 rshd[1018]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:; lsof -t /mnt/lustre || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount  /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre is still busy, wait one second && sleep 1;     umount  /mnt/lustre; fi fi");echo XXRETCODE:$?'
May 17 16:34:21 client-18 xinetd[3390]: EXIT: shell status=0 pid=1017 duration=0(sec)
May 17 16:34:21 client-18 xinetd[3390]: START: shell pid=1054 from=192.168.4.5
May 17 16:34:21 client-18 rshd[1055]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:; lsof -t /mnt/lustre2 || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre2 | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount  /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre2 is still busy, wait one second && sleep 1;     umount  /mnt/lustre2; fi fi");echo XXRETCODE:$?'
May 17 16:34:21 client-18 xinetd[3390]: EXIT: shell status=0 pid=1054 duration=0(sec)
May 17 16:34:33 client-18 xinetd[3390]: START: shell pid=1091 from=192.168.4.5
May 17 16:34:33 client-18 rshd[1092]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark ost now in FULL state");echo XXRETCODE:$?'
May 17 16:34:33 client-18 kernel: Lustre: DEBUG MARKER: ost now in FULL state
May 17 16:34:33 client-18 xinetd[3390]: EXIT: shell status=0 pid=1091 duration=0(sec)
May 17 16:34:58 client-18 xinetd[3390]: START: shell pid=1130 from=192.168.4.5
May 17 16:34:58 client-18 rshd[1131]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark ost now in FULL state");echo XXRETCODE:$?'
May 17 16:34:58 client-18 kernel: Lustre: DEBUG MARKER: ost now in FULL state
May 17 16:34:58 client-18 xinetd[3390]: EXIT: shell status=0 pid=1130 duration=0(sec)
May 17 16:35:03 client-18 xinetd[3390]: START: shell pid=1165 from=192.168.4.5
May 17 16:35:03 client-18 rshd[1166]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark ost2 now in FULL state");echo XXRETCODE:$?'
May 17 16:35:03 client-18 kernel: Lustre: DEBUG MARKER: ost2 now in FULL state
May 17 16:35:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=1165 duration=0(sec)
May 17 16:35:03 client-18 xinetd[3390]: START: shell pid=1200 from=192.168.4.5
May 17 16:35:03 client-18 rshd[1201]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark ost3 now in FULL state");echo XXRETCODE:$?'
May 17 16:35:03 client-18 kernel: Lustre: DEBUG MARKER: ost3 now in FULL state
May 17 16:35:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=1200 duration=0(sec)
May 17 16:35:04 client-18 xinetd[3390]: START: shell pid=1235 from=192.168.4.5
May 17 16:35:04 client-18 rshd[1236]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark ost4 now in FULL state");echo XXRETCODE:$?'
May 17 16:35:04 client-18 kernel: Lustre: DEBUG MARKER: ost4 now in FULL state
May 17 16:35:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=1235 duration=0(sec)
May 17 16:35:05 client-18 xinetd[3390]: START: shell pid=1270 from=192.168.4.5
May 17 16:35:05 client-18 rshd[1271]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark ost5 now in FULL state");echo XXRETCODE:$?'
May 17 16:35:05 client-18 kernel: Lustre: DEBUG MARKER: ost5 now in FULL state
May 17 16:35:05 client-18 xinetd[3390]: EXIT: shell status=0 pid=1270 duration=0(sec)
May 17 16:35:06 client-18 xinetd[3390]: START: shell pid=1305 from=192.168.4.5
May 17 16:35:06 client-18 rshd[1306]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark ost6 now in FULL state");echo XXRETCODE:$?'
May 17 16:35:06 client-18 kernel: Lustre: DEBUG MARKER: ost6 now in FULL state
May 17 16:35:06 client-18 xinetd[3390]: EXIT: shell status=0 pid=1305 duration=0(sec)
May 17 16:35:39 client-18 xinetd[3390]: START: shell pid=1340 from=192.168.4.5
May 17 16:35:39 client-18 rshd[1341]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 16:35:39 client-18 xinetd[3390]: EXIT: shell status=0 pid=1340 duration=0(sec)
May 17 16:35:39 client-18 xinetd[3390]: START: shell pid=1376 from=192.168.4.5
May 17 16:35:39 client-18 rshd[1377]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 16:35:39 client-18 xinetd[3390]: EXIT: shell status=0 pid=1376 duration=0(sec)
May 17 16:35:39 client-18 xinetd[3390]: START: shell pid=1413 from=192.168.4.5
May 17 16:35:39 client-18 rshd[1414]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == conf-sanity test 47: server restart does not make client loss lru_resize settings ================= 16:35:39 \(1305675339\)");echo XXRETCODE:$?'
May 17 16:35:39 client-18 kernel: Lustre: DEBUG MARKER: == conf-sanity test 47: server restart does not make client loss lru_resize settings ================= 16:35:39 (1305675339)
May 17 16:35:39 client-18 xinetd[3390]: EXIT: shell status=0 pid=1413 duration=0(sec)
May 17 16:35:39 client-18 xinetd[3390]: START: shell pid=1449 from=192.168.4.5
May 17 16:35:39 client-18 rshd[1450]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:; lsof -t /mnt/lustre || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount  /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre is still busy, wait one second && sleep 1;     umount  /mnt/lustre; fi fi");echo XXRETCODE:$?'
May 17 16:35:39 client-18 xinetd[3390]: START: shell pid=1487 from=192.168.4.5
May 17 16:35:39 client-18 xinetd[3390]: EXIT: shell status=0 pid=1449 duration=0(sec)
May 17 16:35:39 client-18 rshd[1488]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:; lsof -t /mnt/lustre2 || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre2 | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount  /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre2 is still busy, wait one second && sleep 1;     umount  /mnt/lustre2; fi fi");echo XXRETCODE:$?'
May 17 16:35:40 client-18 xinetd[3390]: EXIT: shell status=0 pid=1487 duration=1(sec)
May 17 16:36:45 client-18 xinetd[3390]: START: shell pid=1528 from=192.168.4.5
May 17 16:36:45 client-18 rshd[1529]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 16:36:45 client-18 xinetd[3390]: EXIT: shell status=0 pid=1528 duration=0(sec)
May 17 16:36:45 client-18 xinetd[3390]: START: shell pid=1564 from=192.168.4.5
May 17 16:36:45 client-18 rshd[1565]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 16:36:45 client-18 xinetd[3390]: EXIT: shell status=0 pid=1564 duration=0(sec)
May 17 16:36:46 client-18 xinetd[3390]: START: shell pid=1601 from=192.168.4.5
May 17 16:36:46 client-18 rshd[1602]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == conf-sanity test 48: too many acls on file ======================================================== 16:36:45 \(1305675405\)");echo XXRETCODE:$?'
May 17 16:36:46 client-18 kernel: Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ======================================================== 16:36:45 (1305675405)
May 17 16:36:46 client-18 xinetd[3390]: EXIT: shell status=0 pid=1601 duration=0(sec)
May 17 16:36:46 client-18 xinetd[3390]: START: shell pid=1636 from=192.168.4.5
May 17 16:36:46 client-18 rshd[1637]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:; lsof -t /mnt/lustre || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount  /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre is still busy, wait one second && sleep 1;     umount  /mnt/lustre; fi fi");echo XXRETCODE:$?'
May 17 16:36:46 client-18 xinetd[3390]: EXIT: shell status=0 pid=1636 duration=0(sec)
May 17 16:36:46 client-18 xinetd[3390]: START: shell pid=1674 from=192.168.4.5
May 17 16:36:46 client-18 rshd[1675]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:; lsof -t /mnt/lustre2 || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre2 | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount  /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre2 is still busy, wait one second && sleep 1;     umount  /mnt/lustre2; fi fi");echo XXRETCODE:$?'
May 17 16:36:46 client-18 xinetd[3390]: EXIT: shell status=0 pid=1674 duration=0(sec)
May 17 16:37:00 client-18 xinetd[3390]: START: shell pid=1713 from=192.168.4.5
May 17 16:37:00 client-18 rshd[1714]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:; lsof -t /mnt/lustre || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount  /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre is still busy, wait one second && sleep 1;     umount  /mnt/lustre; fi fi");echo XXRETCODE:$?'
May 17 16:37:00 client-18 xinetd[3390]: EXIT: shell status=0 pid=1713 duration=0(sec)
May 17 16:37:00 client-18 xinetd[3390]: START: shell pid=1750 from=192.168.4.5
May 17 16:37:00 client-18 rshd[1751]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:; lsof -t /mnt/lustre2 || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre2 | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount  /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre2 is still busy, wait one second && sleep 1;     umount  /mnt/lustre2; fi fi");echo XXRETCODE:$?'
May 17 16:37:00 client-18 xinetd[3390]: EXIT: shell status=0 pid=1750 duration=0(sec)
May 17 16:37:30 client-18 xinetd[3390]: START: shell pid=1791 from=192.168.4.5
May 17 16:37:30 client-18 rshd[1792]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark ost now in FULL state");echo XXRETCODE:$?'
May 17 16:37:30 client-18 kernel: Lustre: DEBUG MARKER: ost now in FULL state
May 17 16:37:30 client-18 xinetd[3390]: EXIT: shell status=0 pid=1791 duration=0(sec)
May 17 16:37:42 client-18 xinetd[3390]: START: shell pid=1826 from=192.168.4.5
May 17 16:37:42 client-18 rshd[1827]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 16:37:42 client-18 xinetd[3390]: EXIT: shell status=0 pid=1826 duration=0(sec)
May 17 16:37:42 client-18 xinetd[3390]: START: shell pid=1862 from=192.168.4.5
May 17 16:37:42 client-18 rshd[1863]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 16:37:42 client-18 xinetd[3390]: EXIT: shell status=0 pid=1862 duration=0(sec)
May 17 16:37:42 client-18 xinetd[3390]: START: shell pid=1904 from=192.168.4.5
May 17 16:37:42 client-18 rshd[1905]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == conf-sanity test 49: check PARAM_SYS_LDLM_TIMEOUT option of MKFS.LUSTRE =========================== 16:37:42 \(1305675462\)");echo XXRETCODE:$?'
May 17 16:37:42 client-18 kernel: Lustre: DEBUG MARKER: == conf-sanity test 49: check PARAM_SYS_LDLM_TIMEOUT option of MKFS.LUSTRE =========================== 16:37:42 (1305675462)
May 17 16:37:42 client-18 xinetd[3390]: EXIT: shell status=0 pid=1904 duration=0(sec)
May 17 16:37:42 client-18 xinetd[3390]: START: shell pid=1939 from=192.168.4.5
May 17 16:37:42 client-18 rshd[1940]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:; lsof -t /mnt/lustre || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount  /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre is still busy, wait one second && sleep 1;     umount  /mnt/lustre; fi fi");echo XXRETCODE:$?'
May 17 16:37:42 client-18 xinetd[3390]: EXIT: shell status=0 pid=1939 duration=0(sec)
May 17 16:37:42 client-18 xinetd[3390]: START: shell pid=1976 from=192.168.4.5
May 17 16:37:42 client-18 rshd[1977]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:; lsof -t /mnt/lustre2 || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre2 | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount  /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre2 is still busy, wait one second && sleep 1;     umount  /mnt/lustre2; fi fi");echo XXRETCODE:$?'
May 17 16:37:42 client-18 xinetd[3390]: EXIT: shell status=0 pid=1976 duration=0(sec)
May 17 16:38:07 client-18 xinetd[3390]: START: shell pid=2013 from=192.168.4.5
May 17 16:38:07 client-18 rshd[2014]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:; lsof -t /mnt/lustre || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount  /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre is still busy, wait one second && sleep 1;     umount  /mnt/lustre; fi fi");echo XXRETCODE:$?'
May 17 16:38:07 client-18 xinetd[3390]: EXIT: shell status=0 pid=2013 duration=0(sec)
May 17 16:38:07 client-18 xinetd[3390]: START: shell pid=2051 from=192.168.4.5
May 17 16:38:07 client-18 rshd[2052]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:; lsof -t /mnt/lustre2 || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre2 | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount  /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre2 is still busy, wait one second && sleep 1;     umount  /mnt/lustre2; fi fi");echo XXRETCODE:$?'
May 17 16:38:07 client-18 xinetd[3390]: EXIT: shell status=0 pid=2051 duration=0(sec)
May 17 16:38:42 client-18 xinetd[3390]: START: shell pid=2092 from=192.168.4.5
May 17 16:38:42 client-18 rshd[2093]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 16:38:42 client-18 xinetd[3390]: EXIT: shell status=0 pid=2092 duration=0(sec)
May 17 16:38:42 client-18 xinetd[3390]: START: shell pid=2129 from=192.168.4.5
May 17 16:38:42 client-18 rshd[2130]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 16:38:42 client-18 xinetd[3390]: EXIT: shell status=0 pid=2129 duration=0(sec)
May 17 16:38:43 client-18 xinetd[3390]: START: shell pid=2167 from=192.168.4.5
May 17 16:38:43 client-18 rshd[2168]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == conf-sanity test 50a: lazystatfs all servers available ============================================ 16:38:43 \(1305675523\)");echo XXRETCODE:$?'
May 17 16:38:43 client-18 kernel: Lustre: DEBUG MARKER: == conf-sanity test 50a: lazystatfs all servers available ============================================ 16:38:43 (1305675523)
May 17 16:38:43 client-18 xinetd[3390]: EXIT: shell status=0 pid=2167 duration=0(sec)
May 17 16:39:20 client-18 xinetd[3390]: START: shell pid=2206 from=192.168.4.5
May 17 16:39:20 client-18 rshd[2207]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 16:39:20 client-18 xinetd[3390]: EXIT: shell status=0 pid=2206 duration=0(sec)
May 17 16:39:20 client-18 xinetd[3390]: START: shell pid=2242 from=192.168.4.5
May 17 16:39:20 client-18 rshd[2246]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 16:39:20 client-18 xinetd[3390]: EXIT: shell status=0 pid=2242 duration=0(sec)
May 17 16:39:21 client-18 xinetd[3390]: START: shell pid=2287 from=192.168.4.5
May 17 16:39:21 client-18 rshd[2288]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == conf-sanity test 50b: lazystatfs all servers down ================================================= 16:39:20 \(1305675560\)");echo XXRETCODE:$?'
May 17 16:39:21 client-18 kernel: Lustre: DEBUG MARKER: == conf-sanity test 50b: lazystatfs all servers down ================================================= 16:39:20 (1305675560)
May 17 16:39:21 client-18 xinetd[3390]: EXIT: shell status=0 pid=2287 duration=0(sec)
May 17 16:40:05 client-18 xinetd[3390]: START: shell pid=2326 from=192.168.4.5
May 17 16:40:05 client-18 rshd[2327]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 16:40:05 client-18 xinetd[3390]: EXIT: shell status=0 pid=2326 duration=0(sec)
May 17 16:40:05 client-18 xinetd[3390]: START: shell pid=2362 from=192.168.4.5
May 17 16:40:05 client-18 rshd[2363]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 16:40:05 client-18 xinetd[3390]: EXIT: shell status=0 pid=2362 duration=0(sec)
May 17 16:40:05 client-18 xinetd[3390]: START: shell pid=2408 from=192.168.4.5
May 17 16:40:05 client-18 rshd[2409]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == conf-sanity test 50c: lazystatfs one server down ================================================== 16:40:05 \(1305675605\)");echo XXRETCODE:$?'
May 17 16:40:05 client-18 kernel: Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ================================================== 16:40:05 (1305675605)
May 17 16:40:05 client-18 xinetd[3390]: EXIT: shell status=0 pid=2408 duration=0(sec)
May 17 16:40:24 client-18 xinetd[3390]: START: shell pid=2445 from=192.168.4.5
May 17 16:40:24 client-18 rshd[2446]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark ost now in DISCONN state");echo XXRETCODE:$?'
May 17 16:40:24 client-18 kernel: Lustre: DEBUG MARKER: ost now in DISCONN state
May 17 16:40:24 client-18 xinetd[3390]: EXIT: shell status=0 pid=2445 duration=0(sec)
May 17 16:40:38 client-18 xinetd[3390]: START: shell pid=2481 from=192.168.4.5
May 17 16:40:38 client-18 rshd[2482]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 16:40:38 client-18 xinetd[3390]: EXIT: shell status=0 pid=2481 duration=0(sec)
May 17 16:40:38 client-18 xinetd[3390]: START: shell pid=2517 from=192.168.4.5
May 17 16:40:38 client-18 rshd[2518]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 16:40:38 client-18 xinetd[3390]: EXIT: shell status=0 pid=2517 duration=0(sec)
May 17 16:40:38 client-18 xinetd[3390]: START: shell pid=2559 from=192.168.4.5
May 17 16:40:38 client-18 rshd[2560]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == conf-sanity test 50d: lazystatfs client\/server conn race ========================================== 16:40:38 \(1305675638\)");echo XXRETCODE:$?'
May 17 16:40:38 client-18 kernel: Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================== 16:40:38 (1305675638)
May 17 16:40:38 client-18 xinetd[3390]: EXIT: shell status=0 pid=2559 duration=0(sec)
May 17 16:41:06 client-18 xinetd[3390]: START: shell pid=2594 from=192.168.4.5
May 17 16:41:06 client-18 rshd[2595]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 16:41:06 client-18 xinetd[3390]: EXIT: shell status=0 pid=2594 duration=0(sec)
May 17 16:41:06 client-18 xinetd[3390]: START: shell pid=2630 from=192.168.4.5
May 17 16:41:06 client-18 rshd[2631]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 16:41:06 client-18 xinetd[3390]: EXIT: shell status=0 pid=2630 duration=0(sec)
May 17 16:41:07 client-18 xinetd[3390]: START: shell pid=2667 from=192.168.4.5
May 17 16:41:07 client-18 rshd[2668]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == conf-sanity test 50e: normal statfs all servers down ============================================== 16:41:06 \(1305675666\)");echo XXRETCODE:$?'
May 17 16:41:07 client-18 kernel: Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down ============================================== 16:41:06 (1305675666)
May 17 16:41:07 client-18 xinetd[3390]: EXIT: shell status=0 pid=2667 duration=0(sec)
May 17 16:41:07 client-18 xinetd[3390]: START: shell pid=2702 from=192.168.4.5
May 17 16:41:07 client-18 rshd[2703]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:; lsof -t /mnt/lustre || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount  /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre is still busy, wait one second && sleep 1;     umount  /mnt/lustre; fi fi");echo XXRETCODE:$?'
May 17 16:41:07 client-18 xinetd[3390]: EXIT: shell status=0 pid=2702 duration=0(sec)
May 17 16:41:07 client-18 xinetd[3390]: START: shell pid=2739 from=192.168.4.5
May 17 16:41:07 client-18 rshd[2740]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:; lsof -t /mnt/lustre2 || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre2 | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount  /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre2 is still busy, wait one second && sleep 1;     umount  /mnt/lustre2; fi fi");echo XXRETCODE:$?'
May 17 16:41:07 client-18 xinetd[3390]: EXIT: shell status=0 pid=2739 duration=0(sec)
May 17 16:41:20 client-18 xinetd[3390]: START: shell pid=2780 from=192.168.4.5
May 17 16:41:20 client-18 rshd[2781]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark ost now in FULL state");echo XXRETCODE:$?'
May 17 16:41:20 client-18 kernel: Lustre: DEBUG MARKER: ost now in FULL state
May 17 16:41:20 client-18 xinetd[3390]: EXIT: shell status=0 pid=2780 duration=0(sec)
May 17 16:41:42 client-18 xinetd[3390]: START: shell pid=2815 from=192.168.4.5
May 17 16:41:42 client-18 rshd[2816]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark ost now in FULL state");echo XXRETCODE:$?'
May 17 16:41:42 client-18 kernel: Lustre: DEBUG MARKER: ost now in FULL state
May 17 16:41:42 client-18 xinetd[3390]: EXIT: shell status=0 pid=2815 duration=0(sec)
May 17 16:41:48 client-18 xinetd[3390]: START: shell pid=2850 from=192.168.4.5
May 17 16:41:48 client-18 rshd[2851]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark ost now in DISCONN state");echo XXRETCODE:$?'
May 17 16:41:48 client-18 kernel: Lustre: DEBUG MARKER: ost now in DISCONN state
May 17 16:41:48 client-18 xinetd[3390]: EXIT: shell status=0 pid=2850 duration=0(sec)
May 17 16:42:26 client-18 xinetd[3390]: START: shell pid=2889 from=192.168.4.5
May 17 16:42:26 client-18 rshd[2890]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 16:42:26 client-18 xinetd[3390]: EXIT: shell status=0 pid=2889 duration=0(sec)
May 17 16:42:26 client-18 xinetd[3390]: START: shell pid=2925 from=192.168.4.5
May 17 16:42:26 client-18 rshd[2926]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 16:42:26 client-18 xinetd[3390]: EXIT: shell status=0 pid=2925 duration=0(sec)
May 17 16:42:26 client-18 xinetd[3390]: START: shell pid=2962 from=192.168.4.5
May 17 16:42:26 client-18 rshd[2963]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == conf-sanity test 50f: normal statfs one server in down ============================================ 16:42:26 \(1305675746\)");echo XXRETCODE:$?'
May 17 16:42:26 client-18 kernel: Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ============================================ 16:42:26 (1305675746)
May 17 16:42:26 client-18 xinetd[3390]: EXIT: shell status=0 pid=2962 duration=0(sec)
May 17 16:42:37 client-18 xinetd[3390]: START: shell pid=3001 from=192.168.4.5
May 17 16:42:37 client-18 rshd[3002]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark ost now in FULL state");echo XXRETCODE:$?'
May 17 16:42:37 client-18 kernel: Lustre: DEBUG MARKER: ost now in FULL state
May 17 16:42:37 client-18 xinetd[3390]: EXIT: shell status=0 pid=3001 duration=0(sec)
May 17 16:42:43 client-18 xinetd[3390]: START: shell pid=3038 from=192.168.4.5
May 17 16:42:43 client-18 rshd[3039]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark ost2 now in FULL state");echo XXRETCODE:$?'
May 17 16:42:43 client-18 kernel: Lustre: DEBUG MARKER: ost2 now in FULL state
May 17 16:42:43 client-18 xinetd[3390]: EXIT: shell status=0 pid=3038 duration=0(sec)
May 17 16:42:48 client-18 xinetd[3390]: START: shell pid=3077 from=192.168.4.5
May 17 16:42:48 client-18 rshd[3078]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark ost2 now in DISCONN state");echo XXRETCODE:$?'
May 17 16:42:48 client-18 kernel: Lustre: DEBUG MARKER: ost2 now in DISCONN state
May 17 16:42:48 client-18 xinetd[3390]: EXIT: shell status=0 pid=3077 duration=0(sec)
May 17 16:43:29 client-18 xinetd[3390]: START: shell pid=3114 from=192.168.4.5
May 17 16:43:29 client-18 rshd[3115]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 16:43:29 client-18 xinetd[3390]: EXIT: shell status=0 pid=3114 duration=0(sec)
May 17 16:43:29 client-18 xinetd[3390]: START: shell pid=3155 from=192.168.4.5
May 17 16:43:29 client-18 rshd[3156]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 16:43:29 client-18 xinetd[3390]: EXIT: shell status=0 pid=3155 duration=0(sec)
May 17 16:43:29 client-18 xinetd[3390]: START: shell pid=3196 from=192.168.4.5
May 17 16:43:29 client-18 rshd[3197]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == conf-sanity test 50g: deactivated OST should not cause panic======================================= 16:43:29 \(1305675809\)");echo XXRETCODE:$?'
May 17 16:43:29 client-18 kernel: Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic======================================= 16:43:29 (1305675809)
May 17 16:43:29 client-18 xinetd[3390]: EXIT: shell status=0 pid=3196 duration=0(sec)
May 17 16:43:41 client-18 xinetd[3390]: START: shell pid=3236 from=192.168.4.5
May 17 16:43:41 client-18 rshd[3237]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark ost2 now in FULL state");echo XXRETCODE:$?'
May 17 16:43:41 client-18 kernel: Lustre: DEBUG MARKER: ost2 now in FULL state
May 17 16:43:41 client-18 xinetd[3390]: EXIT: shell status=0 pid=3236 duration=0(sec)
May 17 16:43:41 client-18 xinetd[3390]: START: shell pid=3276 from=192.168.4.5
May 17 16:43:41 client-18 rshd[3277]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark ost2 now in FULL state");echo XXRETCODE:$?'
May 17 16:43:41 client-18 kernel: Lustre: DEBUG MARKER: ost2 now in FULL state
May 17 16:43:41 client-18 xinetd[3390]: EXIT: shell status=0 pid=3276 duration=0(sec)
May 17 16:43:56 client-18 xinetd[3390]: START: shell pid=3313 from=192.168.4.5
May 17 16:43:56 client-18 rshd[3315]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 16:43:56 client-18 xinetd[3390]: EXIT: shell status=0 pid=3313 duration=0(sec)
May 17 16:43:56 client-18 xinetd[3390]: START: shell pid=3357 from=192.168.4.5
May 17 16:43:56 client-18 rshd[3358]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 16:43:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=3357 duration=1(sec)
May 17 16:43:57 client-18 xinetd[3390]: START: shell pid=3400 from=192.168.4.5
May 17 16:43:57 client-18 rshd[3402]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ====== 16:43:57 \(1305675837\)");echo XXRETCODE:$?'
May 17 16:43:57 client-18 kernel: Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ====== 16:43:57 (1305675837)
May 17 16:43:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=3400 duration=0(sec)
May 17 16:43:57 client-18 xinetd[3390]: START: shell pid=3439 from=192.168.4.5
May 17 16:43:57 client-18 rshd[3440]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:; lsof -t /mnt/lustre || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount  /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre is still busy, wait one second && sleep 1;     umount  /mnt/lustre; fi fi");echo XXRETCODE:$?'
May 17 16:43:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=3439 duration=0(sec)
May 17 16:43:57 client-18 xinetd[3390]: START: shell pid=3479 from=192.168.4.5
May 17 16:43:57 client-18 rshd[3480]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:; lsof -t /mnt/lustre2 || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre2 | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount  /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre2 is still busy, wait one second && sleep 1;     umount  /mnt/lustre2; fi fi");echo XXRETCODE:$?'
May 17 16:43:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=3479 duration=0(sec)
May 17 16:44:57 client-18 xinetd[3390]: START: shell pid=3529 from=192.168.4.5
May 17 16:44:57 client-18 rshd[3530]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 16:44:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=3529 duration=0(sec)
May 17 16:44:57 client-18 xinetd[3390]: START: shell pid=3565 from=192.168.4.5
May 17 16:44:57 client-18 rshd[3566]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 16:44:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=3565 duration=0(sec)
May 17 16:44:58 client-18 xinetd[3390]: START: shell pid=3605 from=192.168.4.5
May 17 16:44:58 client-18 rshd[3606]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == conf-sanity test 52: check recovering objects from lost+found ===================================== 16:44:57 \(1305675897\)");echo XXRETCODE:$?'
May 17 16:44:58 client-18 kernel: Lustre: DEBUG MARKER: == conf-sanity test 52: check recovering objects from lost+found ===================================== 16:44:57 (1305675897)
May 17 16:44:58 client-18 xinetd[3390]: EXIT: shell status=0 pid=3605 duration=0(sec)
May 17 16:45:41 client-18 xinetd[3390]: START: shell pid=3640 from=192.168.4.5
May 17 16:45:41 client-18 rshd[3641]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 16:45:41 client-18 xinetd[3390]: EXIT: shell status=0 pid=3640 duration=0(sec)
May 17 16:45:41 client-18 xinetd[3390]: START: shell pid=3676 from=192.168.4.5
May 17 16:45:41 client-18 rshd[3677]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 16:45:41 client-18 xinetd[3390]: EXIT: shell status=0 pid=3676 duration=0(sec)
May 17 16:45:42 client-18 xinetd[3390]: START: shell pid=3713 from=192.168.4.5
May 17 16:45:42 client-18 rshd[3714]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == conf-sanity test 53a: check OSS thread count params =============================================== 16:45:41 \(1305675941\)");echo XXRETCODE:$?'
May 17 16:45:42 client-18 kernel: Lustre: DEBUG MARKER: == conf-sanity test 53a: check OSS thread count params =============================================== 16:45:41 (1305675941)
May 17 16:45:42 client-18 xinetd[3390]: EXIT: shell status=0 pid=3713 duration=0(sec)
May 17 16:46:17 client-18 xinetd[3390]: START: shell pid=3752 from=192.168.4.5
May 17 16:46:17 client-18 rshd[3753]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/tests:/usr/lib64/openmpi/1.4-gcc/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/usr/lib64/lustre/utils:/usr/lib64/lustre/utils/gss:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey::/usr/lib64/lustre/tests/mpi NAME=sarah sh rpc.sh /usr/sbin/lustre_rmmod ldiskfs ");echo XXRETCODE:$?'
May 17 16:46:23 client-18 kernel: Lustre: Acceptor stopping
May 17 16:46:24 client-18 kernel: Lustre: Removed LNI 10.10.4.18@tcp
May 17 16:46:26 client-18 kernel: Lustre: Removed LNI 192.168.4.18@o2ib
May 17 16:46:27 client-18 xinetd[3390]: EXIT: shell status=0 pid=3752 duration=10(sec)
May 17 16:46:29 client-18 xinetd[3390]: START: shell pid=4126 from=192.168.4.5
May 17 16:46:29 client-18 rshd[4127]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/tests:/usr/lib64/openmpi/1.4-gcc/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/usr/lib64/lustre/utils:/usr/lib64/lustre/utils/gss:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey::/usr/lib64/lustre/tests/mpi NAME=sarah sh rpc.sh check_mem_leak ");echo XXRETCODE:$?'
May 17 16:46:29 client-18 xinetd[3390]: EXIT: shell status=0 pid=4126 duration=0(sec)
May 17 16:46:30 client-18 xinetd[3390]: START: shell pid=4218 from=192.168.4.5
May 17 16:46:30 client-18 rshd[4219]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" MODOPTS_OST=" oss_num_threads=64" sh -c "PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/tests:/usr/lib64/openmpi/1.4-gcc/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/usr/lib64/lustre/utils:/usr/lib64/lustre/utils/gss:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey::/usr/lib64/lustre/tests/mpi NAME=sarah sh rpc.sh load_modules ");echo XXRETCODE:$?'
May 17 16:46:31 client-18 kernel: Lustre: 4323:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release.
May 17 16:46:31 client-18 kernel: Lustre: 4324:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release.
May 17 16:46:31 client-18 kernel: Lustre: OBD class driver, http://www.lustre.org/
May 17 16:46:31 client-18 kernel: Lustre:         Lustre Version: 2.0.60
May 17 16:46:31 client-18 kernel: Lustre:         Build Version: jenkins-g985ac5b-PRISTINE-2.6.18-194.17.1.el5
May 17 16:46:31 client-18 kernel: Lustre: Lustre LU module (ffffffff888eb1c0).
May 17 16:46:31 client-18 kernel: Lustre: Register global MR array, MR size: 0xffffffffffffffff, array size: 1
May 17 16:46:31 client-18 kernel: Lustre: Added LNI 192.168.4.18@o2ib [8/64/0/180]
May 17 16:46:31 client-18 kernel: Lustre: Added LNI 10.10.4.18@tcp [8/256/0/180]
May 17 16:46:31 client-18 kernel: Lustre: Accept all, port 7988
May 17 16:46:31 client-18 kernel: Lustre: Lustre OSC module (ffffffff88b3c6c0).
May 17 16:46:31 client-18 kernel: Lustre: Lustre LOV module (ffffffff88bab540).
May 17 16:46:31 client-18 kernel: Lustre: Lustre client module (ffffffff88c643a0).
May 17 16:46:31 client-18 xinetd[3390]: EXIT: shell status=0 pid=4218 duration=1(sec)
May 17 16:47:26 client-18 xinetd[3390]: START: shell pid=4678 from=192.168.4.5
May 17 16:47:26 client-18 rshd[4679]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/tests:/usr/lib64/openmpi/1.4-gcc/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/usr/lib64/lustre/utils:/usr/lib64/lustre/utils/gss:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey::/usr/lib64/lustre/tests/mpi NAME=sarah sh rpc.sh /usr/sbin/lustre_rmmod ldiskfs ");echo XXRETCODE:$?'
May 17 16:47:34 client-18 kernel: Lustre: Acceptor stopping
May 17 16:47:35 client-18 kernel: Lustre: Removed LNI 192.168.4.18@o2ib
May 17 16:47:36 client-18 kernel: Lustre: Removed LNI 10.10.4.18@tcp
May 17 16:47:37 client-18 xinetd[3390]: EXIT: shell status=0 pid=4678 duration=11(sec)
May 17 16:47:37 client-18 xinetd[3390]: START: shell pid=5072 from=192.168.4.5
May 17 16:47:37 client-18 rshd[5073]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/tests:/usr/lib64/openmpi/1.4-gcc/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/usr/lib64/lustre/utils:/usr/lib64/lustre/utils/gss:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey::/usr/lib64/lustre/tests/mpi NAME=sarah sh rpc.sh check_mem_leak ");echo XXRETCODE:$?'
May 17 16:47:37 client-18 xinetd[3390]: EXIT: shell status=0 pid=5072 duration=0(sec)
May 17 16:48:28 client-18 xinetd[3390]: START: shell pid=5165 from=192.168.4.5
May 17 16:48:28 client-18 rshd[5166]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 16:48:28 client-18 xinetd[3390]: EXIT: shell status=0 pid=5165 duration=0(sec)
May 17 16:48:28 client-18 xinetd[3390]: START: shell pid=5201 from=192.168.4.5
May 17 16:48:28 client-18 rshd[5202]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 16:48:29 client-18 xinetd[3390]: EXIT: shell status=0 pid=5201 duration=1(sec)
May 17 16:48:29 client-18 xinetd[3390]: START: shell pid=5237 from=192.168.4.5
May 17 16:48:29 client-18 rshd[5238]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == conf-sanity test 53b: check MDT thread count params =============================================== 16:48:29 \(1305676109\)");echo XXRETCODE:$?'
May 17 16:48:29 client-18 xinetd[3390]: EXIT: shell status=0 pid=5237 duration=0(sec)
May 17 16:49:05 client-18 xinetd[3390]: START: shell pid=5278 from=192.168.4.5
May 17 16:49:05 client-18 rshd[5279]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/tests:/usr/lib64/openmpi/1.4-gcc/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/usr/lib64/lustre/utils:/usr/lib64/lustre/utils/gss:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey::/usr/lib64/lustre/tests/mpi NAME=sarah sh rpc.sh /usr/sbin/lustre_rmmod ldiskfs ");echo XXRETCODE:$?'
May 17 16:49:05 client-18 xinetd[3390]: EXIT: shell status=0 pid=5278 duration=0(sec)
May 17 16:49:16 client-18 xinetd[3390]: START: shell pid=5379 from=192.168.4.5
May 17 16:49:16 client-18 rshd[5380]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/tests:/usr/lib64/openmpi/1.4-gcc/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/usr/lib64/lustre/utils:/usr/lib64/lustre/utils/gss:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey::/usr/lib64/lustre/tests/mpi NAME=sarah sh rpc.sh check_mem_leak ");echo XXRETCODE:$?'
May 17 16:49:16 client-18 xinetd[3390]: EXIT: shell status=0 pid=5379 duration=0(sec)
May 17 16:49:17 client-18 xinetd[3390]: START: shell pid=5471 from=192.168.4.5
May 17 16:49:17 client-18 rshd[5472]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" MODOPTS_MDT=" mdt_num_threads=64" sh -c "PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/tests:/usr/lib64/openmpi/1.4-gcc/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/usr/lib64/lustre/utils:/usr/lib64/lustre/utils/gss:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey::/usr/lib64/lustre/tests/mpi NAME=sarah sh rpc.sh load_modules ");echo XXRETCODE:$?'
May 17 16:49:17 client-18 kernel: Lustre: 5576:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release.
May 17 16:49:17 client-18 kernel: Lustre: 5579:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release.
May 17 16:49:17 client-18 kernel: Lustre: OBD class driver, http://www.lustre.org/
May 17 16:49:17 client-18 kernel: Lustre:         Lustre Version: 2.0.60
May 17 16:49:17 client-18 kernel: Lustre:         Build Version: jenkins-g985ac5b-PRISTINE-2.6.18-194.17.1.el5
May 17 16:49:17 client-18 kernel: Lustre: Lustre LU module (ffffffff888eb1c0).
May 17 16:49:17 client-18 kernel: Lustre: Register global MR array, MR size: 0xffffffffffffffff, array size: 1
May 17 16:49:17 client-18 kernel: Lustre: Added LNI 192.168.4.18@o2ib [8/64/0/180]
May 17 16:49:17 client-18 kernel: Lustre: Added LNI 10.10.4.18@tcp [8/256/0/180]
May 17 16:49:17 client-18 kernel: Lustre: Accept all, port 7988
May 17 16:49:17 client-18 kernel: Lustre: Lustre OSC module (ffffffff88b3c6c0).
May 17 16:49:17 client-18 kernel: Lustre: Lustre LOV module (ffffffff88bab540).
May 17 16:49:17 client-18 kernel: Lustre: Lustre client module (ffffffff88c643a0).
May 17 16:49:17 client-18 xinetd[3390]: EXIT: shell status=0 pid=5471 duration=0(sec)
May 17 16:50:11 client-18 xinetd[3390]: START: shell pid=5932 from=192.168.4.5
May 17 16:50:11 client-18 rshd[5933]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/tests:/usr/lib64/openmpi/1.4-gcc/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/usr/lib64/lustre/utils:/usr/lib64/lustre/utils/gss:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey::/usr/lib64/lustre/tests/mpi NAME=sarah sh rpc.sh /usr/sbin/lustre_rmmod ldiskfs ");echo XXRETCODE:$?'
May 17 16:50:18 client-18 kernel: Lustre: Acceptor stopping
May 17 16:50:19 client-18 kernel: Lustre: Removed LNI 192.168.4.18@o2ib
May 17 16:50:20 client-18 kernel: Lustre: Removed LNI 10.10.4.18@tcp
May 17 16:50:21 client-18 xinetd[3390]: EXIT: shell status=0 pid=5932 duration=10(sec)
May 17 16:50:21 client-18 xinetd[3390]: START: shell pid=6325 from=192.168.4.5
May 17 16:50:21 client-18 rshd[6326]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/tests:/usr/lib64/openmpi/1.4-gcc/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/usr/lib64/lustre/utils:/usr/lib64/lustre/utils/gss:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey::/usr/lib64/lustre/tests/mpi NAME=sarah sh rpc.sh check_mem_leak ");echo XXRETCODE:$?'
May 17 16:50:21 client-18 xinetd[3390]: EXIT: shell status=0 pid=6325 duration=0(sec)
May 17 16:51:12 client-18 xinetd[3390]: START: shell pid=6422 from=192.168.4.5
May 17 16:51:12 client-18 rshd[6423]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 16:51:12 client-18 xinetd[3390]: EXIT: shell status=0 pid=6422 duration=0(sec)
May 17 16:51:12 client-18 xinetd[3390]: START: shell pid=6458 from=192.168.4.5
May 17 16:51:12 client-18 rshd[6459]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 16:51:12 client-18 xinetd[3390]: EXIT: shell status=0 pid=6458 duration=0(sec)
May 17 16:51:12 client-18 xinetd[3390]: START: shell pid=6494 from=192.168.4.5
May 17 16:51:12 client-18 rshd[6495]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == conf-sanity test 54a: llverdev ==================================================================== 16:51:12 \(1305676272\)");echo XXRETCODE:$?'
May 17 16:51:13 client-18 xinetd[3390]: EXIT: shell status=0 pid=6494 duration=1(sec)
May 17 16:51:17 client-18 xinetd[3390]: START: shell pid=6531 from=192.168.4.5
May 17 16:51:17 client-18 rshd[6532]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:; lsof -t /mnt/lustre || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount  /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre is still busy, wait one second && sleep 1;     umount  /mnt/lustre; fi fi");echo XXRETCODE:$?'
May 17 16:51:17 client-18 xinetd[3390]: EXIT: shell status=0 pid=6531 duration=0(sec)
May 17 16:51:17 client-18 xinetd[3390]: START: shell pid=6568 from=192.168.4.5
May 17 16:51:17 client-18 rshd[6569]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:; lsof -t /mnt/lustre2 || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre2 | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount  /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre2 is still busy, wait one second && sleep 1;     umount  /mnt/lustre2; fi fi");echo XXRETCODE:$?'
May 17 16:51:18 client-18 xinetd[3390]: EXIT: shell status=0 pid=6568 duration=1(sec)
May 17 16:51:32 client-18 xinetd[3390]: START: shell pid=6605 from=192.168.4.5
May 17 16:51:32 client-18 rshd[6606]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark ost now in FULL state");echo XXRETCODE:$?'
May 17 16:51:32 client-18 xinetd[3390]: EXIT: shell status=0 pid=6605 duration=0(sec)
May 17 16:51:43 client-18 xinetd[3390]: START: shell pid=6642 from=192.168.4.5
May 17 16:51:43 client-18 rshd[6643]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 16:51:43 client-18 xinetd[3390]: EXIT: shell status=0 pid=6642 duration=0(sec)
May 17 16:51:43 client-18 xinetd[3390]: START: shell pid=6678 from=192.168.4.5
May 17 16:51:44 client-18 rshd[6679]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 16:51:44 client-18 xinetd[3390]: EXIT: shell status=0 pid=6678 duration=1(sec)
May 17 16:51:44 client-18 xinetd[3390]: START: shell pid=6714 from=192.168.4.5
May 17 16:51:44 client-18 rshd[6715]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == conf-sanity test 54b: llverfs ===================================================================== 16:51:44 \(1305676304\)");echo XXRETCODE:$?'
May 17 16:51:44 client-18 xinetd[3390]: EXIT: shell status=0 pid=6714 duration=0(sec)
May 17 16:52:28 client-18 xinetd[3390]: START: shell pid=6755 from=192.168.4.5
May 17 16:52:28 client-18 rshd[6756]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 16:52:28 client-18 xinetd[3390]: EXIT: shell status=0 pid=6755 duration=0(sec)
May 17 16:52:28 client-18 xinetd[3390]: START: shell pid=6791 from=192.168.4.5
May 17 16:52:28 client-18 rshd[6792]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 16:52:28 client-18 xinetd[3390]: EXIT: shell status=0 pid=6791 duration=0(sec)
May 17 16:52:28 client-18 xinetd[3390]: START: shell pid=6827 from=192.168.4.5
May 17 16:52:29 client-18 rshd[6828]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == conf-sanity test 55: check lov_objid size ========================================================= 16:52:28 \(1305676348\)");echo XXRETCODE:$?'
May 17 16:52:29 client-18 xinetd[3390]: EXIT: shell status=0 pid=6827 duration=1(sec)
May 17 16:52:29 client-18 xinetd[3390]: START: shell pid=6864 from=192.168.4.5
May 17 16:52:29 client-18 rshd[6865]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:; lsof -t /mnt/lustre || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount  /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre is still busy, wait one second && sleep 1;     umount  /mnt/lustre; fi fi");echo XXRETCODE:$?'
May 17 16:52:29 client-18 xinetd[3390]: EXIT: shell status=0 pid=6864 duration=0(sec)
May 17 16:52:29 client-18 xinetd[3390]: START: shell pid=6901 from=192.168.4.5
May 17 16:52:29 client-18 rshd[6902]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:; lsof -t /mnt/lustre2 || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre2 | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount  /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre2 is still busy, wait one second && sleep 1;     umount  /mnt/lustre2; fi fi");echo XXRETCODE:$?'
May 17 16:52:29 client-18 xinetd[3390]: EXIT: shell status=0 pid=6901 duration=0(sec)
May 17 16:52:42 client-18 xinetd[3390]: START: shell pid=6938 from=192.168.4.5
May 17 16:52:42 client-18 rshd[6939]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:; lsof -t /mnt/lustre || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount  /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre is still busy, wait one second && sleep 1;     umount  /mnt/lustre; fi fi");echo XXRETCODE:$?'
May 17 16:52:42 client-18 xinetd[3390]: EXIT: shell status=0 pid=6938 duration=0(sec)
May 17 16:52:42 client-18 xinetd[3390]: START: shell pid=6975 from=192.168.4.5
May 17 16:52:42 client-18 rshd[6976]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:; lsof -t /mnt/lustre2 || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre2 | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount  /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre2 is still busy, wait one second && sleep 1;     umount  /mnt/lustre2; fi fi");echo XXRETCODE:$?'
May 17 16:52:42 client-18 xinetd[3390]: EXIT: shell status=0 pid=6975 duration=0(sec)
May 17 16:53:10 client-18 xinetd[3390]: START: shell pid=7012 from=192.168.4.5
May 17 16:53:10 client-18 rshd[7013]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:; lsof -t /mnt/lustre || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount  /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre is still busy, wait one second && sleep 1;     umount  /mnt/lustre; fi fi");echo XXRETCODE:$?'
May 17 16:53:10 client-18 xinetd[3390]: EXIT: shell status=0 pid=7012 duration=0(sec)
May 17 16:53:10 client-18 xinetd[3390]: START: shell pid=7049 from=192.168.4.5
May 17 16:53:10 client-18 rshd[7050]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:; lsof -t /mnt/lustre2 || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre2 | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount  /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre2 is still busy, wait one second && sleep 1;     umount  /mnt/lustre2; fi fi");echo XXRETCODE:$?'
May 17 16:53:10 client-18 xinetd[3390]: EXIT: shell status=0 pid=7049 duration=0(sec)
May 17 16:53:26 client-18 xinetd[3390]: START: shell pid=7086 from=192.168.4.5
May 17 16:53:26 client-18 rshd[7087]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:; lsof -t /mnt/lustre || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount  /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre is still busy, wait one second && sleep 1;     umount  /mnt/lustre; fi fi");echo XXRETCODE:$?'
May 17 16:53:27 client-18 xinetd[3390]: EXIT: shell status=0 pid=7086 duration=1(sec)
May 17 16:53:27 client-18 xinetd[3390]: START: shell pid=7123 from=192.168.4.5
May 17 16:53:27 client-18 rshd[7124]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:; lsof -t /mnt/lustre2 || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre2 | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount  /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre2 is still busy, wait one second && sleep 1;     umount  /mnt/lustre2; fi fi");echo XXRETCODE:$?'
May 17 16:53:27 client-18 xinetd[3390]: EXIT: shell status=0 pid=7123 duration=0(sec)
May 17 16:53:40 client-18 xinetd[3390]: START: shell pid=7160 from=192.168.4.5
May 17 16:53:40 client-18 rshd[7161]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:; lsof -t /mnt/lustre || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount  /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre is still busy, wait one second && sleep 1;     umount  /mnt/lustre; fi fi");echo XXRETCODE:$?'
May 17 16:53:40 client-18 xinetd[3390]: EXIT: shell status=0 pid=7160 duration=0(sec)
May 17 16:53:40 client-18 xinetd[3390]: START: shell pid=7201 from=192.168.4.5
May 17 16:53:40 client-18 rshd[7202]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:; lsof -t /mnt/lustre2 || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre2 | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount  /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre2 is still busy, wait one second && sleep 1;     umount  /mnt/lustre2; fi fi");echo XXRETCODE:$?'
May 17 16:53:40 client-18 xinetd[3390]: EXIT: shell status=0 pid=7201 duration=0(sec)
May 17 16:54:08 client-18 xinetd[3390]: START: shell pid=7238 from=192.168.4.5
May 17 16:54:08 client-18 rshd[7239]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:; lsof -t /mnt/lustre || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount  /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre is still busy, wait one second && sleep 1;     umount  /mnt/lustre; fi fi");echo XXRETCODE:$?'
May 17 16:54:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=7238 duration=0(sec)
May 17 16:54:08 client-18 xinetd[3390]: START: shell pid=7275 from=192.168.4.5
May 17 16:54:08 client-18 rshd[7276]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:; lsof -t /mnt/lustre2 || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre2 | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount  /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre2 is still busy, wait one second && sleep 1;     umount  /mnt/lustre2; fi fi");echo XXRETCODE:$?'
May 17 16:54:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=7275 duration=0(sec)
May 17 16:54:24 client-18 xinetd[3390]: START: shell pid=7312 from=192.168.4.5
May 17 16:54:24 client-18 rshd[7313]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:; lsof -t /mnt/lustre || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount  /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre is still busy, wait one second && sleep 1;     umount  /mnt/lustre; fi fi");echo XXRETCODE:$?'
May 17 16:54:24 client-18 xinetd[3390]: EXIT: shell status=0 pid=7312 duration=0(sec)
May 17 16:54:24 client-18 xinetd[3390]: START: shell pid=7349 from=192.168.4.5
May 17 16:54:24 client-18 rshd[7350]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:; lsof -t /mnt/lustre2 || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre2 | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount  /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre2 is still busy, wait one second && sleep 1;     umount  /mnt/lustre2; fi fi");echo XXRETCODE:$?'
May 17 16:54:25 client-18 xinetd[3390]: EXIT: shell status=0 pid=7349 duration=1(sec)
May 17 16:54:38 client-18 xinetd[3390]: START: shell pid=7386 from=192.168.4.5
May 17 16:54:38 client-18 rshd[7387]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:; lsof -t /mnt/lustre || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount  /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre is still busy, wait one second && sleep 1;     umount  /mnt/lustre; fi fi");echo XXRETCODE:$?'
May 17 16:54:38 client-18 xinetd[3390]: EXIT: shell status=0 pid=7386 duration=0(sec)
May 17 16:54:38 client-18 xinetd[3390]: START: shell pid=7423 from=192.168.4.5
May 17 16:54:38 client-18 rshd[7424]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:; lsof -t /mnt/lustre2 || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre2 | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount  /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre2 is still busy, wait one second && sleep 1;     umount  /mnt/lustre2; fi fi");echo XXRETCODE:$?'
May 17 16:54:38 client-18 xinetd[3390]: EXIT: shell status=0 pid=7423 duration=0(sec)
May 17 16:55:01 client-18 xinetd[3390]: START: shell pid=7464 from=192.168.4.5
May 17 16:55:01 client-18 rshd[7465]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:; lsof -t /mnt/lustre || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount  /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre is still busy, wait one second && sleep 1;     umount  /mnt/lustre; fi fi");echo XXRETCODE:$?'
May 17 16:55:02 client-18 xinetd[3390]: EXIT: shell status=0 pid=7464 duration=1(sec)
May 17 16:55:02 client-18 xinetd[3390]: START: shell pid=7501 from=192.168.4.5
May 17 16:55:02 client-18 rshd[7502]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:; lsof -t /mnt/lustre2 || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre2 | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount  /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre2 is still busy, wait one second && sleep 1;     umount  /mnt/lustre2; fi fi");echo XXRETCODE:$?'
May 17 16:55:02 client-18 xinetd[3390]: EXIT: shell status=0 pid=7501 duration=0(sec)
May 17 16:55:18 client-18 xinetd[3390]: START: shell pid=7538 from=192.168.4.5
May 17 16:55:18 client-18 rshd[7539]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:; lsof -t /mnt/lustre || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount  /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre is still busy, wait one second && sleep 1;     umount  /mnt/lustre; fi fi");echo XXRETCODE:$?'
May 17 16:55:18 client-18 xinetd[3390]: EXIT: shell status=0 pid=7538 duration=0(sec)
May 17 16:55:18 client-18 xinetd[3390]: START: shell pid=7575 from=192.168.4.5
May 17 16:55:18 client-18 rshd[7576]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:; lsof -t /mnt/lustre2 || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre2 | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount  /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre2 is still busy, wait one second && sleep 1;     umount  /mnt/lustre2; fi fi");echo XXRETCODE:$?'
May 17 16:55:18 client-18 xinetd[3390]: EXIT: shell status=0 pid=7575 duration=0(sec)
May 17 16:55:25 client-18 xinetd[3390]: START: shell pid=7612 from=192.168.4.5
May 17 16:55:25 client-18 rshd[7613]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 16:55:25 client-18 xinetd[3390]: EXIT: shell status=0 pid=7612 duration=0(sec)
May 17 16:55:25 client-18 xinetd[3390]: START: shell pid=7648 from=192.168.4.5
May 17 16:55:25 client-18 rshd[7649]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 16:55:25 client-18 xinetd[3390]: EXIT: shell status=0 pid=7648 duration=0(sec)
May 17 16:55:25 client-18 xinetd[3390]: START: shell pid=7684 from=192.168.4.5
May 17 16:55:25 client-18 rshd[7685]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == conf-sanity test 56: check big indexes ============================================================ 16:55:25 \(1305676525\)");echo XXRETCODE:$?'
May 17 16:55:26 client-18 xinetd[3390]: EXIT: shell status=0 pid=7684 duration=1(sec)
May 17 16:55:35 client-18 xinetd[3390]: START: shell pid=7721 from=192.168.4.5
May 17 16:55:35 client-18 rshd[7722]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:; lsof -t /mnt/lustre || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount  /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre is still busy, wait one second && sleep 1;     umount  /mnt/lustre; fi fi");echo XXRETCODE:$?'
May 17 16:55:35 client-18 xinetd[3390]: EXIT: shell status=0 pid=7721 duration=0(sec)
May 17 16:55:35 client-18 xinetd[3390]: START: shell pid=7758 from=192.168.4.5
May 17 16:55:35 client-18 rshd[7759]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:; lsof -t /mnt/lustre2 || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre2 | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount  /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre2 is still busy, wait one second && sleep 1;     umount  /mnt/lustre2; fi fi");echo XXRETCODE:$?'
May 17 16:55:36 client-18 xinetd[3390]: EXIT: shell status=0 pid=7758 duration=1(sec)
May 17 16:55:53 client-18 xinetd[3390]: START: shell pid=7795 from=192.168.4.5
May 17 16:55:53 client-18 rshd[7796]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:; lsof -t /mnt/lustre || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount  /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre is still busy, wait one second && sleep 1;     umount  /mnt/lustre; fi fi");echo XXRETCODE:$?'
May 17 16:55:53 client-18 xinetd[3390]: EXIT: shell status=0 pid=7795 duration=0(sec)
May 17 16:55:53 client-18 xinetd[3390]: START: shell pid=7832 from=192.168.4.5
May 17 16:55:53 client-18 rshd[7833]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:; lsof -t /mnt/lustre2 || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre2 | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount  /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre2 is still busy, wait one second && sleep 1;     umount  /mnt/lustre2; fi fi");echo XXRETCODE:$?'
May 17 16:55:53 client-18 xinetd[3390]: EXIT: shell status=0 pid=7832 duration=0(sec)
May 17 16:56:00 client-18 xinetd[3390]: START: shell pid=7869 from=192.168.4.5
May 17 16:56:00 client-18 rshd[7870]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 16:56:00 client-18 xinetd[3390]: EXIT: shell status=0 pid=7869 duration=0(sec)
May 17 16:56:00 client-18 xinetd[3390]: START: shell pid=7905 from=192.168.4.5
May 17 16:56:00 client-18 rshd[7906]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 16:56:00 client-18 xinetd[3390]: EXIT: shell status=0 pid=7905 duration=0(sec)
May 17 16:56:00 client-18 xinetd[3390]: START: shell pid=7941 from=192.168.4.5
May 17 16:56:00 client-18 rshd[7942]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == conf-sanity test 57: initial registration from failnode should fail \(should return errs\) ========== 16:56:00 \(1305676560\)");echo XXRETCODE:$?'
May 17 16:56:00 client-18 xinetd[3390]: EXIT: shell status=0 pid=7941 duration=0(sec)
May 17 16:56:09 client-18 xinetd[3390]: START: shell pid=7978 from=192.168.4.5
May 17 16:56:09 client-18 rshd[7979]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:; lsof -t /mnt/lustre || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount  /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre is still busy, wait one second && sleep 1;     umount  /mnt/lustre; fi fi");echo XXRETCODE:$?'
May 17 16:56:09 client-18 xinetd[3390]: EXIT: shell status=0 pid=7978 duration=0(sec)
May 17 16:56:09 client-18 xinetd[3390]: START: shell pid=8015 from=192.168.4.5
May 17 16:56:09 client-18 rshd[8016]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:; lsof -t /mnt/lustre2 || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre2 | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount  /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre2 is still busy, wait one second && sleep 1;     umount  /mnt/lustre2; fi fi");echo XXRETCODE:$?'
May 17 16:56:09 client-18 xinetd[3390]: EXIT: shell status=0 pid=8015 duration=0(sec)
May 17 16:56:16 client-18 xinetd[3390]: START: shell pid=8056 from=192.168.4.5
May 17 16:56:16 client-18 rshd[8057]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 16:56:16 client-18 xinetd[3390]: EXIT: shell status=0 pid=8056 duration=0(sec)
May 17 16:56:16 client-18 xinetd[3390]: START: shell pid=8092 from=192.168.4.5
May 17 16:56:16 client-18 rshd[8093]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 16:56:16 client-18 xinetd[3390]: EXIT: shell status=0 pid=8092 duration=0(sec)
May 17 16:56:16 client-18 xinetd[3390]: START: shell pid=8128 from=192.168.4.5
May 17 16:56:16 client-18 rshd[8129]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == conf-sanity test 59: writeconf mount option ======================================================= 16:56:16 \(1305676576\)");echo XXRETCODE:$?'
May 17 16:56:17 client-18 xinetd[3390]: EXIT: shell status=0 pid=8128 duration=1(sec)
May 17 16:56:18 client-18 xinetd[3390]: START: shell pid=8165 from=192.168.4.5
May 17 16:56:18 client-18 rshd[8166]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:; lsof -t /mnt/lustre || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount  /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre is still busy, wait one second && sleep 1;     umount  /mnt/lustre; fi fi");echo XXRETCODE:$?'
May 17 16:56:18 client-18 xinetd[3390]: EXIT: shell status=0 pid=8165 duration=0(sec)
May 17 16:56:18 client-18 xinetd[3390]: START: shell pid=8202 from=192.168.4.5
May 17 16:56:18 client-18 rshd[8203]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:; lsof -t /mnt/lustre2 || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre2 | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount  /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre2 is still busy, wait one second && sleep 1;     umount  /mnt/lustre2; fi fi");echo XXRETCODE:$?'
May 17 16:56:18 client-18 xinetd[3390]: EXIT: shell status=0 pid=8202 duration=0(sec)
May 17 16:56:58 client-18 xinetd[3390]: START: shell pid=8239 from=192.168.4.5
May 17 16:56:58 client-18 rshd[8240]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 16:56:58 client-18 xinetd[3390]: EXIT: shell status=0 pid=8239 duration=0(sec)
May 17 16:56:58 client-18 xinetd[3390]: START: shell pid=8275 from=192.168.4.5
May 17 16:56:58 client-18 rshd[8276]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 16:56:58 client-18 xinetd[3390]: EXIT: shell status=0 pid=8275 duration=0(sec)
May 17 16:56:59 client-18 xinetd[3390]: START: shell pid=8311 from=192.168.4.5
May 17 16:56:59 client-18 rshd[8312]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == conf-sanity test 58: missing llog files must not prevent MDT from mounting ======================== 16:56:58 \(1305676618\)");echo XXRETCODE:$?'
May 17 16:56:59 client-18 xinetd[3390]: EXIT: shell status=0 pid=8311 duration=0(sec)
May 17 16:57:12 client-18 xinetd[3390]: START: shell pid=8348 from=192.168.4.5
May 17 16:57:12 client-18 rshd[8349]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:; lsof -t /mnt/lustre || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount  /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre is still busy, wait one second && sleep 1;     umount  /mnt/lustre; fi fi");echo XXRETCODE:$?'
May 17 16:57:12 client-18 xinetd[3390]: EXIT: shell status=0 pid=8348 duration=0(sec)
May 17 16:57:13 client-18 xinetd[3390]: START: shell pid=8385 from=192.168.4.5
May 17 16:57:13 client-18 rshd[8386]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:; lsof -t /mnt/lustre2 || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre2 | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount  /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre2 is still busy, wait one second && sleep 1;     umount  /mnt/lustre2; fi fi");echo XXRETCODE:$?'
May 17 16:57:13 client-18 xinetd[3390]: EXIT: shell status=0 pid=8385 duration=0(sec)
May 17 16:57:40 client-18 xinetd[3390]: START: shell pid=8426 from=192.168.4.5
May 17 16:57:40 client-18 rshd[8427]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 16:57:40 client-18 xinetd[3390]: EXIT: shell status=0 pid=8426 duration=0(sec)
May 17 16:57:40 client-18 xinetd[3390]: START: shell pid=8462 from=192.168.4.5
May 17 16:57:40 client-18 rshd[8463]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 16:57:40 client-18 xinetd[3390]: EXIT: shell status=0 pid=8462 duration=0(sec)
May 17 16:57:40 client-18 xinetd[3390]: START: shell pid=8498 from=192.168.4.5
May 17 16:57:40 client-18 rshd[8499]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == conf-sanity conf-sanity.sh test complete, duration 4292 sec ======================================= 16:57:40 \(1305676660\)");echo XXRETCODE:$?'
May 17 16:57:40 client-18 xinetd[3390]: EXIT: shell status=0 pid=8498 duration=0(sec)
May 17 16:57:40 client-18 xinetd[3390]: START: shell pid=8535 from=192.168.4.5
May 17 16:57:40 client-18 rshd[8536]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark -----============= acceptance-small: recovery-small ============----- Tue May 17 16:57:40 PDT 2011");echo XXRETCODE:$?'
May 17 16:57:40 client-18 xinetd[3390]: EXIT: shell status=0 pid=8535 duration=0(sec)
May 17 16:57:40 client-18 xinetd[3390]: START: shell pid=8572 from=192.168.4.5
May 17 16:57:40 client-18 rshd[8573]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark excepting tests: 24 52");echo XXRETCODE:$?'
May 17 16:57:40 client-18 xinetd[3390]: EXIT: shell status=0 pid=8572 duration=0(sec)
May 17 16:57:40 client-18 xinetd[3390]: START: shell pid=8609 from=192.168.4.5
May 17 16:57:41 client-18 rshd[8610]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:; lsof -t /mnt/lustre || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount  /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre is still busy, wait one second && sleep 1;     umount  /mnt/lustre; fi fi");echo XXRETCODE:$?'
May 17 16:57:41 client-18 xinetd[3390]: EXIT: shell status=0 pid=8609 duration=1(sec)
May 17 16:57:41 client-18 xinetd[3390]: START: shell pid=8646 from=192.168.4.5
May 17 16:57:41 client-18 rshd[8647]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:; lsof -t /mnt/lustre2 || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre2 | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount  /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre2 is still busy, wait one second && sleep 1;     umount  /mnt/lustre2; fi fi");echo XXRETCODE:$?'
May 17 16:57:41 client-18 xinetd[3390]: EXIT: shell status=0 pid=8646 duration=0(sec)
May 17 16:58:40 client-18 xinetd[3390]: START: shell pid=8688 from=192.168.4.5
May 17 16:58:40 client-18 rshd[8689]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); mpts=\$(mount | grep -w -c /mnt/lustre); if [ \$running -ne \$mpts ]; then     echo \$(hostname) env are INSANE!;     exit 1; fi");echo XXRETCODE:$?'
May 17 16:58:40 client-18 xinetd[3390]: EXIT: shell status=0 pid=8688 duration=0(sec)
May 17 16:58:40 client-18 xinetd[3390]: START: shell pid=8728 from=192.168.4.5
May 17 16:58:40 client-18 rshd[8729]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); mpts=\$(mount | grep -w -c /mnt/lustre2); if [ \$running -ne \$mpts ]; then     echo \$(hostname) env are INSANE!;     exit 1; fi");echo XXRETCODE:$?'
May 17 16:58:41 client-18 xinetd[3390]: EXIT: shell status=0 pid=8728 duration=1(sec)
May 17 16:58:50 client-18 xinetd[3390]: START: shell pid=8768 from=192.168.4.5
May 17 16:58:50 client-18 rshd[8769]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c " running=\$(mount | grep -c /mnt/lustre' '); rc=0; if [ \$running -eq 0 ] ; then     mkdir -p /mnt/lustre;     mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre;     rc=\$?; fi; exit \$rc");echo XXRETCODE:$?'
May 17 16:58:50 client-18 kernel: Lustre: OBD class driver, http://www.lustre.org/
May 17 16:58:50 client-18 kernel: Lustre:         Lustre Version: 2.0.60
May 17 16:58:50 client-18 kernel: Lustre:         Build Version: jenkins-g985ac5b-PRISTINE-2.6.18-194.17.1.el5
May 17 16:58:50 client-18 kernel: Lustre: Lustre LU module (ffffffff888c11c0).
May 17 16:58:50 client-18 kernel: Lustre: Register global MR array, MR size: 0xffffffffffffffff, array size: 1
May 17 16:58:50 client-18 kernel: Lustre: Added LNI 192.168.4.18@o2ib [8/64/0/180]
May 17 16:58:50 client-18 kernel: Lustre: Added LNI 10.10.4.18@tcp [8/256/0/180]
May 17 16:58:50 client-18 kernel: Lustre: Accept all, port 7988
May 17 16:58:50 client-18 kernel: Lustre: Lustre OSC module (ffffffff88b006c0).
May 17 16:58:50 client-18 kernel: Lustre: Lustre LOV module (ffffffff88b6f540).
May 17 16:58:50 client-18 kernel: Lustre: Lustre client module (ffffffff88c0f3a0).
May 17 16:58:50 client-18 kernel: Lustre: 8808:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null
May 17 16:58:50 client-18 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'.
May 17 16:58:50 client-18 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import
May 17 16:58:50 client-18 kernel: Lustre: 8808:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff81031dc9a800->192.168.4.128@o2ib netid 50000: select flavor null
May 17 16:58:50 client-18 kernel: Lustre: Client lustre-client has started
May 17 16:58:50 client-18 xinetd[3390]: EXIT: shell status=0 pid=8768 duration=0(sec)
May 17 16:58:50 client-18 xinetd[3390]: START: shell pid=8960 from=192.168.4.5
May 17 16:58:50 client-18 rshd[8961]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "mount | grep -w /mnt/lustre");echo XXRETCODE:$?'
May 17 16:58:51 client-18 xinetd[3390]: EXIT: shell status=0 pid=8960 duration=1(sec)
May 17 16:58:51 client-18 xinetd[3390]: START: shell pid=8997 from=192.168.4.5
May 17 16:58:51 client-18 rshd[8998]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param debug=\"-1\";         lctl set_param subsystem_debug=\"0xffb7e3ff\";         lctl set_param debug_mb=32;");echo XXRETCODE:$?'
May 17 16:58:51 client-18 kernel: Lustre: 9032:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release.
May 17 16:58:51 client-18 kernel: Lustre: 9033:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release.
May 17 16:58:51 client-18 xinetd[3390]: EXIT: shell status=0 pid=8997 duration=0(sec)
May 17 16:58:51 client-18 xinetd[3390]: START: shell pid=9034 from=192.168.4.5
May 17 16:58:51 client-18 rshd[9035]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark Using TIMEOUT=20");echo XXRETCODE:$?'
May 17 16:58:51 client-18 kernel: Lustre: DEBUG MARKER: Using TIMEOUT=20
May 17 16:58:51 client-18 xinetd[3390]: EXIT: shell status=0 pid=9034 duration=0(sec)
May 17 16:58:56 client-18 xinetd[3390]: START: shell pid=9069 from=192.168.4.5
May 17 16:58:56 client-18 rshd[9070]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "cat /proc/mounts");echo XXRETCODE:$?'
May 17 16:58:56 client-18 xinetd[3390]: EXIT: shell status=0 pid=9069 duration=0(sec)
May 17 16:58:57 client-18 xinetd[3390]: START: shell pid=9104 from=192.168.4.5
May 17 16:58:57 client-18 rshd[9105]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl get_param -n osc.*OST*-osc-[^M][^D][^T]*.srpc_info");echo XXRETCODE:$?'
May 17 16:58:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=9104 duration=0(sec)
May 17 16:58:57 client-18 xinetd[3390]: START: shell pid=9139 from=192.168.4.5
May 17 16:58:57 client-18 rshd[9140]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl get_param -n mdc.*-MDT*-mdc-*.srpc_info");echo XXRETCODE:$?'
May 17 16:58:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=9139 duration=0(sec)
May 17 16:58:57 client-18 xinetd[3390]: START: shell pid=9174 from=192.168.4.5
May 17 16:58:57 client-18 rshd[9175]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == recovery-small test 1: mcreate: drop req, drop rep ================================================ 16:58:57 \(1305676737\)");echo XXRETCODE:$?'
May 17 16:58:58 client-18 kernel: Lustre: DEBUG MARKER: == recovery-small test 1: mcreate: drop req, drop rep ================================================ 16:58:57 (1305676737)
May 17 16:58:58 client-18 xinetd[3390]: EXIT: shell status=0 pid=9174 duration=1(sec)
May 17 16:59:17 client-18 xinetd[3390]: START: shell pid=9209 from=192.168.4.5
May 17 16:59:17 client-18 rshd[9210]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 16:59:17 client-18 xinetd[3390]: EXIT: shell status=0 pid=9209 duration=0(sec)
May 17 16:59:17 client-18 xinetd[3390]: START: shell pid=9245 from=192.168.4.5
May 17 16:59:17 client-18 rshd[9246]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 16:59:17 client-18 xinetd[3390]: EXIT: shell status=0 pid=9245 duration=0(sec)
May 17 16:59:17 client-18 xinetd[3390]: START: shell pid=9282 from=192.168.4.5
May 17 16:59:18 client-18 rshd[9283]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == recovery-small test 2: chmod: drop req, drop rep ================================================== 16:59:17 \(1305676757\)");echo XXRETCODE:$?'
May 17 16:59:18 client-18 kernel: Lustre: DEBUG MARKER: == recovery-small test 2: chmod: drop req, drop rep ================================================== 16:59:17 (1305676757)
May 17 16:59:18 client-18 xinetd[3390]: EXIT: shell status=0 pid=9282 duration=1(sec)
May 17 16:59:32 client-18 xinetd[3390]: START: shell pid=9317 from=192.168.4.5
May 17 16:59:32 client-18 rshd[9318]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 16:59:32 client-18 xinetd[3390]: EXIT: shell status=0 pid=9317 duration=0(sec)
May 17 16:59:32 client-18 xinetd[3390]: START: shell pid=9353 from=192.168.4.5
May 17 16:59:32 client-18 rshd[9354]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 16:59:32 client-18 xinetd[3390]: EXIT: shell status=0 pid=9353 duration=0(sec)
May 17 16:59:33 client-18 xinetd[3390]: START: shell pid=9390 from=192.168.4.5
May 17 16:59:33 client-18 rshd[9391]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == recovery-small test 3: stat: drop req, drop rep =================================================== 16:59:33 \(1305676773\)");echo XXRETCODE:$?'
May 17 16:59:33 client-18 kernel: Lustre: DEBUG MARKER: == recovery-small test 3: stat: drop req, drop rep =================================================== 16:59:33 (1305676773)
May 17 16:59:33 client-18 xinetd[3390]: EXIT: shell status=0 pid=9390 duration=0(sec)
May 17 16:59:47 client-18 xinetd[3390]: START: shell pid=9425 from=192.168.4.5
May 17 16:59:47 client-18 rshd[9426]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 16:59:47 client-18 xinetd[3390]: EXIT: shell status=0 pid=9425 duration=0(sec)
May 17 16:59:47 client-18 xinetd[3390]: START: shell pid=9461 from=192.168.4.5
May 17 16:59:47 client-18 rshd[9462]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 16:59:48 client-18 xinetd[3390]: EXIT: shell status=0 pid=9461 duration=1(sec)
May 17 16:59:48 client-18 xinetd[3390]: START: shell pid=9498 from=192.168.4.5
May 17 16:59:48 client-18 rshd[9499]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == recovery-small test 4: open: drop req, drop rep =================================================== 16:59:48 \(1305676788\)");echo XXRETCODE:$?'
May 17 16:59:48 client-18 kernel: Lustre: DEBUG MARKER: == recovery-small test 4: open: drop req, drop rep =================================================== 16:59:48 (1305676788)
May 17 16:59:48 client-18 xinetd[3390]: EXIT: shell status=0 pid=9498 duration=0(sec)
May 17 17:00:03 client-18 xinetd[3390]: START: shell pid=9537 from=192.168.4.5
May 17 17:00:03 client-18 rshd[9538]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 17:00:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=9537 duration=0(sec)
May 17 17:00:03 client-18 xinetd[3390]: START: shell pid=9573 from=192.168.4.5
May 17 17:00:03 client-18 rshd[9574]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 17:00:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=9573 duration=0(sec)
May 17 17:00:03 client-18 xinetd[3390]: START: shell pid=9610 from=192.168.4.5
May 17 17:00:04 client-18 rshd[9611]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == recovery-small test 5: rename: drop req, drop rep ================================================= 17:00:03 \(1305676803\)");echo XXRETCODE:$?'
May 17 17:00:04 client-18 kernel: Lustre: DEBUG MARKER: == recovery-small test 5: rename: drop req, drop rep ================================================= 17:00:03 (1305676803)
May 17 17:00:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=9610 duration=1(sec)
May 17 17:00:18 client-18 xinetd[3390]: START: shell pid=9645 from=192.168.4.5
May 17 17:00:18 client-18 rshd[9646]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 17:00:18 client-18 xinetd[3390]: EXIT: shell status=0 pid=9645 duration=0(sec)
May 17 17:00:18 client-18 xinetd[3390]: START: shell pid=9681 from=192.168.4.5
May 17 17:00:18 client-18 rshd[9682]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 17:00:18 client-18 xinetd[3390]: EXIT: shell status=0 pid=9681 duration=0(sec)
May 17 17:00:18 client-18 xinetd[3390]: START: shell pid=9718 from=192.168.4.5
May 17 17:00:19 client-18 rshd[9719]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == recovery-small test 6: link: drop req, drop rep =================================================== 17:00:18 \(1305676818\)");echo XXRETCODE:$?'
May 17 17:00:19 client-18 kernel: Lustre: DEBUG MARKER: == recovery-small test 6: link: drop req, drop rep =================================================== 17:00:18 (1305676818)
May 17 17:00:19 client-18 xinetd[3390]: EXIT: shell status=0 pid=9718 duration=1(sec)
May 17 17:00:33 client-18 xinetd[3390]: START: shell pid=9753 from=192.168.4.5
May 17 17:00:33 client-18 rshd[9754]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 17:00:33 client-18 xinetd[3390]: EXIT: shell status=0 pid=9753 duration=0(sec)
May 17 17:00:33 client-18 xinetd[3390]: START: shell pid=9789 from=192.168.4.5
May 17 17:00:33 client-18 rshd[9790]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 17:00:33 client-18 xinetd[3390]: EXIT: shell status=0 pid=9789 duration=0(sec)
May 17 17:00:33 client-18 xinetd[3390]: START: shell pid=9826 from=192.168.4.5
May 17 17:00:33 client-18 rshd[9827]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == recovery-small test 7: unlink: drop req, drop rep ================================================= 17:00:33 \(1305676833\)");echo XXRETCODE:$?'
May 17 17:00:34 client-18 kernel: Lustre: DEBUG MARKER: == recovery-small test 7: unlink: drop req, drop rep ================================================= 17:00:33 (1305676833)
May 17 17:00:34 client-18 xinetd[3390]: EXIT: shell status=0 pid=9826 duration=1(sec)
May 17 17:00:48 client-18 xinetd[3390]: START: shell pid=9861 from=192.168.4.5
May 17 17:00:48 client-18 rshd[9862]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 17:00:48 client-18 xinetd[3390]: EXIT: shell status=0 pid=9861 duration=0(sec)
May 17 17:00:48 client-18 xinetd[3390]: START: shell pid=9897 from=192.168.4.5
May 17 17:00:48 client-18 rshd[9898]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 17:00:48 client-18 xinetd[3390]: EXIT: shell status=0 pid=9897 duration=0(sec)
May 17 17:00:48 client-18 xinetd[3390]: START: shell pid=9934 from=192.168.4.5
May 17 17:00:48 client-18 rshd[9935]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == recovery-small test 8: touch: drop rep \(bug 1423\) ================================================= 17:00:48 \(1305676848\)");echo XXRETCODE:$?'
May 17 17:00:48 client-18 kernel: Lustre: DEBUG MARKER: == recovery-small test 8: touch: drop rep (bug 1423) ================================================= 17:00:48 (1305676848)
May 17 17:00:48 client-18 xinetd[3390]: EXIT: shell status=0 pid=9934 duration=0(sec)
May 17 17:00:56 client-18 xinetd[3390]: START: shell pid=9969 from=192.168.4.5
May 17 17:00:56 client-18 rshd[9970]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 17:00:56 client-18 xinetd[3390]: EXIT: shell status=0 pid=9969 duration=0(sec)
May 17 17:00:56 client-18 xinetd[3390]: START: shell pid=10005 from=192.168.4.5
May 17 17:00:56 client-18 rshd[10006]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 17:00:56 client-18 xinetd[3390]: EXIT: shell status=0 pid=10005 duration=0(sec)
May 17 17:00:56 client-18 xinetd[3390]: START: shell pid=10042 from=192.168.4.5
May 17 17:00:56 client-18 rshd[10043]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == recovery-small test 9: pause bulk on OST \(bug 1420\) =============================================== 17:00:56 \(1305676856\)");echo XXRETCODE:$?'
May 17 17:00:56 client-18 kernel: Lustre: DEBUG MARKER: == recovery-small test 9: pause bulk on OST (bug 1420) =============================================== 17:00:56 (1305676856)
May 17 17:00:56 client-18 xinetd[3390]: EXIT: shell status=0 pid=10042 duration=0(sec)
May 17 17:01:02 client-18 xinetd[3390]: START: shell pid=10083 from=192.168.4.5
May 17 17:01:02 client-18 rshd[10084]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 17:01:02 client-18 xinetd[3390]: EXIT: shell status=0 pid=10083 duration=0(sec)
May 17 17:01:02 client-18 xinetd[3390]: START: shell pid=10119 from=192.168.4.5
May 17 17:01:02 client-18 rshd[10120]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 17:01:02 client-18 xinetd[3390]: EXIT: shell status=0 pid=10119 duration=0(sec)
May 17 17:01:02 client-18 xinetd[3390]: START: shell pid=10156 from=192.168.4.5
May 17 17:01:02 client-18 rshd[10157]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == recovery-small test 10: finish request on server after client eviction \(bug 1521\) ================= 17:01:02 \(1305676862\)");echo XXRETCODE:$?'
May 17 17:01:02 client-18 kernel: Lustre: DEBUG MARKER: == recovery-small test 10: finish request on server after client eviction (bug 1521) ================= 17:01:02 (1305676862)
May 17 17:01:02 client-18 xinetd[3390]: EXIT: shell status=0 pid=10156 duration=0(sec)
May 17 17:01:13 client-18 xinetd[3390]: START: shell pid=10195 from=192.168.4.5
May 17 17:01:13 client-18 rshd[10196]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 17:01:13 client-18 xinetd[3390]: EXIT: shell status=0 pid=10195 duration=0(sec)
May 17 17:01:13 client-18 xinetd[3390]: START: shell pid=10231 from=192.168.4.5
May 17 17:01:13 client-18 rshd[10232]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 17:01:13 client-18 xinetd[3390]: EXIT: shell status=0 pid=10231 duration=0(sec)
May 17 17:01:14 client-18 xinetd[3390]: START: shell pid=10268 from=192.168.4.5
May 17 17:01:14 client-18 rshd[10269]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == recovery-small test 11: wake up a thread waiting for completion after eviction \(b=2460\) =========== 17:01:14 \(1305676874\)");echo XXRETCODE:$?'
May 17 17:01:14 client-18 kernel: Lustre: DEBUG MARKER: == recovery-small test 11: wake up a thread waiting for completion after eviction (b=2460) =========== 17:01:14 (1305676874)
May 17 17:01:14 client-18 xinetd[3390]: EXIT: shell status=0 pid=10268 duration=0(sec)
May 17 17:01:14 client-18 xinetd[3390]: START: shell pid=10303 from=192.168.4.5
May 17 17:01:14 client-18 rshd[10304]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 17:01:14 client-18 xinetd[3390]: EXIT: shell status=0 pid=10303 duration=0(sec)
May 17 17:01:14 client-18 xinetd[3390]: START: shell pid=10339 from=192.168.4.5
May 17 17:01:14 client-18 rshd[10340]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 17:01:14 client-18 xinetd[3390]: EXIT: shell status=0 pid=10339 duration=0(sec)
May 17 17:01:14 client-18 xinetd[3390]: START: shell pid=10376 from=192.168.4.5
May 17 17:01:14 client-18 rshd[10377]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == recovery-small test 12: recover from timed out resend in ptlrpcd \(b=2494\) ========================= 17:01:14 \(1305676874\)");echo XXRETCODE:$?'
May 17 17:01:14 client-18 kernel: Lustre: DEBUG MARKER: == recovery-small test 12: recover from timed out resend in ptlrpcd (b=2494) ========================= 17:01:14 (1305676874)
May 17 17:01:14 client-18 xinetd[3390]: EXIT: shell status=0 pid=10376 duration=0(sec)
May 17 17:01:22 client-18 xinetd[3390]: START: shell pid=10411 from=192.168.4.5
May 17 17:01:22 client-18 rshd[10412]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 17:01:22 client-18 xinetd[3390]: EXIT: shell status=0 pid=10411 duration=0(sec)
May 17 17:01:22 client-18 xinetd[3390]: START: shell pid=10447 from=192.168.4.5
May 17 17:01:22 client-18 rshd[10448]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 17:01:22 client-18 xinetd[3390]: EXIT: shell status=0 pid=10447 duration=0(sec)
May 17 17:01:55 client-18 xinetd[3390]: START: shell pid=10484 from=192.168.4.5
May 17 17:01:55 client-18 rshd[10485]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == recovery-small test 13: mdc_readpage restart test \(bug 1138\) ====================================== 17:01:55 \(1305676915\)");echo XXRETCODE:$?'
May 17 17:01:55 client-18 kernel: Lustre: DEBUG MARKER: == recovery-small test 13: mdc_readpage restart test (bug 1138) ====================================== 17:01:55 (1305676915)
May 17 17:01:55 client-18 xinetd[3390]: EXIT: shell status=0 pid=10484 duration=0(sec)
May 17 17:02:02 client-18 xinetd[3390]: START: shell pid=10519 from=192.168.4.5
May 17 17:02:02 client-18 rshd[10520]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 17:02:02 client-18 xinetd[3390]: EXIT: shell status=0 pid=10519 duration=0(sec)
May 17 17:02:02 client-18 xinetd[3390]: START: shell pid=10555 from=192.168.4.5
May 17 17:02:02 client-18 rshd[10556]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 17:02:02 client-18 xinetd[3390]: EXIT: shell status=0 pid=10555 duration=0(sec)
May 17 17:02:02 client-18 xinetd[3390]: START: shell pid=10592 from=192.168.4.5
May 17 17:02:03 client-18 rshd[10593]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == recovery-small test 14: mdc_readpage resend test \(bug 1138\) ======================================= 17:02:02 \(1305676922\)");echo XXRETCODE:$?'
May 17 17:02:03 client-18 kernel: Lustre: DEBUG MARKER: == recovery-small test 14: mdc_readpage resend test (bug 1138) ======================================= 17:02:02 (1305676922)
May 17 17:02:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=10592 duration=1(sec)
May 17 17:02:03 client-18 xinetd[3390]: START: shell pid=10627 from=192.168.4.5
May 17 17:02:03 client-18 rshd[10628]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 17:02:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=10627 duration=0(sec)
May 17 17:02:03 client-18 xinetd[3390]: START: shell pid=10663 from=192.168.4.5
May 17 17:02:03 client-18 rshd[10664]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 17:02:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=10663 duration=0(sec)
May 17 17:02:03 client-18 xinetd[3390]: START: shell pid=10700 from=192.168.4.5
May 17 17:02:03 client-18 rshd[10701]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == recovery-small test 15: failed open \(-ENOMEM\) ===================================================== 17:02:03 \(1305676923\)");echo XXRETCODE:$?'
May 17 17:02:03 client-18 kernel: Lustre: DEBUG MARKER: == recovery-small test 15: failed open (-ENOMEM) ===================================================== 17:02:03 (1305676923)
May 17 17:02:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=10700 duration=0(sec)
May 17 17:02:03 client-18 xinetd[3390]: START: shell pid=10735 from=192.168.4.5
May 17 17:02:03 client-18 rshd[10736]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 17:02:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=10735 duration=1(sec)
May 17 17:02:04 client-18 xinetd[3390]: START: shell pid=10771 from=192.168.4.5
May 17 17:02:04 client-18 rshd[10772]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 17:02:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=10771 duration=0(sec)
May 17 17:02:04 client-18 xinetd[3390]: START: shell pid=10808 from=192.168.4.5
May 17 17:02:04 client-18 rshd[10809]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == recovery-small test 16: timeout bulk put, don\'t evict client \(2732\) =============================== 17:02:04 \(1305676924\)");echo XXRETCODE:$?'
May 17 17:02:04 client-18 kernel: Lustre: DEBUG MARKER: == recovery-small test 16: timeout bulk put, don't evict client (2732) =============================== 17:02:04 (1305676924)
May 17 17:02:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=10808 duration=0(sec)
May 17 17:02:38 client-18 xinetd[3390]: START: shell pid=10847 from=192.168.4.5
May 17 17:02:38 client-18 rshd[10848]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 17:02:38 client-18 xinetd[3390]: EXIT: shell status=0 pid=10847 duration=0(sec)
May 17 17:02:38 client-18 xinetd[3390]: START: shell pid=10883 from=192.168.4.5
May 17 17:02:38 client-18 rshd[10884]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 17:02:38 client-18 xinetd[3390]: EXIT: shell status=0 pid=10883 duration=0(sec)
May 17 17:02:38 client-18 xinetd[3390]: START: shell pid=10920 from=192.168.4.5
May 17 17:02:38 client-18 rshd[10921]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == recovery-small test 17: timeout bulk get, don\'t evict client \(2732\) =============================== 17:02:38 \(1305676958\)");echo XXRETCODE:$?'
May 17 17:02:38 client-18 kernel: Lustre: DEBUG MARKER: == recovery-small test 17: timeout bulk get, don't evict client (2732) =============================== 17:02:38 (1305676958)
May 17 17:02:38 client-18 xinetd[3390]: EXIT: shell status=0 pid=10920 duration=0(sec)
May 17 17:03:19 client-18 xinetd[3390]: START: shell pid=10955 from=192.168.4.5
May 17 17:03:19 client-18 rshd[10956]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 17:03:19 client-18 xinetd[3390]: EXIT: shell status=0 pid=10955 duration=0(sec)
May 17 17:03:20 client-18 xinetd[3390]: START: shell pid=10991 from=192.168.4.5
May 17 17:03:20 client-18 rshd[10992]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 17:03:20 client-18 xinetd[3390]: EXIT: shell status=0 pid=10991 duration=0(sec)
May 17 17:03:20 client-18 xinetd[3390]: START: shell pid=11028 from=192.168.4.5
May 17 17:03:20 client-18 rshd[11029]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == recovery-small test 18a: manual ost invalidate clears page cache immediately ====================== 17:03:20 \(1305677000\)");echo XXRETCODE:$?'
May 17 17:03:20 client-18 kernel: Lustre: DEBUG MARKER: == recovery-small test 18a: manual ost invalidate clears page cache immediately ====================== 17:03:20 (1305677000)
May 17 17:03:20 client-18 xinetd[3390]: EXIT: shell status=0 pid=11028 duration=0(sec)
May 17 17:03:22 client-18 xinetd[3390]: START: shell pid=11063 from=192.168.4.5
May 17 17:03:22 client-18 rshd[11064]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 17:03:22 client-18 xinetd[3390]: EXIT: shell status=0 pid=11063 duration=0(sec)
May 17 17:03:22 client-18 xinetd[3390]: START: shell pid=11099 from=192.168.4.5
May 17 17:03:22 client-18 rshd[11100]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 17:03:22 client-18 xinetd[3390]: EXIT: shell status=0 pid=11099 duration=0(sec)
May 17 17:03:22 client-18 xinetd[3390]: START: shell pid=11136 from=192.168.4.5
May 17 17:03:22 client-18 rshd[11137]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == recovery-small test 18b: eviction and reconnect clears page cache \(2766\) ========================== 17:03:22 \(1305677002\)");echo XXRETCODE:$?'
May 17 17:03:22 client-18 kernel: Lustre: DEBUG MARKER: == recovery-small test 18b: eviction and reconnect clears page cache (2766) ========================== 17:03:22 (1305677002)
May 17 17:03:22 client-18 xinetd[3390]: EXIT: shell status=0 pid=11136 duration=0(sec)
May 17 17:03:45 client-18 xinetd[3390]: START: shell pid=11175 from=192.168.4.5
May 17 17:03:45 client-18 rshd[11176]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 17:03:46 client-18 xinetd[3390]: EXIT: shell status=0 pid=11175 duration=1(sec)
May 17 17:03:46 client-18 xinetd[3390]: START: shell pid=11211 from=192.168.4.5
May 17 17:03:46 client-18 rshd[11212]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 17:03:46 client-18 xinetd[3390]: EXIT: shell status=0 pid=11211 duration=0(sec)
May 17 17:03:46 client-18 xinetd[3390]: START: shell pid=11248 from=192.168.4.5
May 17 17:03:46 client-18 rshd[11249]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == recovery-small test 18c: Dropped connect reply after eviction handing \(14755\) ===================== 17:03:46 \(1305677026\)");echo XXRETCODE:$?'
May 17 17:03:46 client-18 kernel: Lustre: DEBUG MARKER: == recovery-small test 18c: Dropped connect reply after eviction handing (14755) ===================== 17:03:46 (1305677026)
May 17 17:03:46 client-18 xinetd[3390]: EXIT: shell status=0 pid=11248 duration=0(sec)
May 17 17:03:59 client-18 xinetd[3390]: START: shell pid=11283 from=192.168.4.5
May 17 17:03:59 client-18 rshd[11284]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 17:03:59 client-18 xinetd[3390]: EXIT: shell status=0 pid=11283 duration=0(sec)
May 17 17:03:59 client-18 xinetd[3390]: START: shell pid=11319 from=192.168.4.5
May 17 17:03:59 client-18 rshd[11320]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 17:03:59 client-18 xinetd[3390]: EXIT: shell status=0 pid=11319 duration=0(sec)
May 17 17:03:59 client-18 xinetd[3390]: START: shell pid=11356 from=192.168.4.5
May 17 17:03:59 client-18 rshd[11357]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == recovery-small test 19a: test expired_lock_main on mds \(2867\) ===================================== 17:03:59 \(1305677039\)");echo XXRETCODE:$?'
May 17 17:04:00 client-18 kernel: Lustre: DEBUG MARKER: == recovery-small test 19a: test expired_lock_main on mds (2867) ===================================== 17:03:59 (1305677039)
May 17 17:04:00 client-18 xinetd[3390]: EXIT: shell status=0 pid=11356 duration=1(sec)
May 17 17:04:05 client-18 xinetd[3390]: START: shell pid=11391 from=192.168.4.5
May 17 17:04:05 client-18 rshd[11392]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 17:04:05 client-18 xinetd[3390]: EXIT: shell status=0 pid=11391 duration=0(sec)
May 17 17:04:05 client-18 xinetd[3390]: START: shell pid=11427 from=192.168.4.5
May 17 17:04:05 client-18 rshd[11428]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 17:04:05 client-18 xinetd[3390]: EXIT: shell status=0 pid=11427 duration=0(sec)
May 17 17:04:05 client-18 xinetd[3390]: START: shell pid=11464 from=192.168.4.5
May 17 17:04:05 client-18 rshd[11465]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == recovery-small test 19b: test expired_lock_main on ost \(2867\) ===================================== 17:04:05 \(1305677045\)");echo XXRETCODE:$?'
May 17 17:04:05 client-18 kernel: Lustre: DEBUG MARKER: == recovery-small test 19b: test expired_lock_main on ost (2867) ===================================== 17:04:05 (1305677045)
May 17 17:04:05 client-18 xinetd[3390]: EXIT: shell status=0 pid=11464 duration=0(sec)
May 17 17:04:05 client-18 xinetd[3390]: START: shell pid=11499 from=192.168.4.5
May 17 17:04:05 client-18 rshd[11500]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 17:04:05 client-18 xinetd[3390]: EXIT: shell status=0 pid=11499 duration=0(sec)
May 17 17:04:05 client-18 xinetd[3390]: START: shell pid=11535 from=192.168.4.5
May 17 17:04:05 client-18 rshd[11536]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 17:04:05 client-18 xinetd[3390]: EXIT: shell status=0 pid=11535 duration=0(sec)
May 17 17:04:06 client-18 xinetd[3390]: START: shell pid=11572 from=192.168.4.5
May 17 17:04:06 client-18 rshd[11573]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == recovery-small test 20a: ldlm_handle_enqueue error \(should return error\) ========================== 17:04:06 \(1305677046\)");echo XXRETCODE:$?'
May 17 17:04:06 client-18 kernel: Lustre: DEBUG MARKER: == recovery-small test 20a: ldlm_handle_enqueue error (should return error) ========================== 17:04:06 (1305677046)
May 17 17:04:06 client-18 xinetd[3390]: EXIT: shell status=0 pid=11572 duration=0(sec)
May 17 17:04:06 client-18 xinetd[3390]: START: shell pid=11607 from=192.168.4.5
May 17 17:04:06 client-18 rshd[11608]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 17:04:06 client-18 xinetd[3390]: EXIT: shell status=0 pid=11607 duration=0(sec)
May 17 17:04:06 client-18 xinetd[3390]: START: shell pid=11643 from=192.168.4.5
May 17 17:04:06 client-18 rshd[11644]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 17:04:06 client-18 xinetd[3390]: EXIT: shell status=0 pid=11643 duration=0(sec)
May 17 17:04:06 client-18 xinetd[3390]: START: shell pid=11680 from=192.168.4.5
May 17 17:04:06 client-18 rshd[11681]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == recovery-small test 20b: ldlm_handle_enqueue error \(should return error\) ========================== 17:04:06 \(1305677046\)");echo XXRETCODE:$?'
May 17 17:04:06 client-18 kernel: Lustre: DEBUG MARKER: == recovery-small test 20b: ldlm_handle_enqueue error (should return error) ========================== 17:04:06 (1305677046)
May 17 17:04:06 client-18 xinetd[3390]: EXIT: shell status=0 pid=11680 duration=0(sec)
May 17 17:04:07 client-18 xinetd[3390]: START: shell pid=11715 from=192.168.4.5
May 17 17:04:07 client-18 rshd[11716]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 17:04:07 client-18 xinetd[3390]: EXIT: shell status=0 pid=11715 duration=0(sec)
May 17 17:04:07 client-18 xinetd[3390]: START: shell pid=11751 from=192.168.4.5
May 17 17:04:07 client-18 rshd[11752]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 17:04:07 client-18 xinetd[3390]: EXIT: shell status=0 pid=11751 duration=0(sec)
May 17 17:04:07 client-18 xinetd[3390]: START: shell pid=11788 from=192.168.4.5
May 17 17:04:07 client-18 rshd[11789]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == recovery-small test 21a: drop close request while close and open are both in flight =============== 17:04:07 \(1305677047\)");echo XXRETCODE:$?'
May 17 17:04:07 client-18 kernel: Lustre: DEBUG MARKER: == recovery-small test 21a: drop close request while close and open are both in flight =============== 17:04:07 (1305677047)
May 17 17:04:07 client-18 xinetd[3390]: EXIT: shell status=0 pid=11788 duration=0(sec)
May 17 17:04:16 client-18 xinetd[3390]: START: shell pid=11823 from=192.168.4.5
May 17 17:04:16 client-18 rshd[11824]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 17:04:16 client-18 xinetd[3390]: EXIT: shell status=0 pid=11823 duration=0(sec)
May 17 17:04:16 client-18 xinetd[3390]: START: shell pid=11859 from=192.168.4.5
May 17 17:04:16 client-18 rshd[11860]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 17:04:16 client-18 xinetd[3390]: EXIT: shell status=0 pid=11859 duration=0(sec)
May 17 17:04:16 client-18 xinetd[3390]: START: shell pid=11896 from=192.168.4.5
May 17 17:04:16 client-18 rshd[11897]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == recovery-small test 21b: drop open request while close and open are both in flight ================ 17:04:16 \(1305677056\)");echo XXRETCODE:$?'
May 17 17:04:16 client-18 kernel: Lustre: DEBUG MARKER: == recovery-small test 21b: drop open request while close and open are both in flight ================ 17:04:16 (1305677056)
May 17 17:04:16 client-18 xinetd[3390]: EXIT: shell status=0 pid=11896 duration=0(sec)
May 17 17:05:00 client-18 xinetd[3390]: START: shell pid=11935 from=192.168.4.5
May 17 17:05:00 client-18 rshd[11936]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 17:05:00 client-18 xinetd[3390]: EXIT: shell status=0 pid=11935 duration=0(sec)
May 17 17:05:00 client-18 xinetd[3390]: START: shell pid=11971 from=192.168.4.5
May 17 17:05:00 client-18 rshd[11972]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 17:05:01 client-18 xinetd[3390]: EXIT: shell status=0 pid=11971 duration=1(sec)
May 17 17:05:01 client-18 xinetd[3390]: START: shell pid=12008 from=192.168.4.5
May 17 17:05:01 client-18 rshd[12009]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == recovery-small test 21c: drop both request while close and open are both in flight ================ 17:05:01 \(1305677101\)");echo XXRETCODE:$?'
May 17 17:05:01 client-18 kernel: Lustre: DEBUG MARKER: == recovery-small test 21c: drop both request while close and open are both in flight ================ 17:05:01 (1305677101)
May 17 17:05:01 client-18 xinetd[3390]: EXIT: shell status=0 pid=12008 duration=0(sec)
May 17 17:05:11 client-18 xinetd[3390]: START: shell pid=12043 from=192.168.4.5
May 17 17:05:11 client-18 rshd[12044]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 17:05:11 client-18 xinetd[3390]: EXIT: shell status=0 pid=12043 duration=0(sec)
May 17 17:05:11 client-18 xinetd[3390]: START: shell pid=12079 from=192.168.4.5
May 17 17:05:11 client-18 rshd[12080]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 17:05:11 client-18 xinetd[3390]: EXIT: shell status=0 pid=12079 duration=0(sec)
May 17 17:05:12 client-18 xinetd[3390]: START: shell pid=12116 from=192.168.4.5
May 17 17:05:12 client-18 rshd[12117]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == recovery-small test 21d: drop close reply while close and open are both in flight ================= 17:05:12 \(1305677112\)");echo XXRETCODE:$?'
May 17 17:05:12 client-18 kernel: Lustre: DEBUG MARKER: == recovery-small test 21d: drop close reply while close and open are both in flight ================= 17:05:12 (1305677112)
May 17 17:05:12 client-18 xinetd[3390]: EXIT: shell status=0 pid=12116 duration=0(sec)
May 17 17:05:20 client-18 xinetd[3390]: START: shell pid=12151 from=192.168.4.5
May 17 17:05:20 client-18 rshd[12152]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 17:05:20 client-18 xinetd[3390]: EXIT: shell status=0 pid=12151 duration=0(sec)
May 17 17:05:20 client-18 xinetd[3390]: START: shell pid=12187 from=192.168.4.5
May 17 17:05:20 client-18 rshd[12188]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 17:05:20 client-18 xinetd[3390]: EXIT: shell status=0 pid=12187 duration=0(sec)
May 17 17:05:21 client-18 xinetd[3390]: START: shell pid=12224 from=192.168.4.5
May 17 17:05:21 client-18 rshd[12225]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == recovery-small test 21e: drop open reply while close and open are both in flight ================== 17:05:20 \(1305677120\)");echo XXRETCODE:$?'
May 17 17:05:21 client-18 kernel: Lustre: DEBUG MARKER: == recovery-small test 21e: drop open reply while close and open are both in flight ================== 17:05:20 (1305677120)
May 17 17:05:21 client-18 xinetd[3390]: EXIT: shell status=0 pid=12224 duration=0(sec)
May 17 17:06:05 client-18 xinetd[3390]: START: shell pid=12259 from=192.168.4.5
May 17 17:06:05 client-18 rshd[12260]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 17:06:05 client-18 xinetd[3390]: EXIT: shell status=0 pid=12259 duration=0(sec)
May 17 17:06:05 client-18 xinetd[3390]: START: shell pid=12295 from=192.168.4.5
May 17 17:06:05 client-18 rshd[12296]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 17:06:05 client-18 xinetd[3390]: EXIT: shell status=0 pid=12295 duration=0(sec)
May 17 17:06:05 client-18 xinetd[3390]: START: shell pid=12332 from=192.168.4.5
May 17 17:06:05 client-18 rshd[12333]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == recovery-small test 21f: drop both reply while close and open are both in flight ================== 17:06:05 \(1305677165\)");echo XXRETCODE:$?'
May 17 17:06:05 client-18 kernel: Lustre: DEBUG MARKER: == recovery-small test 21f: drop both reply while close and open are both in flight ================== 17:06:05 (1305677165)
May 17 17:06:05 client-18 xinetd[3390]: EXIT: shell status=0 pid=12332 duration=0(sec)
May 17 17:06:14 client-18 xinetd[3390]: START: shell pid=12371 from=192.168.4.5
May 17 17:06:14 client-18 rshd[12372]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 17:06:14 client-18 xinetd[3390]: EXIT: shell status=0 pid=12371 duration=0(sec)
May 17 17:06:14 client-18 xinetd[3390]: START: shell pid=12407 from=192.168.4.5
May 17 17:06:14 client-18 rshd[12408]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 17:06:14 client-18 xinetd[3390]: EXIT: shell status=0 pid=12407 duration=0(sec)
May 17 17:06:14 client-18 xinetd[3390]: START: shell pid=12444 from=192.168.4.5
May 17 17:06:14 client-18 rshd[12445]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == recovery-small test 21g: drop open reply and close request while close and open are both in flight ====================================================================================================== 17:06:14 \(1305677174\)");echo XXRETCODE:$?'
May 17 17:06:14 client-18 kernel: Lustre: DEBUG MARKER: == recovery-small test 21g: drop open reply and close request while close and open are both in flight ====================================================================================================== 17:06:14 (1305677174)
May 17 17:06:14 client-18 xinetd[3390]: EXIT: shell status=0 pid=12444 duration=0(sec)
May 17 17:06:23 client-18 xinetd[3390]: START: shell pid=12479 from=192.168.4.5
May 17 17:06:23 client-18 rshd[12480]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 17:06:23 client-18 xinetd[3390]: EXIT: shell status=0 pid=12479 duration=0(sec)
May 17 17:06:23 client-18 xinetd[3390]: START: shell pid=12515 from=192.168.4.5
May 17 17:06:23 client-18 rshd[12516]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 17:06:23 client-18 xinetd[3390]: EXIT: shell status=0 pid=12515 duration=0(sec)
May 17 17:06:23 client-18 xinetd[3390]: START: shell pid=12552 from=192.168.4.5
May 17 17:06:23 client-18 rshd[12553]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == recovery-small test 21h: drop open request and close reply while close and open are both in flight ====================================================================================================== 17:06:23 \(1305677183\)");echo XXRETCODE:$?'
May 17 17:06:23 client-18 kernel: Lustre: DEBUG MARKER: == recovery-small test 21h: drop open request and close reply while close and open are both in flight ====================================================================================================== 17:06:23 (1305677183)
May 17 17:06:23 client-18 xinetd[3390]: EXIT: shell status=0 pid=12552 duration=0(sec)
May 17 17:06:32 client-18 xinetd[3390]: START: shell pid=12587 from=192.168.4.5
May 17 17:06:32 client-18 rshd[12588]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 17:06:32 client-18 xinetd[3390]: EXIT: shell status=0 pid=12587 duration=0(sec)
May 17 17:06:32 client-18 xinetd[3390]: START: shell pid=12623 from=192.168.4.5
May 17 17:06:32 client-18 rshd[12624]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 17:06:32 client-18 xinetd[3390]: EXIT: shell status=0 pid=12623 duration=0(sec)
May 17 17:06:32 client-18 xinetd[3390]: START: shell pid=12660 from=192.168.4.5
May 17 17:06:32 client-18 rshd[12661]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == recovery-small test 22: drop close request and do mknod =========================================== 17:06:32 \(1305677192\)");echo XXRETCODE:$?'
May 17 17:06:32 client-18 kernel: Lustre: DEBUG MARKER: == recovery-small test 22: drop close request and do mknod =========================================== 17:06:32 (1305677192)
May 17 17:06:32 client-18 xinetd[3390]: EXIT: shell status=0 pid=12660 duration=0(sec)
May 17 17:06:39 client-18 xinetd[3390]: START: shell pid=12695 from=192.168.4.5
May 17 17:06:39 client-18 rshd[12696]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 17:06:39 client-18 xinetd[3390]: EXIT: shell status=0 pid=12695 duration=0(sec)
May 17 17:06:39 client-18 xinetd[3390]: START: shell pid=12731 from=192.168.4.5
May 17 17:06:40 client-18 rshd[12732]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 17:06:40 client-18 xinetd[3390]: EXIT: shell status=0 pid=12731 duration=1(sec)
May 17 17:06:40 client-18 xinetd[3390]: START: shell pid=12768 from=192.168.4.5
May 17 17:06:40 client-18 rshd[12769]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == recovery-small test 23: client hang when close a file after mds crash ============================= 17:06:40 \(1305677200\)");echo XXRETCODE:$?'
May 17 17:06:40 client-18 kernel: Lustre: DEBUG MARKER: == recovery-small test 23: client hang when close a file after mds crash ============================= 17:06:40 (1305677200)
May 17 17:06:40 client-18 xinetd[3390]: EXIT: shell status=0 pid=12768 duration=0(sec)
May 17 17:06:57 client-18 kernel: Lustre: 8884:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369101282837261 sent from MGC192.168.4.128@o2ib to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305677210] [real_sent 1305677210] [current 1305677217] [deadline 7s] [delay 0s]  req@ffff8102f2f9ac00 x1369101282837261/t0(0) o-1->MGS@MGC192.168.4.128@o2ib_0:26/25 lens 192/192 e 0 to 1 dl 1305677217 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
May 17 17:06:57 client-18 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail.
May 17 17:07:03 client-18 kernel: Lustre: 8885:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369101282837277 sent from MGC192.168.4.128@o2ib to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305677217] [real_sent 1305677217] [current 1305677223] [deadline 6s] [delay 0s]  req@ffff8102f76dac00 x1369101282837277/t0(0) o-1->MGS@MGC192.168.4.128@o2ib_0:26/25 lens 368/392 e 0 to 1 dl 1305677223 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
May 17 17:07:04 client-18 xinetd[3390]: START: shell pid=12803 from=192.168.4.5
May 17 17:07:04 client-18 rshd[12804]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 17:07:04 client-18 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_statfs operation failed with -107
May 17 17:07:04 client-18 kernel: Lustre: lustre-MDT0000-mdc-ffff81031dc9a800: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete.
May 17 17:07:07 client-18 kernel: Lustre: 8886:0:(import.c:529:import_select_connection()) MGC192.168.4.128@o2ib: tried all connections, increasing latency to 6s
May 17 17:07:07 client-18 kernel: Lustre: 8885:0:(import.c:885:ptlrpc_connect_interpret()) MGS@MGC192.168.4.128@o2ib_0 changed server handle from 0xfff98862bf2aa6b3 to 0xfff98862bf2acc6f
May 17 17:07:07 client-18 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import
May 17 17:07:07 client-18 kernel: Lustre: MGC192.168.4.128@o2ib: Connection restored to service MGS using nid 192.168.4.128@o2ib.
May 17 17:07:16 client-18 kernel: Lustre: lustre-MDT0000-mdc-ffff81031dc9a800: Connection restored to service lustre-MDT0000 using nid 192.168.4.128@o2ib.
May 17 17:07:16 client-18 xinetd[3390]: EXIT: shell status=0 pid=12803 duration=12(sec)
May 17 17:07:16 client-18 xinetd[3390]: START: shell pid=12841 from=192.168.4.5
May 17 17:07:16 client-18 rshd[12842]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 17:07:16 client-18 xinetd[3390]: EXIT: shell status=0 pid=12841 duration=0(sec)
May 17 17:07:17 client-18 xinetd[3390]: START: shell pid=12877 from=192.168.4.5
May 17 17:07:17 client-18 rshd[12878]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 17:07:17 client-18 xinetd[3390]: EXIT: shell status=0 pid=12877 duration=0(sec)
May 17 17:07:17 client-18 xinetd[3390]: START: shell pid=12914 from=192.168.4.5
May 17 17:07:17 client-18 rshd[12915]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark  SKIP: recovery-small test_24 skipping ALWAYS excluded test 24");echo XXRETCODE:$?'
May 17 17:07:17 client-18 kernel: Lustre: DEBUG MARKER: SKIP: recovery-small test_24 skipping ALWAYS excluded test 24
May 17 17:07:17 client-18 xinetd[3390]: EXIT: shell status=0 pid=12914 duration=0(sec)
May 17 17:07:17 client-18 xinetd[3390]: START: shell pid=12949 from=192.168.4.5
May 17 17:07:17 client-18 rshd[12950]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == recovery-small test 26a: evict dead exports ======================================================= 17:07:17 \(1305677237\)");echo XXRETCODE:$?'
May 17 17:07:17 client-18 kernel: Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports ======================================================= 17:07:17 (1305677237)
May 17 17:07:17 client-18 xinetd[3390]: EXIT: shell status=0 pid=12949 duration=0(sec)
May 17 17:07:29 client-18 kernel: Lustre: 8884:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369101282837254 sent from lustre-MDT0000-mdc-ffff81031dc9a800 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305677205] [real_sent 1305677205] [current 1305677249] [deadline 44s] [delay 0s]  req@ffff810303497000 x1369101282837254/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 192/192 e 0 to 1 dl 1305677249 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
May 17 17:07:34 client-18 kernel: Lustre: 8884:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369101282837262 sent from lustre-MDT0000-mdc-ffff81031dc9a800 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305677210] [real_sent 1305677210] [current 1305677254] [deadline 44s] [delay 0s]  req@ffff8102f240e000 x1369101282837262/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 192/192 e 0 to 1 dl 1305677254 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
May 17 17:07:39 client-18 kernel: Lustre: 8884:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369101282837270 sent from lustre-MDT0000-mdc-ffff81031dc9a800 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305677215] [real_sent 1305677215] [current 1305677259] [deadline 44s] [delay 0s]  req@ffff8102fb82a000 x1369101282837270/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 192/192 e 0 to 1 dl 1305677259 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
May 17 17:07:46 client-18 kernel: Lustre: 8884:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369101282837278 sent from lustre-MDT0000-mdc-ffff81031dc9a800 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305677222] [real_sent 1305677222] [current 1305677266] [deadline 44s] [delay 0s]  req@ffff81032fe41c00 x1369101282837278/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 192/192 e 0 to 1 dl 1305677266 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
May 17 17:08:03 client-18 xinetd[3390]: START: shell pid=12988 from=192.168.4.5
May 17 17:08:03 client-18 rshd[12989]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 17:08:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=12988 duration=0(sec)
May 17 17:08:04 client-18 xinetd[3390]: START: shell pid=13024 from=192.168.4.5
May 17 17:08:04 client-18 rshd[13025]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 17:08:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=13024 duration=0(sec)
May 17 17:08:04 client-18 xinetd[3390]: START: shell pid=13061 from=192.168.4.5
May 17 17:08:04 client-18 rshd[13062]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == recovery-small test 26b: evict dead exports ======================================================= 17:08:04 \(1305677284\)");echo XXRETCODE:$?'
May 17 17:08:04 client-18 kernel: Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports ======================================================= 17:08:04 (1305677284)
May 17 17:08:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=13061 duration=0(sec)
May 17 17:08:05 client-18 xinetd[3390]: START: shell pid=13096 from=192.168.4.5
May 17 17:08:05 client-18 rshd[13097]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 17:08:05 client-18 xinetd[3390]: EXIT: shell status=0 pid=13096 duration=0(sec)
May 17 17:09:01 client-18 xinetd[3390]: START: shell pid=13134 from=192.168.4.5
May 17 17:09:01 client-18 rshd[13135]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 17:09:01 client-18 xinetd[3390]: EXIT: shell status=0 pid=13134 duration=0(sec)
May 17 17:09:01 client-18 xinetd[3390]: START: shell pid=13170 from=192.168.4.5
May 17 17:09:01 client-18 rshd[13171]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 17:09:02 client-18 xinetd[3390]: EXIT: shell status=0 pid=13170 duration=1(sec)
May 17 17:09:02 client-18 xinetd[3390]: START: shell pid=13207 from=192.168.4.5
May 17 17:09:02 client-18 rshd[13208]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == recovery-small test 27: fail LOV while using OSC\'s ================================================ 17:09:02 \(1305677342\)");echo XXRETCODE:$?'
May 17 17:09:02 client-18 kernel: Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ================================================ 17:09:02 (1305677342)
May 17 17:09:02 client-18 xinetd[3390]: EXIT: shell status=0 pid=13207 duration=0(sec)
May 17 17:09:19 client-18 kernel: Lustre: 8884:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369101282837476 sent from MGC192.168.4.128@o2ib to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305677347] [real_sent 1305677347] [current 1305677359] [deadline 12s] [delay 0s]  req@ffff81033cfc4800 x1369101282837476/t0(0) o-1->MGS@MGC192.168.4.128@o2ib_0:26/25 lens 192/192 e 0 to 1 dl 1305677359 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
May 17 17:09:19 client-18 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail.
May 17 17:09:34 client-18 kernel: Lustre: 8886:0:(import.c:529:import_select_connection()) MGC192.168.4.128@o2ib: tried all connections, increasing latency to 11s
May 17 17:09:49 client-18 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The obd_ping operation failed with -107
May 17 17:09:49 client-18 kernel: Lustre: lustre-MDT0000-mdc-ffff81031dc9a800: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete.
May 17 17:09:51 client-18 kernel: Lustre: 8884:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369101282837477 sent from lustre-MDT0000-mdc-ffff81031dc9a800 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305677347] [real_sent 1305677347] [current 1305677390] [deadline 43s] [delay 0s]  req@ffff810327f61400 x1369101282837477/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 192/192 e 0 to 1 dl 1305677390 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
May 17 17:09:51 client-18 kernel: Lustre: 8884:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 1 previous similar message
May 17 17:09:54 client-18 kernel: Lustre: 8886:0:(import.c:529:import_select_connection()) MGC192.168.4.128@o2ib: tried all connections, increasing latency to 16s
May 17 17:09:55 client-18 kernel: Lustre: 8885:0:(import.c:885:ptlrpc_connect_interpret()) MGS@MGC192.168.4.128@o2ib_0 changed server handle from 0xfff98862bf2acc6f to 0xfff98862bf2b0ea9
May 17 17:09:55 client-18 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import
May 17 17:09:55 client-18 kernel: Lustre: MGC192.168.4.128@o2ib: Connection restored to service MGS using nid 192.168.4.128@o2ib.
May 17 17:10:03 client-18 kernel: Lustre: lustre-MDT0000-mdc-ffff81031dc9a800: Connection restored to service lustre-MDT0000 using nid 192.168.4.128@o2ib.
May 17 17:10:04 client-18 xinetd[3390]: START: shell pid=13250 from=192.168.4.5
May 17 17:10:04 client-18 rshd[13251]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 17:10:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=13250 duration=0(sec)
May 17 17:10:04 client-18 xinetd[3390]: START: shell pid=13286 from=192.168.4.5
May 17 17:10:04 client-18 rshd[13287]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 17:10:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=13286 duration=0(sec)
May 17 17:10:04 client-18 xinetd[3390]: START: shell pid=13323 from=192.168.4.5
May 17 17:10:04 client-18 rshd[13324]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == recovery-small test 28: handle error adding new clients \(bug 6086\) ================================ 17:10:04 \(1305677404\)");echo XXRETCODE:$?'
May 17 17:10:04 client-18 kernel: Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ================================ 17:10:04 (1305677404)
May 17 17:10:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=13323 duration=0(sec)
May 17 17:10:27 client-18 kernel: Lustre: 8884:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369101282837523 sent from lustre-MDT0000-mdc-ffff81031dc9a800 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305677384] [real_sent 1305677384] [current 1305677427] [deadline 43s] [delay 0s]  req@ffff8102ff3cec00 x1369101282837523/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 192/192 e 0 to 1 dl 1305677427 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
May 17 17:10:28 client-18 kernel: Lustre: 8884:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 5 previous similar messages
May 17 17:10:53 client-18 xinetd[3390]: START: shell pid=13358 from=192.168.4.5
May 17 17:10:53 client-18 rshd[13359]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 17:10:53 client-18 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_statfs operation failed with -107
May 17 17:10:53 client-18 kernel: Lustre: lustre-MDT0000-mdc-ffff81031dc9a800: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete.
May 17 17:10:53 client-18 kernel: Lustre: lustre-MDT0000-mdc-ffff81031dc9a800: Connection restored to service lustre-MDT0000 using nid 192.168.4.128@o2ib.
May 17 17:10:53 client-18 xinetd[3390]: EXIT: shell status=0 pid=13358 duration=0(sec)
May 17 17:10:53 client-18 xinetd[3390]: START: shell pid=13392 from=192.168.4.5
May 17 17:10:53 client-18 rshd[13393]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 17:10:53 client-18 xinetd[3390]: EXIT: shell status=0 pid=13392 duration=0(sec)
May 17 17:10:53 client-18 xinetd[3390]: START: shell pid=13428 from=192.168.4.5
May 17 17:10:54 client-18 rshd[13429]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 17:10:54 client-18 xinetd[3390]: EXIT: shell status=0 pid=13428 duration=1(sec)
May 17 17:10:54 client-18 xinetd[3390]: START: shell pid=13465 from=192.168.4.5
May 17 17:10:54 client-18 rshd[13466]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == recovery-small test 29a: error adding new clients doesn\'t cause LBUG \(bug 22273\) ================== 17:10:54 \(1305677454\)");echo XXRETCODE:$?'
May 17 17:10:54 client-18 kernel: Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ================== 17:10:54 (1305677454)
May 17 17:10:54 client-18 xinetd[3390]: EXIT: shell status=0 pid=13465 duration=0(sec)
May 17 17:10:54 client-18 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The obd_ping operation failed with -107
May 17 17:10:54 client-18 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail.
May 17 17:11:02 client-18 xinetd[3390]: START: shell pid=13500 from=192.168.4.5
May 17 17:11:02 client-18 rshd[13501]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 17:11:03 client-18 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_statfs operation failed with -107
May 17 17:11:03 client-18 kernel: Lustre: lustre-MDT0000-mdc-ffff81031dc9a800: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete.
May 17 17:11:03 client-18 kernel: LustreError: 167-0: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
May 17 17:11:03 client-18 kernel: LustreError: 13533:0:(lmv_obd.c:1198:lmv_statfs()) can't stat MDS #0 (lustre-MDT0000-mdc-ffff81031dc9a800), error -4
May 17 17:11:03 client-18 kernel: Lustre: lustre-MDT0000-mdc-ffff81031dc9a800: Connection restored to service lustre-MDT0000 using nid 192.168.4.128@o2ib.
May 17 17:11:03 client-18 kernel: LustreError: 13533:0:(llite_lib.c:1370:ll_statfs_internal()) md_statfs fails: rc = -4
May 17 17:11:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=13500 duration=1(sec)
May 17 17:11:09 client-18 xinetd[3390]: START: shell pid=13535 from=192.168.4.5
May 17 17:11:09 client-18 rshd[13536]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 17:11:10 client-18 xinetd[3390]: EXIT: shell status=0 pid=13535 duration=1(sec)
May 17 17:11:11 client-18 xinetd[3390]: START: shell pid=13573 from=192.168.4.5
May 17 17:11:11 client-18 rshd[13574]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 17:11:11 client-18 xinetd[3390]: EXIT: shell status=0 pid=13573 duration=0(sec)
May 17 17:11:11 client-18 xinetd[3390]: START: shell pid=13609 from=192.168.4.5
May 17 17:11:11 client-18 rshd[13610]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 17:11:11 client-18 xinetd[3390]: EXIT: shell status=0 pid=13609 duration=0(sec)
May 17 17:11:11 client-18 xinetd[3390]: START: shell pid=13646 from=192.168.4.5
May 17 17:11:11 client-18 rshd[13647]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == recovery-small test 29b: error adding new clients doesn\'t cause LBUG \(bug 22273\) ================== 17:11:11 \(1305677471\)");echo XXRETCODE:$?'
May 17 17:11:11 client-18 kernel: Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ================== 17:11:11 (1305677471)
May 17 17:11:11 client-18 xinetd[3390]: EXIT: shell status=0 pid=13646 duration=0(sec)
May 17 17:11:15 client-18 xinetd[3390]: START: shell pid=13681 from=192.168.4.5
May 17 17:11:15 client-18 rshd[13682]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 17:11:15 client-18 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.129@o2ib. The ost_statfs operation failed with -107
May 17 17:11:15 client-18 kernel: Lustre: lustre-OST0000-osc-ffff81031dc9a800: Connection to service lustre-OST0000 via nid 192.168.4.129@o2ib was lost; in progress operations using this service will wait for recovery to complete.
May 17 17:11:15 client-18 kernel: LustreError: 167-0: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
May 17 17:11:15 client-18 kernel: Lustre: lustre-OST0000-osc-ffff81031dc9a800: Connection restored to service lustre-OST0000 using nid 192.168.4.129@o2ib.
May 17 17:11:15 client-18 xinetd[3390]: EXIT: shell status=0 pid=13681 duration=0(sec)
May 17 17:11:19 client-18 kernel: Lustre: 8886:0:(import.c:529:import_select_connection()) MGC192.168.4.128@o2ib: tried all connections, increasing latency to 21s
May 17 17:11:19 client-18 kernel: Lustre: 8885:0:(import.c:885:ptlrpc_connect_interpret()) MGS@MGC192.168.4.128@o2ib_0 changed server handle from 0xfff98862bf2b0ea9 to 0xfff98862bf2b124c
May 17 17:11:19 client-18 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import
May 17 17:11:24 client-18 xinetd[3390]: START: shell pid=13720 from=192.168.4.5
May 17 17:11:24 client-18 rshd[13721]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 17:11:25 client-18 xinetd[3390]: EXIT: shell status=0 pid=13720 duration=1(sec)
May 17 17:11:26 client-18 xinetd[3390]: START: shell pid=13754 from=192.168.4.5
May 17 17:11:26 client-18 rshd[13755]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 17:11:26 client-18 xinetd[3390]: EXIT: shell status=0 pid=13754 duration=0(sec)
May 17 17:11:26 client-18 xinetd[3390]: START: shell pid=13790 from=192.168.4.5
May 17 17:11:26 client-18 rshd[13791]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 17:11:26 client-18 xinetd[3390]: EXIT: shell status=0 pid=13790 duration=0(sec)
May 17 17:11:26 client-18 xinetd[3390]: START: shell pid=13827 from=192.168.4.5
May 17 17:11:26 client-18 rshd[13828]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == recovery-small test 50: failover MDS under load =================================================== 17:11:26 \(1305677486\)");echo XXRETCODE:$?'
May 17 17:11:26 client-18 kernel: Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load =================================================== 17:11:26 (1305677486)
May 17 17:11:26 client-18 xinetd[3390]: EXIT: shell status=0 pid=13827 duration=0(sec)
May 17 17:11:46 client-18 kernel: Lustre: 8884:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369101282837711 sent from lustre-MDT0000-mdc-ffff81031dc9a800 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305677499] [real_sent 1305677499] [current 1305677506] [deadline 7s] [delay 0s]  req@ffff8102f65ccc00 x1369101282837711/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 192/192 e 0 to 1 dl 1305677506 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
May 17 17:11:47 client-18 kernel: Lustre: 8884:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 1 previous similar message
May 17 17:11:47 client-18 kernel: Lustre: lustre-MDT0000-mdc-ffff81031dc9a800: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete.
May 17 17:11:55 client-18 xinetd[3390]: START: shell pid=13865 from=192.168.4.5
May 17 17:11:55 client-18 rshd[13866]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 17:11:57 client-18 kernel: Lustre: 8886:0:(import.c:529:import_select_connection()) lustre-MDT0000-mdc-ffff81031dc9a800: tried all connections, increasing latency to 6s
May 17 17:11:57 client-18 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The obd_ping operation failed with -107
May 17 17:11:57 client-18 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail.
May 17 17:11:57 client-18 kernel: Lustre: 8885:0:(import.c:885:ptlrpc_connect_interpret()) MGS@MGC192.168.4.128@o2ib_0 changed server handle from 0xfff98862bf2b124c to 0xfff98862bf2d23f9
May 17 17:11:57 client-18 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import
May 17 17:11:57 client-18 kernel: Lustre: MGC192.168.4.128@o2ib: Connection restored to service MGS using nid 192.168.4.128@o2ib.
May 17 17:11:57 client-18 kernel: Lustre: Skipped 1 previous similar message
May 17 17:12:13 client-18 xinetd[3390]: EXIT: shell status=0 pid=13865 duration=18(sec)
May 17 17:13:32 client-18 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The obd_ping operation failed with -107
May 17 17:13:32 client-18 kernel: Lustre: lustre-MDT0000-mdc-ffff81031dc9a800: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete.
May 17 17:13:32 client-18 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail.
May 17 17:13:32 client-18 kernel: Lustre: 8885:0:(import.c:885:ptlrpc_connect_interpret()) MGS@MGC192.168.4.128@o2ib_0 changed server handle from 0xfff98862bf2d23f9 to 0xfff98862bf39135d
May 17 17:13:32 client-18 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import
May 17 17:13:32 client-18 kernel: Lustre: MGC192.168.4.128@o2ib: Connection restored to service MGS using nid 192.168.4.128@o2ib.
May 17 17:13:32 client-18 kernel: Lustre: Skipped 1 previous similar message
May 17 17:13:32 client-18 xinetd[3390]: START: shell pid=13908 from=192.168.4.5
May 17 17:13:32 client-18 rshd[13909]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 17:13:43 client-18 xinetd[3390]: EXIT: shell status=0 pid=13908 duration=11(sec)
May 17 17:13:58 client-18 kernel: Lustre: 8884:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369101282837877 sent from lustre-MDT0000-mdc-ffff81031dc9a800 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305677607] [real_sent 1305677607] [current 1305677638] [deadline 31s] [delay 0s]  req@ffff8102f3202400 x1369101282837877/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 192/192 e 0 to 1 dl 1305677638 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
May 17 17:13:58 client-18 kernel: Lustre: 8884:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 4 previous similar messages
May 17 17:15:02 client-18 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The obd_ping operation failed with -107
May 17 17:15:02 client-18 kernel: LustreError: Skipped 1 previous similar message
May 17 17:15:02 client-18 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail.
May 17 17:15:02 client-18 kernel: Lustre: lustre-MDT0000-mdc-ffff81031dc9a800: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete.
May 17 17:15:02 client-18 kernel: Lustre: 8885:0:(import.c:885:ptlrpc_connect_interpret()) MGS@MGC192.168.4.128@o2ib_0 changed server handle from 0xfff98862bf39135d to 0xfff98862bf44ae7b
May 17 17:15:02 client-18 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import
May 17 17:15:02 client-18 kernel: Lustre: MGC192.168.4.128@o2ib: Connection restored to service MGS using nid 192.168.4.128@o2ib.
May 17 17:15:02 client-18 kernel: Lustre: Skipped 1 previous similar message
May 17 17:15:02 client-18 xinetd[3390]: START: shell pid=13954 from=192.168.4.5
May 17 17:15:02 client-18 rshd[13955]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 17:15:23 client-18 xinetd[3390]: EXIT: shell status=0 pid=13954 duration=21(sec)
May 17 17:15:43 client-18 xinetd[3390]: START: shell pid=13990 from=192.168.4.5
May 17 17:15:43 client-18 rshd[13991]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 17:15:44 client-18 xinetd[3390]: EXIT: shell status=0 pid=13990 duration=1(sec)
May 17 17:15:44 client-18 xinetd[3390]: START: shell pid=14026 from=192.168.4.5
May 17 17:15:44 client-18 rshd[14027]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 17:15:44 client-18 xinetd[3390]: EXIT: shell status=0 pid=14026 duration=0(sec)
May 17 17:15:44 client-18 xinetd[3390]: START: shell pid=14063 from=192.168.4.5
May 17 17:15:44 client-18 rshd[14064]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == recovery-small test 51: failover MDS during recovery ============================================== 17:15:44 \(1305677744\)");echo XXRETCODE:$?'
May 17 17:15:44 client-18 kernel: Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery ============================================== 17:15:44 (1305677744)
May 17 17:15:44 client-18 xinetd[3390]: EXIT: shell status=0 pid=14063 duration=0(sec)
May 17 17:16:14 client-18 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail.
May 17 17:16:24 client-18 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The obd_ping operation failed with -107
May 17 17:16:24 client-18 kernel: LustreError: Skipped 1 previous similar message
May 17 17:16:24 client-18 kernel: Lustre: lustre-MDT0000-mdc-ffff81031dc9a800: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete.
May 17 17:16:44 client-18 kernel: Lustre: 8886:0:(import.c:529:import_select_connection()) MGC192.168.4.128@o2ib: tried all connections, increasing latency to 21s
May 17 17:17:14 client-18 kernel: Lustre: 8886:0:(import.c:529:import_select_connection()) MGC192.168.4.128@o2ib: tried all connections, increasing latency to 21s
May 17 17:17:14 client-18 kernel: Lustre: 8885:0:(import.c:885:ptlrpc_connect_interpret()) MGS@MGC192.168.4.128@o2ib_0 changed server handle from 0xfff98862bf44ae7b to 0xfff98862bf49c768
May 17 17:17:14 client-18 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import
May 17 17:17:14 client-18 kernel: Lustre: MGC192.168.4.128@o2ib: Connection restored to service MGS using nid 192.168.4.128@o2ib.
May 17 17:17:14 client-18 kernel: Lustre: Skipped 3 previous similar messages
May 17 17:17:54 client-18 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail.
May 17 17:17:54 client-18 kernel: Lustre: 8885:0:(import.c:885:ptlrpc_connect_interpret()) MGS@MGC192.168.4.128@o2ib_0 changed server handle from 0xfff98862bf49c768 to 0xfff98862bf4bc630
May 17 17:17:54 client-18 kernel: Lustre: lustre-MDT0000-mdc-ffff81031dc9a800: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete.
May 17 17:17:54 client-18 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import
May 17 17:17:54 client-18 kernel: Lustre: Skipped 2 previous similar messages
May 17 17:18:15 client-18 kernel: Lustre: 8884:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369101282838274 sent from lustre-MDT0000-mdc-ffff81031dc9a800 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305677859] [real_sent 1305677859] [current 1305677895] [deadline 36s] [delay 0s]  req@ffff8102fb0e4000 x1369101282838274/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 192/192 e 0 to 1 dl 1305677895 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
May 17 17:18:15 client-18 kernel: Lustre: 8884:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 20 previous similar messages
May 17 17:18:34 client-18 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The obd_ping operation failed with -107
May 17 17:18:34 client-18 kernel: LustreError: Skipped 4 previous similar messages
May 17 17:18:34 client-18 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail.
May 17 17:18:34 client-18 kernel: Lustre: 8885:0:(import.c:885:ptlrpc_connect_interpret()) MGS@MGC192.168.4.128@o2ib_0 changed server handle from 0xfff98862bf4bc630 to 0xfff98862bf4df73a
May 17 17:19:26 client-18 kernel: Lustre: 8886:0:(import.c:529:import_select_connection()) MGC192.168.4.128@o2ib: tried all connections, increasing latency to 6s
May 17 17:19:26 client-18 kernel: Lustre: 8885:0:(import.c:885:ptlrpc_connect_interpret()) MGS@MGC192.168.4.128@o2ib_0 changed server handle from 0xfff98862bf4df73a to 0xfff98862bf51af09
May 17 17:19:26 client-18 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import
May 17 17:19:26 client-18 kernel: Lustre: Skipped 1 previous similar message
May 17 17:19:42 client-18 xinetd[3390]: START: shell pid=14123 from=192.168.4.5
May 17 17:19:42 client-18 rshd[14124]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 17:19:42 client-18 xinetd[3390]: EXIT: shell status=0 pid=14123 duration=0(sec)
May 17 17:19:42 client-18 xinetd[3390]: START: shell pid=14159 from=192.168.4.5
May 17 17:19:42 client-18 rshd[14160]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 17:19:42 client-18 xinetd[3390]: EXIT: shell status=0 pid=14159 duration=0(sec)
May 17 17:19:42 client-18 xinetd[3390]: START: shell pid=14196 from=192.168.4.5
May 17 17:19:42 client-18 rshd[14197]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark  SKIP: recovery-small test_52 skipping ALWAYS excluded test 52");echo XXRETCODE:$?'
May 17 17:19:42 client-18 kernel: Lustre: DEBUG MARKER: SKIP: recovery-small test_52 skipping ALWAYS excluded test 52
May 17 17:19:42 client-18 xinetd[3390]: EXIT: shell status=0 pid=14196 duration=0(sec)
May 17 17:19:42 client-18 xinetd[3390]: START: shell pid=14231 from=192.168.4.5
May 17 17:19:42 client-18 rshd[14232]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == recovery-small test 53: touch: drop rep =========================================================== 17:19:42 \(1305677982\)");echo XXRETCODE:$?'
May 17 17:19:42 client-18 kernel: Lustre: DEBUG MARKER: == recovery-small test 53: touch: drop rep =========================================================== 17:19:42 (1305677982)
May 17 17:19:42 client-18 xinetd[3390]: EXIT: shell status=0 pid=14231 duration=0(sec)
May 17 17:20:34 client-18 xinetd[3390]: START: shell pid=14270 from=192.168.4.5
May 17 17:20:34 client-18 rshd[14271]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 17:20:34 client-18 xinetd[3390]: EXIT: shell status=0 pid=14270 duration=0(sec)
May 17 17:20:34 client-18 xinetd[3390]: START: shell pid=14306 from=192.168.4.5
May 17 17:20:34 client-18 rshd[14307]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 17:20:34 client-18 xinetd[3390]: EXIT: shell status=0 pid=14306 duration=0(sec)
May 17 17:20:34 client-18 xinetd[3390]: START: shell pid=14343 from=192.168.4.5
May 17 17:20:34 client-18 rshd[14344]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == recovery-small test 54: back in time ============================================================== 17:20:34 \(1305678034\)");echo XXRETCODE:$?'
May 17 17:20:34 client-18 kernel: Lustre: DEBUG MARKER: == recovery-small test 54: back in time ============================================================== 17:20:34 (1305678034)
May 17 17:20:34 client-18 xinetd[3390]: EXIT: shell status=0 pid=14343 duration=0(sec)
May 17 17:20:58 client-18 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail.
May 17 17:20:58 client-18 kernel: LustreError: Skipped 1 previous similar message
May 17 17:21:03 client-18 kernel: Lustre: lustre-MDT0000-mdc-ffff81031dc9a800: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete.
May 17 17:21:03 client-18 kernel: Lustre: Skipped 2 previous similar messages
May 17 17:21:03 client-18 xinetd[3390]: START: shell pid=14378 from=192.168.4.5
May 17 17:21:04 client-18 rshd[14379]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 17:21:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=14378 duration=1(sec)
May 17 17:21:05 client-18 xinetd[3390]: START: shell pid=14412 from=192.168.4.5
May 17 17:21:05 client-18 rshd[14413]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 17:21:05 client-18 xinetd[3390]: EXIT: shell status=0 pid=14412 duration=0(sec)
May 17 17:21:05 client-18 xinetd[3390]: START: shell pid=14448 from=192.168.4.5
May 17 17:21:05 client-18 rshd[14449]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 17:21:05 client-18 xinetd[3390]: EXIT: shell status=0 pid=14448 duration=0(sec)
May 17 17:21:05 client-18 xinetd[3390]: START: shell pid=14485 from=192.168.4.5
May 17 17:21:05 client-18 rshd[14486]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == recovery-small test 55: ost_brw_read\/write drops timed-out read\/write request ===================== 17:21:05 \(1305678065\)");echo XXRETCODE:$?'
May 17 17:21:05 client-18 kernel: Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ===================== 17:21:05 (1305678065)
May 17 17:21:05 client-18 xinetd[3390]: EXIT: shell status=0 pid=14485 duration=0(sec)
May 17 17:21:13 client-18 kernel: Lustre: 8886:0:(import.c:529:import_select_connection()) MGC192.168.4.128@o2ib: tried all connections, increasing latency to 11s
May 17 17:21:13 client-18 kernel: Lustre: 8885:0:(import.c:885:ptlrpc_connect_interpret()) MGS@MGC192.168.4.128@o2ib_0 changed server handle from 0xfff98862bf51af09 to 0xfff98862bf52b2b1
May 17 17:22:37 client-18 xinetd[3390]: START: shell pid=14532 from=192.168.4.5
May 17 17:22:37 client-18 rshd[14533]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 17:22:37 client-18 xinetd[3390]: EXIT: shell status=0 pid=14532 duration=0(sec)
May 17 17:22:37 client-18 xinetd[3390]: START: shell pid=14568 from=192.168.4.5
May 17 17:22:37 client-18 rshd[14569]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 17:22:37 client-18 xinetd[3390]: EXIT: shell status=0 pid=14568 duration=0(sec)
May 17 17:22:37 client-18 xinetd[3390]: START: shell pid=14605 from=192.168.4.5
May 17 17:22:37 client-18 rshd[14606]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == recovery-small test 56: do not allow reconnect to busy exports ==================================== 17:22:37 \(1305678157\)");echo XXRETCODE:$?'
May 17 17:22:38 client-18 kernel: Lustre: DEBUG MARKER: == recovery-small test 56: do not allow reconnect to busy exports ==================================== 17:22:37 (1305678157)
May 17 17:22:38 client-18 xinetd[3390]: EXIT: shell status=0 pid=14605 duration=1(sec)
May 17 17:23:18 client-18 xinetd[3390]: START: shell pid=14640 from=192.168.4.5
May 17 17:23:18 client-18 rshd[14641]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 17:23:18 client-18 xinetd[3390]: EXIT: shell status=0 pid=14640 duration=0(sec)
May 17 17:23:18 client-18 xinetd[3390]: START: shell pid=14676 from=192.168.4.5
May 17 17:23:18 client-18 rshd[14677]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 17:23:18 client-18 xinetd[3390]: EXIT: shell status=0 pid=14676 duration=0(sec)
May 17 17:23:18 client-18 xinetd[3390]: START: shell pid=14713 from=192.168.4.5
May 17 17:23:18 client-18 rshd[14714]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == recovery-small test 57: read procfs entries causes kernel crash =================================== 17:23:18 \(1305678198\)");echo XXRETCODE:$?'
May 17 17:23:18 client-18 kernel: Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash =================================== 17:23:18 (1305678198)
May 17 17:23:18 client-18 xinetd[3390]: EXIT: shell status=0 pid=14713 duration=0(sec)
May 17 17:23:38 client-18 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The obd_ping operation failed with -107
May 17 17:23:38 client-18 kernel: LustreError: Skipped 3 previous similar messages
May 17 17:23:38 client-18 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail.
May 17 17:23:38 client-18 kernel: Lustre: 8885:0:(import.c:885:ptlrpc_connect_interpret()) MGS@MGC192.168.4.128@o2ib_0 changed server handle from 0xfff98862bf52b2b1 to 0xfff98862bf52b566
May 17 17:23:38 client-18 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import
May 17 17:23:38 client-18 kernel: Lustre: Skipped 1 previous similar message
May 17 17:23:38 client-18 kernel: Lustre: MGC192.168.4.128@o2ib: Connection restored to service MGS using nid 192.168.4.128@o2ib.
May 17 17:23:38 client-18 kernel: Lustre: Skipped 9 previous similar messages
May 17 17:23:39 client-18 xinetd[3390]: START: shell pid=14750 from=192.168.4.5
May 17 17:23:39 client-18 rshd[14751]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 17:23:48 client-18 kernel: LustreError: 167-0: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
May 17 17:23:48 client-18 kernel: LustreError: 14783:0:(lmv_obd.c:1198:lmv_statfs()) can't stat MDS #0 (lustre-MDT0000-mdc-ffff81031dc9a800), error -4
May 17 17:23:48 client-18 kernel: LustreError: 14783:0:(llite_lib.c:1370:ll_statfs_internal()) md_statfs fails: rc = -4
May 17 17:23:48 client-18 xinetd[3390]: EXIT: shell status=0 pid=14750 duration=9(sec)
May 17 17:23:49 client-18 xinetd[3390]: START: shell pid=14791 from=192.168.4.5
May 17 17:23:49 client-18 rshd[14792]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 17:23:49 client-18 xinetd[3390]: EXIT: shell status=0 pid=14791 duration=0(sec)
May 17 17:23:49 client-18 xinetd[3390]: START: shell pid=14825 from=192.168.4.5
May 17 17:23:49 client-18 rshd[14826]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 17:23:49 client-18 xinetd[3390]: EXIT: shell status=0 pid=14825 duration=0(sec)
May 17 17:23:49 client-18 xinetd[3390]: START: shell pid=14861 from=192.168.4.5
May 17 17:23:49 client-18 rshd[14862]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 17:23:49 client-18 xinetd[3390]: EXIT: shell status=0 pid=14861 duration=0(sec)
May 17 17:23:49 client-18 xinetd[3390]: START: shell pid=14898 from=192.168.4.5
May 17 17:23:49 client-18 rshd[14899]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == recovery-small test 58: Eviction in the middle of open RPC reply processing ======================= 17:23:49 \(1305678229\)");echo XXRETCODE:$?'
May 17 17:23:49 client-18 kernel: Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ======================= 17:23:49 (1305678229)
May 17 17:23:49 client-18 xinetd[3390]: EXIT: shell status=0 pid=14898 duration=0(sec)
May 17 17:24:10 client-18 xinetd[3390]: START: shell pid=14933 from=192.168.4.5
May 17 17:24:10 client-18 rshd[14934]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 17:24:10 client-18 xinetd[3390]: EXIT: shell status=0 pid=14933 duration=0(sec)
May 17 17:24:10 client-18 xinetd[3390]: START: shell pid=14969 from=192.168.4.5
May 17 17:24:10 client-18 rshd[14970]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 17:24:10 client-18 xinetd[3390]: EXIT: shell status=0 pid=14969 duration=0(sec)
May 17 17:24:10 client-18 xinetd[3390]: START: shell pid=15006 from=192.168.4.5
May 17 17:24:10 client-18 rshd[15007]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == recovery-small test 59: Read cancel race on client eviction ======================================= 17:24:10 \(1305678250\)");echo XXRETCODE:$?'
May 17 17:24:10 client-18 kernel: Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ======================================= 17:24:10 (1305678250)
May 17 17:24:10 client-18 xinetd[3390]: EXIT: shell status=0 pid=15006 duration=0(sec)
May 17 17:24:11 client-18 xinetd[3390]: START: shell pid=15041 from=192.168.4.5
May 17 17:24:11 client-18 rshd[15042]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 17:24:11 client-18 xinetd[3390]: EXIT: shell status=0 pid=15041 duration=0(sec)
May 17 17:24:11 client-18 xinetd[3390]: START: shell pid=15077 from=192.168.4.5
May 17 17:24:11 client-18 rshd[15078]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 17:24:11 client-18 xinetd[3390]: EXIT: shell status=0 pid=15077 duration=0(sec)
May 17 17:24:11 client-18 xinetd[3390]: START: shell pid=15114 from=192.168.4.5
May 17 17:24:11 client-18 rshd[15115]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == recovery-small test 60: Add Changelog entries during MDS failover ================================= 17:24:11 \(1305678251\)");echo XXRETCODE:$?'
May 17 17:24:11 client-18 kernel: Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ================================= 17:24:11 (1305678251)
May 17 17:24:11 client-18 xinetd[3390]: EXIT: shell status=0 pid=15114 duration=0(sec)
May 17 17:25:23 client-18 kernel: Lustre: lustre-MDT0000-mdc-ffff81031dc9a800: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete.
May 17 17:25:23 client-18 kernel: Lustre: Skipped 1 previous similar message
May 17 17:25:23 client-18 kernel: Lustre: 8885:0:(import.c:885:ptlrpc_connect_interpret()) MGS@MGC192.168.4.128@o2ib_0 changed server handle from 0xfff98862bf52b566 to 0xfff98862bf595978
May 17 17:26:40 client-18 xinetd[3390]: START: shell pid=15161 from=192.168.4.5
May 17 17:26:40 client-18 rshd[15162]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 17:26:40 client-18 xinetd[3390]: EXIT: shell status=0 pid=15161 duration=0(sec)
May 17 17:26:40 client-18 xinetd[3390]: START: shell pid=15197 from=192.168.4.5
May 17 17:26:40 client-18 rshd[15198]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 17:26:40 client-18 xinetd[3390]: EXIT: shell status=0 pid=15197 duration=0(sec)
May 17 17:26:40 client-18 xinetd[3390]: START: shell pid=15234 from=192.168.4.5
May 17 17:26:40 client-18 rshd[15235]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ============================ 17:26:40 \(1305678400\)");echo XXRETCODE:$?'
May 17 17:26:41 client-18 kernel: Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ============================ 17:26:40 (1305678400)
May 17 17:26:41 client-18 xinetd[3390]: EXIT: shell status=0 pid=15234 duration=1(sec)
May 17 17:26:41 client-18 xinetd[3390]: START: shell pid=15269 from=192.168.4.5
May 17 17:26:41 client-18 rshd[15270]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 17:26:41 client-18 xinetd[3390]: EXIT: shell status=0 pid=15269 duration=0(sec)
May 17 17:26:53 client-18 kernel: Lustre: 8885:0:(import.c:885:ptlrpc_connect_interpret()) MGS@MGC192.168.4.128@o2ib_0 changed server handle from 0xfff98862bf595978 to 0xfff98862bf645ce9
May 17 17:26:53 client-18 kernel: LustreError: 167-0: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
May 17 17:26:54 client-18 xinetd[3390]: START: shell pid=15311 from=192.168.4.5
May 17 17:26:54 client-18 rshd[15312]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 17:26:54 client-18 xinetd[3390]: EXIT: shell status=0 pid=15311 duration=0(sec)
May 17 17:27:03 client-18 xinetd[3390]: START: shell pid=15347 from=192.168.4.5
May 17 17:27:03 client-18 rshd[15348]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 17:27:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=15347 duration=0(sec)
May 17 17:27:03 client-18 xinetd[3390]: START: shell pid=15381 from=192.168.4.5
May 17 17:27:03 client-18 rshd[15382]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 17:27:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=15381 duration=0(sec)
May 17 17:27:03 client-18 xinetd[3390]: START: shell pid=15417 from=192.168.4.5
May 17 17:27:03 client-18 rshd[15418]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 17:27:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=15417 duration=1(sec)
May 17 17:27:04 client-18 xinetd[3390]: START: shell pid=15454 from=192.168.4.5
May 17 17:27:04 client-18 rshd[15455]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == recovery-small recovery-small.sh test complete, duration 1764 sec ================================= 17:27:04 \(1305678424\)");echo XXRETCODE:$?'
May 17 17:27:04 client-18 kernel: Lustre: DEBUG MARKER: == recovery-small recovery-small.sh test complete, duration 1764 sec ================================= 17:27:04 (1305678424)
May 17 17:27:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=15454 duration=0(sec)
May 17 17:27:20 client-18 kernel: Lustre: 8884:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369101282839223 sent from lustre-MDT0000-mdc-ffff81031dc9a800 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305678433] [real_sent 1305678433] [current 1305678440] [deadline 7s] [delay 0s]  req@ffff81033cfca800 x1369101282839223/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 192/192 e 0 to 1 dl 1305678440 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
May 17 17:27:20 client-18 kernel: Lustre: 8884:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 18 previous similar messages
May 17 17:27:23 client-18 xinetd[3390]: START: shell pid=15489 from=192.168.4.5
May 17 17:27:23 client-18 rshd[15490]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 17:27:25 client-18 kernel: LustreError: 8884:0:(client.c:1057:ptlrpc_import_delay_req()) @@@ IMP_INVALID  req@ffff81033cfc4400 x1369101282839240/t0(0) o-1->MGS@MGC192.168.4.128@o2ib_0:26/25 lens 192/192 e 0 to 0 dl 0 ref 1 fl Rpc:N/ffffffff/ffffffff rc 0/-1
May 17 17:27:25 client-18 kernel: Lustre: 8885:0:(import.c:885:ptlrpc_connect_interpret()) MGS@MGC192.168.4.128@o2ib_0 changed server handle from 0xfff98862bf645ce9 to 0xfff98862bf6467d2
May 17 17:27:30 client-18 kernel: Lustre: 8886:0:(import.c:529:import_select_connection()) lustre-MDT0000-mdc-ffff81031dc9a800: tried all connections, increasing latency to 6s
May 17 17:27:30 client-18 xinetd[3390]: EXIT: shell status=0 pid=15489 duration=7(sec)
May 17 17:27:30 client-18 xinetd[3390]: START: shell pid=15528 from=192.168.4.5
May 17 17:27:30 client-18 rshd[15529]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:-f; lsof -t /mnt/lustre || need_kill=no; if [ x-f != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount -f /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre is still busy, wait one second && sleep 1;     umount -f /mnt/lustre; fi fi");echo XXRETCODE:$?'
May 17 17:27:30 client-18 kernel: Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
May 17 17:27:30 client-18 kernel: Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request
May 17 17:27:30 client-18 kernel: LustreError: 15570:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway
May 17 17:27:30 client-18 kernel: LustreError: 15570:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108
May 17 17:27:30 client-18 kernel: Lustre: client ffff81031dc9a800 umount complete
May 17 17:27:30 client-18 xinetd[3390]: EXIT: shell status=0 pid=15528 duration=0(sec)
May 17 17:27:30 client-18 xinetd[3390]: START: shell pid=15573 from=192.168.4.5
May 17 17:27:30 client-18 rshd[15574]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:-f; lsof -t /mnt/lustre2 || need_kill=no; if [ x-f != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre2 | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount -f /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre2 is still busy, wait one second && sleep 1;     umount -f /mnt/lustre2; fi fi");echo XXRETCODE:$?'
May 17 17:27:30 client-18 xinetd[3390]: EXIT: shell status=0 pid=15573 duration=0(sec)
May 17 17:28:16 client-18 xinetd[3390]: START: shell pid=15610 from=192.168.4.5
May 17 17:28:16 client-18 rshd[15611]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark -----============= acceptance-small: replay-ost-single ============----- Tue May 17 17:28:16 PDT 2011");echo XXRETCODE:$?'
May 17 17:28:16 client-18 kernel: Lustre: DEBUG MARKER: -----============= acceptance-small: replay-ost-single ============----- Tue May 17 17:28:16 PDT 2011
May 17 17:28:16 client-18 xinetd[3390]: EXIT: shell status=0 pid=15610 duration=0(sec)
May 17 17:28:16 client-18 xinetd[3390]: START: shell pid=15645 from=192.168.4.5
May 17 17:28:16 client-18 rshd[15646]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark excepting tests: ");echo XXRETCODE:$?'
May 17 17:28:17 client-18 kernel: Lustre: DEBUG MARKER: excepting tests:
May 17 17:28:17 client-18 xinetd[3390]: EXIT: shell status=0 pid=15645 duration=1(sec)
May 17 17:28:17 client-18 xinetd[3390]: START: shell pid=15680 from=192.168.4.5
May 17 17:28:17 client-18 rshd[15681]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:; lsof -t /mnt/lustre || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount  /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre is still busy, wait one second && sleep 1;     umount  /mnt/lustre; fi fi");echo XXRETCODE:$?'
May 17 17:28:17 client-18 xinetd[3390]: EXIT: shell status=0 pid=15680 duration=0(sec)
May 17 17:28:17 client-18 xinetd[3390]: START: shell pid=15717 from=192.168.4.5
May 17 17:28:17 client-18 rshd[15718]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:; lsof -t /mnt/lustre2 || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre2 | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount  /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre2 is still busy, wait one second && sleep 1;     umount  /mnt/lustre2; fi fi");echo XXRETCODE:$?'
May 17 17:28:17 client-18 xinetd[3390]: EXIT: shell status=0 pid=15717 duration=0(sec)
May 17 17:29:16 client-18 xinetd[3390]: START: shell pid=15758 from=192.168.4.5
May 17 17:29:16 client-18 rshd[15759]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); mpts=\$(mount | grep -w -c /mnt/lustre); if [ \$running -ne \$mpts ]; then     echo \$(hostname) env are INSANE!;     exit 1; fi");echo XXRETCODE:$?'
May 17 17:29:16 client-18 xinetd[3390]: EXIT: shell status=0 pid=15758 duration=0(sec)
May 17 17:29:16 client-18 xinetd[3390]: START: shell pid=15798 from=192.168.4.5
May 17 17:29:16 client-18 rshd[15799]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); mpts=\$(mount | grep -w -c /mnt/lustre2); if [ \$running -ne \$mpts ]; then     echo \$(hostname) env are INSANE!;     exit 1; fi");echo XXRETCODE:$?'
May 17 17:29:17 client-18 xinetd[3390]: EXIT: shell status=0 pid=15798 duration=1(sec)
May 17 17:29:26 client-18 xinetd[3390]: START: shell pid=15838 from=192.168.4.5
May 17 17:29:26 client-18 rshd[15839]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c " running=\$(mount | grep -c /mnt/lustre' '); rc=0; if [ \$running -eq 0 ] ; then     mkdir -p /mnt/lustre;     mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre;     rc=\$?; fi; exit \$rc");echo XXRETCODE:$?'
May 17 17:29:26 client-18 kernel: Lustre: 15878:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null
May 17 17:29:26 client-18 kernel: Lustre: 15878:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 6 previous similar messages
May 17 17:29:26 client-18 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'.
May 17 17:29:26 client-18 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import
May 17 17:29:26 client-18 kernel: Lustre: Skipped 3 previous similar messages
May 17 17:29:26 client-18 kernel: Lustre: Client lustre-client has started
May 17 17:29:26 client-18 xinetd[3390]: EXIT: shell status=0 pid=15838 duration=0(sec)
May 17 17:29:26 client-18 xinetd[3390]: START: shell pid=15891 from=192.168.4.5
May 17 17:29:26 client-18 rshd[15892]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "mount | grep -w /mnt/lustre");echo XXRETCODE:$?'
May 17 17:29:26 client-18 xinetd[3390]: EXIT: shell status=0 pid=15891 duration=0(sec)
May 17 17:29:26 client-18 xinetd[3390]: START: shell pid=15928 from=192.168.4.5
May 17 17:29:26 client-18 rshd[15929]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param debug=\"0x33f0406\";         lctl set_param subsystem_debug=\"0xffb7e3ff\";         lctl set_param debug_mb=32;");echo XXRETCODE:$?'
May 17 17:29:26 client-18 kernel: Lustre: 15963:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release.
May 17 17:29:26 client-18 xinetd[3390]: EXIT: shell status=0 pid=15928 duration=0(sec)
May 17 17:29:27 client-18 xinetd[3390]: START: shell pid=15965 from=192.168.4.5
May 17 17:29:27 client-18 rshd[15966]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark Using TIMEOUT=20");echo XXRETCODE:$?'
May 17 17:29:27 client-18 kernel: Lustre: DEBUG MARKER: Using TIMEOUT=20
May 17 17:29:27 client-18 xinetd[3390]: EXIT: shell status=0 pid=15965 duration=0(sec)
May 17 17:29:32 client-18 xinetd[3390]: START: shell pid=16000 from=192.168.4.5
May 17 17:29:32 client-18 rshd[16001]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "cat /proc/mounts");echo XXRETCODE:$?'
May 17 17:29:32 client-18 xinetd[3390]: EXIT: shell status=0 pid=16000 duration=0(sec)
May 17 17:29:33 client-18 xinetd[3390]: START: shell pid=16035 from=192.168.4.5
May 17 17:29:33 client-18 rshd[16036]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl get_param -n osc.*OST*-osc-[^M][^D][^T]*.srpc_info");echo XXRETCODE:$?'
May 17 17:29:33 client-18 xinetd[3390]: EXIT: shell status=0 pid=16035 duration=0(sec)
May 17 17:29:33 client-18 xinetd[3390]: START: shell pid=16070 from=192.168.4.5
May 17 17:29:33 client-18 rshd[16071]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl get_param -n mdc.*-MDT*-mdc-*.srpc_info");echo XXRETCODE:$?'
May 17 17:29:33 client-18 xinetd[3390]: EXIT: shell status=0 pid=16070 duration=0(sec)
May 17 17:29:33 client-18 xinetd[3390]: START: shell pid=16105 from=192.168.4.5
May 17 17:29:33 client-18 rshd[16106]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-ost-single test 0a: target handle mismatch \(bug 5317\) 17:29:33 ============================= 17:29:33 \(1305678573\)");echo XXRETCODE:$?'
May 17 17:29:33 client-18 kernel: Lustre: DEBUG MARKER: == replay-ost-single test 0a: target handle mismatch (bug 5317) 17:29:33 ============================= 17:29:33 (1305678573)
May 17 17:29:33 client-18 xinetd[3390]: EXIT: shell status=0 pid=16105 duration=0(sec)
May 17 17:29:44 client-18 xinetd[3390]: START: shell pid=16140 from=192.168.4.5
May 17 17:29:44 client-18 rshd[16141]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 17:29:44 client-18 xinetd[3390]: EXIT: shell status=0 pid=16140 duration=0(sec)
May 17 17:29:44 client-18 xinetd[3390]: START: shell pid=16176 from=192.168.4.5
May 17 17:29:44 client-18 rshd[16177]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 17:29:44 client-18 xinetd[3390]: EXIT: shell status=0 pid=16176 duration=0(sec)
May 17 17:29:44 client-18 xinetd[3390]: START: shell pid=16213 from=192.168.4.5
May 17 17:29:44 client-18 rshd[16214]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-ost-single test 0b: empty replay =========================================================== 17:29:44 \(1305678584\)");echo XXRETCODE:$?'
May 17 17:29:44 client-18 kernel: Lustre: DEBUG MARKER: == replay-ost-single test 0b: empty replay =========================================================== 17:29:44 (1305678584)
May 17 17:29:44 client-18 xinetd[3390]: EXIT: shell status=0 pid=16213 duration=0(sec)
May 17 17:29:56 client-18 kernel: Lustre: 8886:0:(import.c:529:import_select_connection()) lustre-OST0000-osc-ffff810310746c00: tried all connections, increasing latency to 6s
May 17 17:29:58 client-18 xinetd[3390]: START: shell pid=16252 from=192.168.4.5
May 17 17:29:58 client-18 rshd[16253]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 17:31:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=16252 duration=66(sec)
May 17 17:31:04 client-18 xinetd[3390]: START: shell pid=16286 from=192.168.4.5
May 17 17:31:04 client-18 rshd[16287]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 17:31:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=16286 duration=0(sec)
May 17 17:31:04 client-18 xinetd[3390]: START: shell pid=16322 from=192.168.4.5
May 17 17:31:04 client-18 rshd[16323]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 17:31:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=16322 duration=0(sec)
May 17 17:31:04 client-18 xinetd[3390]: START: shell pid=16359 from=192.168.4.5
May 17 17:31:04 client-18 rshd[16360]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-ost-single test 1: touch =================================================================== 17:31:04 \(1305678664\)");echo XXRETCODE:$?'
May 17 17:31:04 client-18 kernel: Lustre: DEBUG MARKER: == replay-ost-single test 1: touch =================================================================== 17:31:04 (1305678664)
May 17 17:31:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=16359 duration=0(sec)
May 17 17:31:18 client-18 xinetd[3390]: START: shell pid=16398 from=192.168.4.5
May 17 17:31:18 client-18 rshd[16399]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 17:31:24 client-18 xinetd[3390]: EXIT: shell status=0 pid=16398 duration=6(sec)
May 17 17:31:24 client-18 xinetd[3390]: START: shell pid=16432 from=192.168.4.5
May 17 17:31:24 client-18 rshd[16433]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 17:31:24 client-18 xinetd[3390]: EXIT: shell status=0 pid=16432 duration=0(sec)
May 17 17:31:24 client-18 xinetd[3390]: START: shell pid=16468 from=192.168.4.5
May 17 17:31:24 client-18 rshd[16469]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 17:31:24 client-18 xinetd[3390]: EXIT: shell status=0 pid=16468 duration=0(sec)
May 17 17:31:24 client-18 xinetd[3390]: START: shell pid=16505 from=192.168.4.5
May 17 17:31:24 client-18 rshd[16506]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-ost-single test 2: \|x\| 10 open\(O_CREAT\)s =================================================== 17:31:24 \(1305678684\)");echo XXRETCODE:$?'
May 17 17:31:24 client-18 kernel: Lustre: DEBUG MARKER: == replay-ost-single test 2: |x| 10 open(O_CREAT)s =================================================== 17:31:24 (1305678684)
May 17 17:31:24 client-18 xinetd[3390]: EXIT: shell status=0 pid=16505 duration=0(sec)
May 17 17:31:38 client-18 xinetd[3390]: START: shell pid=16540 from=192.168.4.5
May 17 17:31:38 client-18 rshd[16541]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 17:31:44 client-18 xinetd[3390]: EXIT: shell status=0 pid=16540 duration=6(sec)
May 17 17:31:44 client-18 xinetd[3390]: START: shell pid=16574 from=192.168.4.5
May 17 17:31:44 client-18 rshd[16575]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 17:31:44 client-18 xinetd[3390]: EXIT: shell status=0 pid=16574 duration=0(sec)
May 17 17:31:44 client-18 xinetd[3390]: START: shell pid=16610 from=192.168.4.5
May 17 17:31:44 client-18 rshd[16611]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 17:31:44 client-18 xinetd[3390]: EXIT: shell status=0 pid=16610 duration=0(sec)
May 17 17:31:44 client-18 xinetd[3390]: START: shell pid=16647 from=192.168.4.5
May 17 17:31:44 client-18 rshd[16648]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-ost-single test 3: Fail OST during write, with verification ================================ 17:31:44 \(1305678704\)");echo XXRETCODE:$?'
May 17 17:31:44 client-18 kernel: Lustre: DEBUG MARKER: == replay-ost-single test 3: Fail OST during write, with verification ================================ 17:31:44 (1305678704)
May 17 17:31:44 client-18 xinetd[3390]: EXIT: shell status=0 pid=16647 duration=0(sec)
May 17 17:31:57 client-18 xinetd[3390]: START: shell pid=16682 from=192.168.4.5
May 17 17:31:58 client-18 rshd[16683]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 17:32:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=16682 duration=7(sec)
May 17 17:32:04 client-18 xinetd[3390]: START: shell pid=16716 from=192.168.4.5
May 17 17:32:04 client-18 rshd[16717]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 17:32:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=16716 duration=0(sec)
May 17 17:32:04 client-18 xinetd[3390]: START: shell pid=16752 from=192.168.4.5
May 17 17:32:04 client-18 rshd[16753]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 17:32:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=16752 duration=0(sec)
May 17 17:32:04 client-18 xinetd[3390]: START: shell pid=16789 from=192.168.4.5
May 17 17:32:05 client-18 rshd[16790]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-ost-single test 4: Fail OST during read, with verification ================================= 17:32:04 \(1305678724\)");echo XXRETCODE:$?'
May 17 17:32:05 client-18 kernel: Lustre: DEBUG MARKER: == replay-ost-single test 4: Fail OST during read, with verification ================================= 17:32:04 (1305678724)
May 17 17:32:05 client-18 xinetd[3390]: EXIT: shell status=0 pid=16789 duration=1(sec)
May 17 17:32:11 client-18 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.129@o2ib. The obd_ping operation failed with -107
May 17 17:32:11 client-18 kernel: LustreError: Skipped 17 previous similar messages
May 17 17:32:19 client-18 xinetd[3390]: START: shell pid=16824 from=192.168.4.5
May 17 17:32:19 client-18 rshd[16825]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 17:32:24 client-18 kernel: Lustre: lustre-OST0000-osc-ffff810310746c00: Connection restored to service lustre-OST0000 using nid 192.168.4.129@o2ib.
May 17 17:32:24 client-18 kernel: Lustre: Skipped 11 previous similar messages
May 17 17:32:24 client-18 xinetd[3390]: EXIT: shell status=0 pid=16824 duration=5(sec)
May 17 17:32:24 client-18 xinetd[3390]: START: shell pid=16858 from=192.168.4.5
May 17 17:32:24 client-18 rshd[16859]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 17:32:24 client-18 xinetd[3390]: EXIT: shell status=0 pid=16858 duration=0(sec)
May 17 17:32:24 client-18 xinetd[3390]: START: shell pid=16894 from=192.168.4.5
May 17 17:32:24 client-18 rshd[16895]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 17:32:24 client-18 xinetd[3390]: EXIT: shell status=0 pid=16894 duration=0(sec)
May 17 17:32:24 client-18 xinetd[3390]: START: shell pid=16931 from=192.168.4.5
May 17 17:32:24 client-18 rshd[16932]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-ost-single test 5: Fail OST during iozone ================================================== 17:32:24 \(1305678744\)");echo XXRETCODE:$?'
May 17 17:32:24 client-18 kernel: Lustre: DEBUG MARKER: == replay-ost-single test 5: Fail OST during iozone ================================================== 17:32:24 (1305678744)
May 17 17:32:24 client-18 xinetd[3390]: EXIT: shell status=0 pid=16931 duration=0(sec)
May 17 17:32:46 client-18 xinetd[3390]: START: shell pid=16970 from=192.168.4.5
May 17 17:32:46 client-18 rshd[16971]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 17:32:54 client-18 xinetd[3390]: EXIT: shell status=0 pid=16970 duration=8(sec)
May 17 17:42:42 client-18 xinetd[3390]: START: shell pid=17036 from=192.168.4.5
May 17 17:42:42 client-18 rshd[17037]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark iozone rc=0");echo XXRETCODE:$?'
May 17 17:42:42 client-18 kernel: Lustre: DEBUG MARKER: iozone rc=0
May 17 17:42:42 client-18 xinetd[3390]: EXIT: shell status=0 pid=17036 duration=0(sec)
May 17 17:42:42 client-18 xinetd[3390]: START: shell pid=17071 from=192.168.4.5
May 17 17:42:42 client-18 rshd[17072]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 17:42:42 client-18 xinetd[3390]: EXIT: shell status=0 pid=17071 duration=0(sec)
May 17 17:42:42 client-18 xinetd[3390]: START: shell pid=17107 from=192.168.4.5
May 17 17:42:42 client-18 rshd[17108]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 17:42:42 client-18 xinetd[3390]: EXIT: shell status=0 pid=17107 duration=0(sec)
May 17 17:42:42 client-18 xinetd[3390]: START: shell pid=17144 from=192.168.4.5
May 17 17:42:42 client-18 rshd[17145]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-ost-single test 6: Fail OST before obd_destroy ============================================= 17:42:42 \(1305679362\)");echo XXRETCODE:$?'
May 17 17:42:42 client-18 kernel: Lustre: DEBUG MARKER: == replay-ost-single test 6: Fail OST before obd_destroy ============================================= 17:42:42 (1305679362)
May 17 17:42:42 client-18 xinetd[3390]: EXIT: shell status=0 pid=17144 duration=0(sec)
May 17 17:42:48 client-18 xinetd[3390]: START: shell pid=17179 from=192.168.4.5
May 17 17:42:48 client-18 rshd[17180]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark before: 56540232 after_dd: 56535112");echo XXRETCODE:$?'
May 17 17:42:48 client-18 kernel: Lustre: DEBUG MARKER: before: 56540232 after_dd: 56535112
May 17 17:42:48 client-18 xinetd[3390]: EXIT: shell status=0 pid=17179 duration=0(sec)
May 17 17:42:56 client-18 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.129@o2ib. The obd_ping operation failed with -107
May 17 17:42:56 client-18 kernel: LustreError: Skipped 4 previous similar messages
May 17 17:42:56 client-18 kernel: Lustre: lustre-OST0000-osc-ffff810310746c00: Connection to service lustre-OST0000 via nid 192.168.4.129@o2ib was lost; in progress operations using this service will wait for recovery to complete.
May 17 17:42:56 client-18 kernel: Lustre: Skipped 8 previous similar messages
May 17 17:43:06 client-18 kernel: Lustre: 8886:0:(import.c:529:import_select_connection()) lustre-OST0000-osc-ffff810310746c00: tried all connections, increasing latency to 6s
May 17 17:43:06 client-18 kernel: Lustre: 8886:0:(import.c:529:import_select_connection()) Skipped 9 previous similar messages
May 17 17:43:09 client-18 xinetd[3390]: START: shell pid=17214 from=192.168.4.5
May 17 17:43:09 client-18 rshd[17215]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 17:43:16 client-18 kernel: Lustre: lustre-OST0000-osc-ffff810310746c00: Connection restored to service lustre-OST0000 using nid 192.168.4.129@o2ib.
May 17 17:43:16 client-18 kernel: Lustre: Skipped 1 previous similar message
May 17 17:43:16 client-18 xinetd[3390]: EXIT: shell status=0 pid=17214 duration=7(sec)
May 17 17:43:19 client-18 xinetd[3390]: START: shell pid=17248 from=192.168.4.5
May 17 17:43:19 client-18 rshd[17249]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark before: 56540232 after: 56540232");echo XXRETCODE:$?'
May 17 17:43:20 client-18 kernel: Lustre: DEBUG MARKER: before: 56540232 after: 56540232
May 17 17:43:20 client-18 xinetd[3390]: EXIT: shell status=0 pid=17248 duration=1(sec)
May 17 17:43:20 client-18 xinetd[3390]: START: shell pid=17283 from=192.168.4.5
May 17 17:43:20 client-18 rshd[17284]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 17:43:20 client-18 xinetd[3390]: EXIT: shell status=0 pid=17283 duration=0(sec)
May 17 17:43:20 client-18 xinetd[3390]: START: shell pid=17319 from=192.168.4.5
May 17 17:43:20 client-18 rshd[17320]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 17:43:20 client-18 xinetd[3390]: EXIT: shell status=0 pid=17319 duration=0(sec)
May 17 17:43:20 client-18 xinetd[3390]: START: shell pid=17356 from=192.168.4.5
May 17 17:43:20 client-18 rshd[17357]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-ost-single test 7: Fail OST before obd_destroy ============================================= 17:43:20 \(1305679400\)");echo XXRETCODE:$?'
May 17 17:43:20 client-18 kernel: Lustre: DEBUG MARKER: == replay-ost-single test 7: Fail OST before obd_destroy ============================================= 17:43:20 (1305679400)
May 17 17:43:20 client-18 xinetd[3390]: EXIT: shell status=0 pid=17356 duration=0(sec)
May 17 17:43:28 client-18 xinetd[3390]: START: shell pid=17391 from=192.168.4.5
May 17 17:43:28 client-18 rshd[17392]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark before: 56540232 after_dd: 56535112");echo XXRETCODE:$?'
May 17 17:43:28 client-18 kernel: Lustre: DEBUG MARKER: before: 56540232 after_dd: 56535112
May 17 17:43:28 client-18 xinetd[3390]: EXIT: shell status=0 pid=17391 duration=0(sec)
May 17 17:43:28 client-18 xinetd[3390]: START: shell pid=17426 from=192.168.4.5
May 17 17:43:28 client-18 rshd[17427]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 17:43:28 client-18 xinetd[3390]: EXIT: shell status=0 pid=17426 duration=0(sec)
May 17 17:43:42 client-18 xinetd[3390]: START: shell pid=17469 from=192.168.4.5
May 17 17:43:42 client-18 rshd[17470]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 17:43:45 client-18 xinetd[3390]: EXIT: shell status=0 pid=17469 duration=3(sec)
May 17 17:43:48 client-18 xinetd[3390]: START: shell pid=17503 from=192.168.4.5
May 17 17:43:48 client-18 rshd[17504]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark before: 56540232 after: 56540232");echo XXRETCODE:$?'
May 17 17:43:48 client-18 kernel: Lustre: DEBUG MARKER: before: 56540232 after: 56540232
May 17 17:43:48 client-18 xinetd[3390]: EXIT: shell status=0 pid=17503 duration=0(sec)
May 17 17:43:48 client-18 xinetd[3390]: START: shell pid=17538 from=192.168.4.5
May 17 17:43:48 client-18 rshd[17539]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 17:43:49 client-18 xinetd[3390]: EXIT: shell status=0 pid=17538 duration=1(sec)
May 17 17:43:49 client-18 xinetd[3390]: START: shell pid=17574 from=192.168.4.5
May 17 17:43:49 client-18 rshd[17575]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 17:43:49 client-18 xinetd[3390]: EXIT: shell status=0 pid=17574 duration=0(sec)
May 17 17:43:49 client-18 xinetd[3390]: START: shell pid=17611 from=192.168.4.5
May 17 17:43:49 client-18 rshd[17612]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-ost-single replay-ost-single.sh test complete, duration 933 sec ============================ 17:43:49 \(1305679429\)");echo XXRETCODE:$?'
May 17 17:43:49 client-18 kernel: Lustre: DEBUG MARKER: == replay-ost-single replay-ost-single.sh test complete, duration 933 sec ============================ 17:43:49 (1305679429)
May 17 17:43:49 client-18 xinetd[3390]: EXIT: shell status=0 pid=17611 duration=0(sec)
May 17 17:43:49 client-18 xinetd[3390]: START: shell pid=17646 from=192.168.4.5
May 17 17:43:49 client-18 rshd[17647]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:-f; lsof -t /mnt/lustre || need_kill=no; if [ x-f != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount -f /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre is still busy, wait one second && sleep 1;     umount -f /mnt/lustre; fi fi");echo XXRETCODE:$?'
May 17 17:43:49 client-18 kernel: Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
May 17 17:43:49 client-18 kernel: Lustre: Skipped 5 previous similar messages
May 17 17:43:49 client-18 kernel: LustreError: 17688:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway
May 17 17:43:49 client-18 kernel: LustreError: 17688:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108
May 17 17:43:49 client-18 kernel: Lustre: client ffff810310746c00 umount complete
May 17 17:43:49 client-18 xinetd[3390]: EXIT: shell status=0 pid=17646 duration=0(sec)
May 17 17:43:49 client-18 xinetd[3390]: START: shell pid=17690 from=192.168.4.5
May 17 17:43:49 client-18 rshd[17691]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:-f; lsof -t /mnt/lustre2 || need_kill=no; if [ x-f != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre2 | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount -f /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre2 is still busy, wait one second && sleep 1;     umount -f /mnt/lustre2; fi fi");echo XXRETCODE:$?'
May 17 17:43:49 client-18 xinetd[3390]: EXIT: shell status=0 pid=17690 duration=0(sec)
May 17 17:44:19 client-18 xinetd[3390]: START: shell pid=17727 from=192.168.4.5
May 17 17:44:19 client-18 rshd[17728]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark -----============= acceptance-small: replay-dual ============----- Tue May 17 17:44:18 PDT 2011");echo XXRETCODE:$?'
May 17 17:44:19 client-18 kernel: Lustre: DEBUG MARKER: -----============= acceptance-small: replay-dual ============----- Tue May 17 17:44:18 PDT 2011
May 17 17:44:19 client-18 xinetd[3390]: EXIT: shell status=0 pid=17727 duration=0(sec)
May 17 17:44:19 client-18 xinetd[3390]: START: shell pid=17762 from=192.168.4.5
May 17 17:44:19 client-18 rshd[17763]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark excepting tests: 15c");echo XXRETCODE:$?'
May 17 17:44:19 client-18 kernel: Lustre: DEBUG MARKER: excepting tests: 15c
May 17 17:44:19 client-18 xinetd[3390]: EXIT: shell status=0 pid=17762 duration=0(sec)
May 17 17:44:19 client-18 xinetd[3390]: START: shell pid=17797 from=192.168.4.5
May 17 17:44:19 client-18 rshd[17798]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:; lsof -t /mnt/lustre || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount  /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre is still busy, wait one second && sleep 1;     umount  /mnt/lustre; fi fi");echo XXRETCODE:$?'
May 17 17:44:19 client-18 xinetd[3390]: EXIT: shell status=0 pid=17797 duration=0(sec)
May 17 17:44:19 client-18 xinetd[3390]: START: shell pid=17834 from=192.168.4.5
May 17 17:44:19 client-18 rshd[17835]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:; lsof -t /mnt/lustre2 || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre2 | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount  /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre2 is still busy, wait one second && sleep 1;     umount  /mnt/lustre2; fi fi");echo XXRETCODE:$?'
May 17 17:44:19 client-18 xinetd[3390]: EXIT: shell status=0 pid=17834 duration=0(sec)
May 17 17:45:19 client-18 xinetd[3390]: START: shell pid=17875 from=192.168.4.5
May 17 17:45:19 client-18 rshd[17876]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); mpts=\$(mount | grep -w -c /mnt/lustre); if [ \$running -ne \$mpts ]; then     echo \$(hostname) env are INSANE!;     exit 1; fi");echo XXRETCODE:$?'
May 17 17:45:19 client-18 xinetd[3390]: EXIT: shell status=0 pid=17875 duration=0(sec)
May 17 17:45:19 client-18 xinetd[3390]: START: shell pid=17915 from=192.168.4.5
May 17 17:45:19 client-18 rshd[17916]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); mpts=\$(mount | grep -w -c /mnt/lustre2); if [ \$running -ne \$mpts ]; then     echo \$(hostname) env are INSANE!;     exit 1; fi");echo XXRETCODE:$?'
May 17 17:45:19 client-18 xinetd[3390]: EXIT: shell status=0 pid=17915 duration=0(sec)
May 17 17:45:28 client-18 xinetd[3390]: START: shell pid=17955 from=192.168.4.5
May 17 17:45:28 client-18 rshd[17956]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c " running=\$(mount | grep -c /mnt/lustre' '); rc=0; if [ \$running -eq 0 ] ; then     mkdir -p /mnt/lustre;     mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre;     rc=\$?; fi; exit \$rc");echo XXRETCODE:$?'
May 17 17:45:29 client-18 kernel: Lustre: 17995:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null
May 17 17:45:29 client-18 kernel: Lustre: 17995:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 7 previous similar messages
May 17 17:45:29 client-18 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'.
May 17 17:45:29 client-18 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import
May 17 17:45:29 client-18 kernel: Lustre: 17995:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff8102f4b82000->192.168.4.128@o2ib netid 50000: select flavor null
May 17 17:45:29 client-18 kernel: Lustre: Client lustre-client has started
May 17 17:45:29 client-18 xinetd[3390]: EXIT: shell status=0 pid=17955 duration=1(sec)
May 17 17:45:29 client-18 xinetd[3390]: START: shell pid=18008 from=192.168.4.5
May 17 17:45:29 client-18 rshd[18009]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "mount | grep -w /mnt/lustre");echo XXRETCODE:$?'
May 17 17:45:29 client-18 xinetd[3390]: EXIT: shell status=0 pid=18008 duration=0(sec)
May 17 17:45:29 client-18 xinetd[3390]: START: shell pid=18045 from=192.168.4.5
May 17 17:45:29 client-18 rshd[18046]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param debug=\"-1\";         lctl set_param subsystem_debug=\"0xffb7e3ff\";         lctl set_param debug_mb=32;");echo XXRETCODE:$?'
May 17 17:45:29 client-18 kernel: Lustre: 18080:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release.
May 17 17:45:29 client-18 kernel: Lustre: 18080:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message
May 17 17:45:29 client-18 kernel: Lustre: 18081:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release.
May 17 17:45:29 client-18 xinetd[3390]: EXIT: shell status=0 pid=18045 duration=0(sec)
May 17 17:45:29 client-18 xinetd[3390]: START: shell pid=18082 from=192.168.4.5
May 17 17:45:29 client-18 rshd[18083]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c " running=\$(mount | grep -c /mnt/lustre2' '); rc=0; if [ \$running -eq 0 ] ; then     mkdir -p /mnt/lustre2;     mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre2;     rc=\$?; fi; exit \$rc");echo XXRETCODE:$?'
May 17 17:45:29 client-18 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'.
May 17 17:45:29 client-18 kernel: Lustre: 18122:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-OST0000-osc-ffff810321a69800->192.168.4.129@o2ib netid 50000: select flavor null
May 17 17:45:29 client-18 kernel: Lustre: 18122:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 7 previous similar messages
May 17 17:45:29 client-18 kernel: Lustre: Client lustre-client has started
May 17 17:45:29 client-18 xinetd[3390]: EXIT: shell status=0 pid=18082 duration=0(sec)
May 17 17:45:29 client-18 xinetd[3390]: START: shell pid=18125 from=192.168.4.5
May 17 17:45:29 client-18 rshd[18126]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "mount | grep -w /mnt/lustre2");echo XXRETCODE:$?'
May 17 17:45:29 client-18 xinetd[3390]: EXIT: shell status=0 pid=18125 duration=0(sec)
May 17 17:45:29 client-18 xinetd[3390]: START: shell pid=18162 from=192.168.4.5
May 17 17:45:29 client-18 rshd[18163]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param debug=\"-1\";         lctl set_param subsystem_debug=\"0xffb7e3ff\";         lctl set_param debug_mb=32;");echo XXRETCODE:$?'
May 17 17:45:29 client-18 xinetd[3390]: EXIT: shell status=0 pid=18162 duration=0(sec)
May 17 17:45:30 client-18 xinetd[3390]: START: shell pid=18199 from=192.168.4.5
May 17 17:45:30 client-18 rshd[18200]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark Using TIMEOUT=20");echo XXRETCODE:$?'
May 17 17:45:30 client-18 kernel: Lustre: DEBUG MARKER: Using TIMEOUT=20
May 17 17:45:30 client-18 xinetd[3390]: EXIT: shell status=0 pid=18199 duration=0(sec)
May 17 17:45:35 client-18 xinetd[3390]: START: shell pid=18234 from=192.168.4.5
May 17 17:45:35 client-18 rshd[18235]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "cat /proc/mounts");echo XXRETCODE:$?'
May 17 17:45:35 client-18 xinetd[3390]: EXIT: shell status=0 pid=18234 duration=0(sec)
May 17 17:45:36 client-18 xinetd[3390]: START: shell pid=18269 from=192.168.4.5
May 17 17:45:36 client-18 rshd[18270]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl get_param -n osc.*OST*-osc-[^M][^D][^T]*.srpc_info");echo XXRETCODE:$?'
May 17 17:45:36 client-18 xinetd[3390]: EXIT: shell status=0 pid=18269 duration=0(sec)
May 17 17:45:36 client-18 xinetd[3390]: START: shell pid=18304 from=192.168.4.5
May 17 17:45:36 client-18 rshd[18305]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl get_param -n mdc.*-MDT*-mdc-*.srpc_info");echo XXRETCODE:$?'
May 17 17:45:36 client-18 xinetd[3390]: EXIT: shell status=0 pid=18304 duration=0(sec)
May 17 17:45:36 client-18 xinetd[3390]: START: shell pid=18339 from=192.168.4.5
May 17 17:45:36 client-18 rshd[18340]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-dual test 0a: expired recovery with lost client ============================================ 17:45:36 \(1305679536\)");echo XXRETCODE:$?'
May 17 17:45:36 client-18 kernel: Lustre: DEBUG MARKER: == replay-dual test 0a: expired recovery with lost client ============================================ 17:45:36 (1305679536)
May 17 17:45:36 client-18 xinetd[3390]: EXIT: shell status=0 pid=18339 duration=0(sec)
May 17 17:45:36 client-18 xinetd[3390]: START: shell pid=18374 from=192.168.4.5
May 17 17:45:36 client-18 rshd[18375]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 17:45:37 client-18 xinetd[3390]: EXIT: shell status=0 pid=18374 duration=1(sec)
May 17 17:45:46 client-18 kernel: Lustre: 8884:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369101282840727 sent from MGC192.168.4.128@o2ib to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305679539] [real_sent 1305679539] [current 1305679546] [deadline 7s] [delay 0s]  req@ffff810302bf8000 x1369101282840727/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 192/192 e 0 to 1 dl 1305679546 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
May 17 17:45:46 client-18 kernel: Lustre: 8884:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
May 17 17:45:46 client-18 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail.
May 17 17:45:46 client-18 kernel: LustreError: Skipped 3 previous similar messages
May 17 17:46:07 client-18 kernel: Lustre: 8885:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf64777d to 0xfff98862bf647fe2
May 17 17:47:09 client-18 xinetd[3390]: START: shell pid=18422 from=192.168.4.5
May 17 17:47:09 client-18 rshd[18423]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 17:47:09 client-18 xinetd[3390]: EXIT: shell status=0 pid=18422 duration=0(sec)
May 17 17:47:09 client-18 xinetd[3390]: START: shell pid=18458 from=192.168.4.5
May 17 17:47:09 client-18 rshd[18459]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 17:47:09 client-18 xinetd[3390]: EXIT: shell status=0 pid=18458 duration=0(sec)
May 17 17:47:09 client-18 xinetd[3390]: START: shell pid=18495 from=192.168.4.5
May 17 17:47:09 client-18 rshd[18496]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-dual test 0b: lost client during waiting for next transno ================================== 17:47:09 \(1305679629\)");echo XXRETCODE:$?'
May 17 17:47:10 client-18 kernel: Lustre: DEBUG MARKER: == replay-dual test 0b: lost client during waiting for next transno ================================== 17:47:09 (1305679629)
May 17 17:47:10 client-18 xinetd[3390]: EXIT: shell status=0 pid=18495 duration=1(sec)
May 17 17:47:10 client-18 xinetd[3390]: START: shell pid=18530 from=192.168.4.5
May 17 17:47:10 client-18 rshd[18531]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 17:47:10 client-18 xinetd[3390]: EXIT: shell status=0 pid=18530 duration=0(sec)
May 17 17:47:29 client-18 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail.
May 17 17:47:29 client-18 kernel: Lustre: 8885:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf647fe2 to 0xfff98862bf6491fc
May 17 17:49:02 client-18 xinetd[3390]: START: shell pid=18582 from=192.168.4.5
May 17 17:49:02 client-18 rshd[18583]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 17:49:02 client-18 xinetd[3390]: EXIT: shell status=0 pid=18582 duration=0(sec)
May 17 17:49:02 client-18 xinetd[3390]: START: shell pid=18618 from=192.168.4.5
May 17 17:49:02 client-18 rshd[18619]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 17:49:02 client-18 xinetd[3390]: EXIT: shell status=0 pid=18618 duration=0(sec)
May 17 17:49:03 client-18 xinetd[3390]: START: shell pid=18655 from=192.168.4.5
May 17 17:49:03 client-18 rshd[18656]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-dual test 1: \|X\| simple create ============================================================= 17:49:02 \(1305679742\)");echo XXRETCODE:$?'
May 17 17:49:03 client-18 kernel: Lustre: DEBUG MARKER: == replay-dual test 1: |X| simple create ============================================================= 17:49:02 (1305679742)
May 17 17:49:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=18655 duration=0(sec)
May 17 17:49:03 client-18 xinetd[3390]: START: shell pid=18690 from=192.168.4.5
May 17 17:49:03 client-18 rshd[18691]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 17:49:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=18690 duration=0(sec)
May 17 17:49:21 client-18 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail.
May 17 17:49:21 client-18 kernel: Lustre: 8885:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf6491fc to 0xfff98862bf64950c
May 17 17:49:22 client-18 xinetd[3390]: START: shell pid=18731 from=192.168.4.5
May 17 17:49:22 client-18 rshd[18732]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 17:49:26 client-18 xinetd[3390]: EXIT: shell status=0 pid=18731 duration=4(sec)
May 17 17:49:26 client-18 xinetd[3390]: START: shell pid=18768 from=192.168.4.5
May 17 17:49:26 client-18 rshd[18769]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 17:49:26 client-18 xinetd[3390]: EXIT: shell status=0 pid=18768 duration=0(sec)
May 17 17:49:27 client-18 xinetd[3390]: START: shell pid=18804 from=192.168.4.5
May 17 17:49:27 client-18 rshd[18805]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 17:49:27 client-18 xinetd[3390]: EXIT: shell status=0 pid=18804 duration=0(sec)
May 17 17:49:27 client-18 xinetd[3390]: START: shell pid=18841 from=192.168.4.5
May 17 17:49:27 client-18 rshd[18842]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-dual test 2: \|X\| mkdir adir ================================================================ 17:49:27 \(1305679767\)");echo XXRETCODE:$?'
May 17 17:49:27 client-18 kernel: Lustre: DEBUG MARKER: == replay-dual test 2: |X| mkdir adir ================================================================ 17:49:27 (1305679767)
May 17 17:49:27 client-18 xinetd[3390]: EXIT: shell status=0 pid=18841 duration=0(sec)
May 17 17:49:27 client-18 xinetd[3390]: START: shell pid=18876 from=192.168.4.5
May 17 17:49:27 client-18 rshd[18877]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 17:49:27 client-18 xinetd[3390]: EXIT: shell status=0 pid=18876 duration=0(sec)
May 17 17:49:46 client-18 xinetd[3390]: START: shell pid=18915 from=192.168.4.5
May 17 17:49:46 client-18 rshd[18916]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 17:49:46 client-18 kernel: Lustre: 8885:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf64950c to 0xfff98862bf6498d9
May 17 17:49:46 client-18 xinetd[3390]: EXIT: shell status=0 pid=18915 duration=0(sec)
May 17 17:49:46 client-18 xinetd[3390]: START: shell pid=18950 from=192.168.4.5
May 17 17:49:46 client-18 rshd[18951]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 17:49:46 client-18 xinetd[3390]: EXIT: shell status=0 pid=18950 duration=0(sec)
May 17 17:49:46 client-18 xinetd[3390]: START: shell pid=18986 from=192.168.4.5
May 17 17:49:46 client-18 rshd[18987]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 17:49:47 client-18 xinetd[3390]: EXIT: shell status=0 pid=18986 duration=1(sec)
May 17 17:49:47 client-18 xinetd[3390]: START: shell pid=19023 from=192.168.4.5
May 17 17:49:47 client-18 rshd[19024]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-dual test 3: \|X\| mkdir adir, mkdir adir\/bdir =============================================== 17:49:47 \(1305679787\)");echo XXRETCODE:$?'
May 17 17:49:47 client-18 kernel: Lustre: DEBUG MARKER: == replay-dual test 3: |X| mkdir adir, mkdir adir/bdir =============================================== 17:49:47 (1305679787)
May 17 17:49:47 client-18 xinetd[3390]: EXIT: shell status=0 pid=19023 duration=0(sec)
May 17 17:49:47 client-18 xinetd[3390]: START: shell pid=19058 from=192.168.4.5
May 17 17:49:47 client-18 rshd[19059]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 17:49:47 client-18 xinetd[3390]: EXIT: shell status=0 pid=19058 duration=0(sec)
May 17 17:50:06 client-18 xinetd[3390]: START: shell pid=19101 from=192.168.4.5
May 17 17:50:06 client-18 rshd[19102]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 17:50:06 client-18 kernel: LustreError: 18730:0:(client.c:1057:ptlrpc_import_delay_req()) @@@ IMP_INVALID  req@ffff8102ec391800 x1369101282841516/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:/ffffffff/ffffffff rc 0/-1
May 17 17:50:06 client-18 kernel: Lustre: 8885:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf6498d9 to 0xfff98862bf649b48
May 17 17:50:06 client-18 kernel: LustreError: 18730:0:(client.c:1057:ptlrpc_import_delay_req()) @@@ IMP_INVALID  req@ffff8102ec391800 x1369101282841518/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:/ffffffff/ffffffff rc 0/-1
May 17 17:50:06 client-18 xinetd[3390]: EXIT: shell status=0 pid=19101 duration=0(sec)
May 17 17:50:06 client-18 xinetd[3390]: START: shell pid=19136 from=192.168.4.5
May 17 17:50:06 client-18 rshd[19137]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 17:50:06 client-18 xinetd[3390]: EXIT: shell status=0 pid=19136 duration=0(sec)
May 17 17:50:07 client-18 xinetd[3390]: START: shell pid=19172 from=192.168.4.5
May 17 17:50:07 client-18 rshd[19173]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 17:50:07 client-18 xinetd[3390]: EXIT: shell status=0 pid=19172 duration=0(sec)
May 17 17:50:07 client-18 xinetd[3390]: START: shell pid=19209 from=192.168.4.5
May 17 17:50:07 client-18 rshd[19210]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-dual test 4: \|X\| mkdir adir \(-EEXIST\), mkdir adir\/bdir ===================================== 17:50:07 \(1305679807\)");echo XXRETCODE:$?'
May 17 17:50:07 client-18 kernel: Lustre: DEBUG MARKER: == replay-dual test 4: |X| mkdir adir (-EEXIST), mkdir adir/bdir ===================================== 17:50:07 (1305679807)
May 17 17:50:07 client-18 xinetd[3390]: EXIT: shell status=0 pid=19209 duration=0(sec)
May 17 17:50:07 client-18 xinetd[3390]: START: shell pid=19244 from=192.168.4.5
May 17 17:50:07 client-18 rshd[19245]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 17:50:07 client-18 xinetd[3390]: EXIT: shell status=0 pid=19244 duration=0(sec)
May 17 17:50:26 client-18 xinetd[3390]: START: shell pid=19283 from=192.168.4.5
May 17 17:50:26 client-18 rshd[19284]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 17:50:26 client-18 kernel: LustreError: 18730:0:(client.c:1057:ptlrpc_import_delay_req()) @@@ IMP_INVALID  req@ffff8102f898c000 x1369101282841587/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:/ffffffff/ffffffff rc 0/-1
May 17 17:50:26 client-18 kernel: LustreError: 18730:0:(client.c:1057:ptlrpc_import_delay_req()) Skipped 2 previous similar messages
May 17 17:50:26 client-18 kernel: Lustre: 8885:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf649b48 to 0xfff98862bf649ed6
May 17 17:50:26 client-18 xinetd[3390]: EXIT: shell status=0 pid=19283 duration=0(sec)
May 17 17:50:26 client-18 xinetd[3390]: START: shell pid=19318 from=192.168.4.5
May 17 17:50:26 client-18 rshd[19319]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 17:50:26 client-18 xinetd[3390]: EXIT: shell status=0 pid=19318 duration=0(sec)
May 17 17:50:26 client-18 xinetd[3390]: START: shell pid=19354 from=192.168.4.5
May 17 17:50:27 client-18 rshd[19355]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 17:50:27 client-18 xinetd[3390]: EXIT: shell status=0 pid=19354 duration=1(sec)
May 17 17:50:27 client-18 xinetd[3390]: START: shell pid=19391 from=192.168.4.5
May 17 17:50:27 client-18 rshd[19392]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-dual test 5: open, unlink \|X\| close ======================================================== 17:50:27 \(1305679827\)");echo XXRETCODE:$?'
May 17 17:50:27 client-18 kernel: Lustre: DEBUG MARKER: == replay-dual test 5: open, unlink |X| close ======================================================== 17:50:27 (1305679827)
May 17 17:50:27 client-18 xinetd[3390]: EXIT: shell status=0 pid=19391 duration=0(sec)
May 17 17:50:27 client-18 xinetd[3390]: START: shell pid=19426 from=192.168.4.5
May 17 17:50:27 client-18 rshd[19427]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 17:50:27 client-18 xinetd[3390]: EXIT: shell status=0 pid=19426 duration=0(sec)
May 17 17:50:46 client-18 xinetd[3390]: START: shell pid=19465 from=192.168.4.5
May 17 17:50:46 client-18 rshd[19466]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 17:50:46 client-18 kernel: LustreError: 18730:0:(client.c:1057:ptlrpc_import_delay_req()) @@@ IMP_INVALID  req@ffff8102ed5c9000 x1369101282841657/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:/ffffffff/ffffffff rc 0/-1
May 17 17:50:46 client-18 kernel: LustreError: 18730:0:(client.c:1057:ptlrpc_import_delay_req()) Skipped 3 previous similar messages
May 17 17:50:46 client-18 kernel: Lustre: 8885:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf649ed6 to 0xfff98862bf64a272
May 17 17:50:50 client-18 xinetd[3390]: EXIT: shell status=0 pid=19465 duration=4(sec)
May 17 17:50:50 client-18 xinetd[3390]: START: shell pid=19500 from=192.168.4.5
May 17 17:50:50 client-18 rshd[19501]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 17:50:50 client-18 xinetd[3390]: EXIT: shell status=0 pid=19500 duration=0(sec)
May 17 17:50:50 client-18 xinetd[3390]: START: shell pid=19536 from=192.168.4.5
May 17 17:50:50 client-18 rshd[19537]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 17:50:51 client-18 xinetd[3390]: EXIT: shell status=0 pid=19536 duration=1(sec)
May 17 17:50:51 client-18 xinetd[3390]: START: shell pid=19573 from=192.168.4.5
May 17 17:50:51 client-18 rshd[19574]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-dual test 6: open1, open2, unlink \|X\| close1 [fail mds1] close2 ============================ 17:50:51 \(1305679851\)");echo XXRETCODE:$?'
May 17 17:50:51 client-18 kernel: Lustre: DEBUG MARKER: == replay-dual test 6: open1, open2, unlink |X| close1 [fail mds1] close2 ============================ 17:50:51 (1305679851)
May 17 17:50:51 client-18 xinetd[3390]: EXIT: shell status=0 pid=19573 duration=0(sec)
May 17 17:50:51 client-18 xinetd[3390]: START: shell pid=19608 from=192.168.4.5
May 17 17:50:51 client-18 rshd[19609]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 17:50:51 client-18 xinetd[3390]: EXIT: shell status=0 pid=19608 duration=0(sec)
May 17 17:51:10 client-18 xinetd[3390]: START: shell pid=19650 from=192.168.4.5
May 17 17:51:10 client-18 rshd[19651]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 17:51:11 client-18 kernel: Lustre: 8885:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf64a272 to 0xfff98862bf64a5c8
May 17 17:51:11 client-18 xinetd[3390]: EXIT: shell status=0 pid=19650 duration=1(sec)
May 17 17:51:11 client-18 xinetd[3390]: START: shell pid=19689 from=192.168.4.5
May 17 17:51:11 client-18 rshd[19690]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 17:51:11 client-18 xinetd[3390]: EXIT: shell status=0 pid=19689 duration=0(sec)
May 17 17:51:11 client-18 xinetd[3390]: START: shell pid=19725 from=192.168.4.5
May 17 17:51:11 client-18 rshd[19726]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 17:51:12 client-18 xinetd[3390]: EXIT: shell status=0 pid=19725 duration=1(sec)
May 17 17:51:12 client-18 xinetd[3390]: START: shell pid=19762 from=192.168.4.5
May 17 17:51:12 client-18 rshd[19763]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-dual test 8: replay of resent request ====================================================== 17:51:12 \(1305679872\)");echo XXRETCODE:$?'
May 17 17:51:12 client-18 kernel: Lustre: DEBUG MARKER: == replay-dual test 8: replay of resent request ====================================================== 17:51:12 (1305679872)
May 17 17:51:12 client-18 xinetd[3390]: EXIT: shell status=0 pid=19762 duration=0(sec)
May 17 17:51:12 client-18 xinetd[3390]: START: shell pid=19797 from=192.168.4.5
May 17 17:51:12 client-18 rshd[19798]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 17:51:12 client-18 xinetd[3390]: EXIT: shell status=0 pid=19797 duration=0(sec)
May 17 17:52:01 client-18 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail.
May 17 17:52:01 client-18 kernel: LustreError: Skipped 5 previous similar messages
May 17 17:52:01 client-18 kernel: Lustre: 8885:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf64a5c8 to 0xfff98862bf64a88b
May 17 17:52:02 client-18 xinetd[3390]: START: shell pid=19840 from=192.168.4.5
May 17 17:52:02 client-18 rshd[19841]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 17:52:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=19840 duration=2(sec)
May 17 17:52:04 client-18 xinetd[3390]: START: shell pid=19874 from=192.168.4.5
May 17 17:52:04 client-18 rshd[19875]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 17:52:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=19874 duration=0(sec)
May 17 17:52:04 client-18 xinetd[3390]: START: shell pid=19910 from=192.168.4.5
May 17 17:52:04 client-18 rshd[19911]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 17:52:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=19910 duration=0(sec)
May 17 17:52:04 client-18 xinetd[3390]: START: shell pid=19947 from=192.168.4.5
May 17 17:52:04 client-18 rshd[19948]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-dual test 9: resending a replayed create =================================================== 17:52:04 \(1305679924\)");echo XXRETCODE:$?'
May 17 17:52:04 client-18 kernel: Lustre: DEBUG MARKER: == replay-dual test 9: resending a replayed create =================================================== 17:52:04 (1305679924)
May 17 17:52:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=19947 duration=0(sec)
May 17 17:52:04 client-18 xinetd[3390]: START: shell pid=19982 from=192.168.4.5
May 17 17:52:04 client-18 rshd[19983]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 17:52:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=19982 duration=0(sec)
May 17 17:52:23 client-18 kernel: LustreError: 18730:0:(client.c:1057:ptlrpc_import_delay_req()) @@@ IMP_INVALID  req@ffff8102eaa1e000 x1369101282841978/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:/ffffffff/ffffffff rc 0/-1
May 17 17:52:23 client-18 kernel: LustreError: 18730:0:(client.c:1057:ptlrpc_import_delay_req()) Skipped 3 previous similar messages
May 17 17:52:23 client-18 kernel: Lustre: 8885:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf64a88b to 0xfff98862bf64ab86
May 17 17:52:24 client-18 xinetd[3390]: START: shell pid=20022 from=192.168.4.5
May 17 17:52:24 client-18 rshd[20023]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 17:53:00 client-18 kernel: LustreError: 8885:0:(client.c:2515:ptlrpc_replay_interpret()) request replay timed out, restarting recovery
May 17 17:53:00 client-18 xinetd[3390]: EXIT: shell status=0 pid=20022 duration=36(sec)
May 17 17:53:00 client-18 xinetd[3390]: START: shell pid=20063 from=192.168.4.5
May 17 17:53:00 client-18 rshd[20064]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 17:53:00 client-18 xinetd[3390]: EXIT: shell status=0 pid=20063 duration=0(sec)
May 17 17:53:00 client-18 xinetd[3390]: START: shell pid=20099 from=192.168.4.5
May 17 17:53:00 client-18 rshd[20100]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 17:53:00 client-18 xinetd[3390]: EXIT: shell status=0 pid=20099 duration=0(sec)
May 17 17:53:00 client-18 xinetd[3390]: START: shell pid=20136 from=192.168.4.5
May 17 17:53:00 client-18 rshd[20137]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-dual test 10: resending a replayed unlink ================================================== 17:53:00 \(1305679980\)");echo XXRETCODE:$?'
May 17 17:53:00 client-18 kernel: Lustre: DEBUG MARKER: == replay-dual test 10: resending a replayed unlink ================================================== 17:53:00 (1305679980)
May 17 17:53:00 client-18 xinetd[3390]: EXIT: shell status=0 pid=20136 duration=0(sec)
May 17 17:53:00 client-18 xinetd[3390]: START: shell pid=20171 from=192.168.4.5
May 17 17:53:00 client-18 rshd[20172]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 17:53:01 client-18 xinetd[3390]: EXIT: shell status=0 pid=20171 duration=1(sec)
May 17 17:53:20 client-18 xinetd[3390]: START: shell pid=20210 from=192.168.4.5
May 17 17:53:20 client-18 rshd[20211]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 17:53:20 client-18 kernel: Lustre: 8885:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf64ab86 to 0xfff98862bf64af22
May 17 17:53:20 client-18 kernel: Lustre: MGC192.168.4.128@o2ib: Connection restored to service MGS using nid 192.168.4.128@o2ib.
May 17 17:53:20 client-18 kernel: Lustre: Skipped 31 previous similar messages
May 17 17:53:20 client-18 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_statfs operation failed with -107
May 17 17:53:20 client-18 kernel: LustreError: Skipped 29 previous similar messages
May 17 17:53:20 client-18 kernel: Lustre: lustre-MDT0000-mdc-ffff8102f4b82000: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete.
May 17 17:53:20 client-18 kernel: Lustre: Skipped 21 previous similar messages
May 17 17:53:51 client-18 xinetd[3390]: EXIT: shell status=0 pid=20210 duration=31(sec)
May 17 17:53:51 client-18 xinetd[3390]: START: shell pid=20252 from=192.168.4.5
May 17 17:53:51 client-18 rshd[20253]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 17:53:51 client-18 xinetd[3390]: EXIT: shell status=0 pid=20252 duration=0(sec)
May 17 17:53:51 client-18 xinetd[3390]: START: shell pid=20288 from=192.168.4.5
May 17 17:53:51 client-18 rshd[20289]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 17:53:51 client-18 xinetd[3390]: EXIT: shell status=0 pid=20288 duration=0(sec)
May 17 17:53:51 client-18 xinetd[3390]: START: shell pid=20325 from=192.168.4.5
May 17 17:53:51 client-18 rshd[20326]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-dual test 11: both clients timeout during replay =========================================== 17:53:51 \(1305680031\)");echo XXRETCODE:$?'
May 17 17:53:51 client-18 kernel: Lustre: DEBUG MARKER: == replay-dual test 11: both clients timeout during replay =========================================== 17:53:51 (1305680031)
May 17 17:53:51 client-18 xinetd[3390]: EXIT: shell status=0 pid=20325 duration=0(sec)
May 17 17:53:51 client-18 xinetd[3390]: START: shell pid=20360 from=192.168.4.5
May 17 17:53:51 client-18 rshd[20361]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 17:53:51 client-18 xinetd[3390]: EXIT: shell status=0 pid=20360 duration=0(sec)
May 17 17:54:10 client-18 kernel: Lustre: 8885:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf64af22 to 0xfff98862bf64b263
May 17 17:54:50 client-18 xinetd[3390]: START: shell pid=20403 from=192.168.4.5
May 17 17:54:50 client-18 rshd[20404]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 17:54:50 client-18 xinetd[3390]: EXIT: shell status=0 pid=20403 duration=0(sec)
May 17 17:54:50 client-18 xinetd[3390]: START: shell pid=20439 from=192.168.4.5
May 17 17:54:50 client-18 rshd[20440]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 17:54:50 client-18 xinetd[3390]: EXIT: shell status=0 pid=20439 duration=0(sec)
May 17 17:54:50 client-18 xinetd[3390]: START: shell pid=20476 from=192.168.4.5
May 17 17:54:50 client-18 rshd[20477]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-dual test 12: open resend timeout ========================================================== 17:54:50 \(1305680090\)");echo XXRETCODE:$?'
May 17 17:54:50 client-18 kernel: Lustre: DEBUG MARKER: == replay-dual test 12: open resend timeout ========================================================== 17:54:50 (1305680090)
May 17 17:54:50 client-18 xinetd[3390]: EXIT: shell status=0 pid=20476 duration=0(sec)
May 17 17:54:50 client-18 xinetd[3390]: START: shell pid=20511 from=192.168.4.5
May 17 17:54:50 client-18 rshd[20512]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 17:54:51 client-18 xinetd[3390]: EXIT: shell status=0 pid=20511 duration=1(sec)
May 17 17:55:10 client-18 kernel: Lustre: 8885:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf64b263 to 0xfff98862bf64b676
May 17 17:55:10 client-18 xinetd[3390]: START: shell pid=20555 from=192.168.4.5
May 17 17:55:10 client-18 rshd[20556]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 17:55:18 client-18 xinetd[3390]: EXIT: shell status=0 pid=20555 duration=8(sec)
May 17 17:55:19 client-18 xinetd[3390]: START: shell pid=20592 from=192.168.4.5
May 17 17:55:19 client-18 rshd[20593]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 17:55:19 client-18 xinetd[3390]: EXIT: shell status=0 pid=20592 duration=0(sec)
May 17 17:55:19 client-18 xinetd[3390]: START: shell pid=20628 from=192.168.4.5
May 17 17:55:19 client-18 rshd[20629]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 17:55:19 client-18 xinetd[3390]: EXIT: shell status=0 pid=20628 duration=0(sec)
May 17 17:55:19 client-18 xinetd[3390]: START: shell pid=20665 from=192.168.4.5
May 17 17:55:19 client-18 rshd[20666]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-dual test 13: close resend timeout ========================================================= 17:55:19 \(1305680119\)");echo XXRETCODE:$?'
May 17 17:55:19 client-18 kernel: Lustre: DEBUG MARKER: == replay-dual test 13: close resend timeout ========================================================= 17:55:19 (1305680119)
May 17 17:55:19 client-18 xinetd[3390]: EXIT: shell status=0 pid=20665 duration=0(sec)
May 17 17:55:19 client-18 xinetd[3390]: START: shell pid=20700 from=192.168.4.5
May 17 17:55:19 client-18 rshd[20701]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 17:55:19 client-18 xinetd[3390]: EXIT: shell status=0 pid=20700 duration=0(sec)
May 17 17:55:38 client-18 xinetd[3390]: START: shell pid=20739 from=192.168.4.5
May 17 17:55:38 client-18 rshd[20740]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 17:55:40 client-18 kernel: Lustre: 8885:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf64b676 to 0xfff98862bf64b99b
May 17 17:55:40 client-18 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import
May 17 17:55:40 client-18 kernel: Lustre: Skipped 13 previous similar messages
May 17 17:55:40 client-18 xinetd[3390]: EXIT: shell status=0 pid=20739 duration=2(sec)
May 17 17:55:40 client-18 xinetd[3390]: START: shell pid=20774 from=192.168.4.5
May 17 17:55:40 client-18 rshd[20775]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 17:55:40 client-18 xinetd[3390]: EXIT: shell status=0 pid=20774 duration=0(sec)
May 17 17:55:40 client-18 xinetd[3390]: START: shell pid=20810 from=192.168.4.5
May 17 17:55:40 client-18 rshd[20811]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 17:55:40 client-18 xinetd[3390]: EXIT: shell status=0 pid=20810 duration=0(sec)
May 17 17:55:40 client-18 xinetd[3390]: START: shell pid=20847 from=192.168.4.5
May 17 17:55:40 client-18 rshd[20848]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-dual test 14b: delete ost orphans if gap occured in objids due to VBR ====================== 17:55:40 \(1305680140\)");echo XXRETCODE:$?'
May 17 17:55:40 client-18 kernel: Lustre: DEBUG MARKER: == replay-dual test 14b: delete ost orphans if gap occured in objids due to VBR ====================== 17:55:40 (1305680140)
May 17 17:55:40 client-18 xinetd[3390]: EXIT: shell status=0 pid=20847 duration=0(sec)
May 17 17:55:43 client-18 xinetd[3390]: START: shell pid=20882 from=192.168.4.5
May 17 17:55:43 client-18 rshd[20883]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 17:55:44 client-18 xinetd[3390]: EXIT: shell status=0 pid=20882 duration=1(sec)
May 17 17:55:48 client-18 kernel: Lustre: 8884:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369101282842467 sent from lustre-MDT0000-mdc-ffff810321a69800 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305680105] [real_sent 1305680105] [current 1305680148] [deadline 43s] [delay 0s]  req@ffff81031c09ac00 x1369101282842467/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 192/192 e 0 to 1 dl 1305680148 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
May 17 17:55:48 client-18 kernel: Lustre: 8884:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 80 previous similar messages
May 17 17:56:03 client-18 xinetd[3390]: START: shell pid=20924 from=192.168.4.5
May 17 17:56:03 client-18 rshd[20925]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 17:56:07 client-18 kernel: Lustre: 8886:0:(import.c:529:import_select_connection()) MGC192.168.4.128@o2ib: tried all connections, increasing latency to 6s
May 17 17:56:07 client-18 kernel: Lustre: 8886:0:(import.c:529:import_select_connection()) Skipped 8 previous similar messages
May 17 17:56:07 client-18 kernel: Lustre: 8885:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf64b99b to 0xfff98862bf64beb1
May 17 17:57:33 client-18 xinetd[3390]: EXIT: shell status=0 pid=20924 duration=90(sec)
May 17 17:57:37 client-18 xinetd[3390]: START: shell pid=20970 from=192.168.4.5
May 17 17:57:37 client-18 rshd[20971]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark before 2516856, after 2516856");echo XXRETCODE:$?'
May 17 17:57:37 client-18 kernel: Lustre: DEBUG MARKER: before 2516856, after 2516856
May 17 17:57:37 client-18 xinetd[3390]: EXIT: shell status=0 pid=20970 duration=0(sec)
May 17 17:57:37 client-18 xinetd[3390]: START: shell pid=21005 from=192.168.4.5
May 17 17:57:37 client-18 rshd[21006]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 17:57:37 client-18 xinetd[3390]: EXIT: shell status=0 pid=21005 duration=0(sec)
May 17 17:57:37 client-18 xinetd[3390]: START: shell pid=21041 from=192.168.4.5
May 17 17:57:37 client-18 rshd[21042]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 17:57:37 client-18 xinetd[3390]: EXIT: shell status=0 pid=21041 duration=0(sec)
May 17 17:57:37 client-18 xinetd[3390]: START: shell pid=21078 from=192.168.4.5
May 17 17:57:37 client-18 rshd[21079]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-dual test 15a: timeout waiting for lost client during replay, 1 client completes =========== 17:57:37 \(1305680257\)");echo XXRETCODE:$?'
May 17 17:57:37 client-18 kernel: Lustre: DEBUG MARKER: == replay-dual test 15a: timeout waiting for lost client during replay, 1 client completes =========== 17:57:37 (1305680257)
May 17 17:57:37 client-18 xinetd[3390]: EXIT: shell status=0 pid=21078 duration=0(sec)
May 17 17:57:37 client-18 xinetd[3390]: START: shell pid=21113 from=192.168.4.5
May 17 17:57:37 client-18 rshd[21114]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 17:57:37 client-18 xinetd[3390]: EXIT: shell status=0 pid=21113 duration=0(sec)
May 17 17:57:54 client-18 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail.
May 17 17:57:54 client-18 kernel: LustreError: Skipped 6 previous similar messages
May 17 17:57:57 client-18 xinetd[3390]: START: shell pid=21152 from=192.168.4.5
May 17 17:57:57 client-18 rshd[21153]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 17:58:09 client-18 kernel: Lustre: 8885:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf64beb1 to 0xfff98862bf64c7a9
May 17 17:59:28 client-18 xinetd[3390]: EXIT: shell status=0 pid=21152 duration=91(sec)
May 17 17:59:28 client-18 xinetd[3390]: START: shell pid=21195 from=192.168.4.5
May 17 17:59:28 client-18 rshd[21196]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 17:59:28 client-18 xinetd[3390]: EXIT: shell status=0 pid=21195 duration=0(sec)
May 17 17:59:29 client-18 xinetd[3390]: START: shell pid=21231 from=192.168.4.5
May 17 17:59:29 client-18 rshd[21232]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 17:59:29 client-18 xinetd[3390]: EXIT: shell status=0 pid=21231 duration=0(sec)
May 17 17:59:29 client-18 xinetd[3390]: START: shell pid=21268 from=192.168.4.5
May 17 17:59:29 client-18 rshd[21269]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark  SKIP: replay-dual test_15c skipping ALWAYS excluded test 15c");echo XXRETCODE:$?'
May 17 17:59:29 client-18 kernel: Lustre: DEBUG MARKER: SKIP: replay-dual test_15c skipping ALWAYS excluded test 15c
May 17 17:59:29 client-18 xinetd[3390]: EXIT: shell status=0 pid=21268 duration=0(sec)
May 17 17:59:29 client-18 xinetd[3390]: START: shell pid=21303 from=192.168.4.5
May 17 17:59:29 client-18 rshd[21304]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-dual test 16: fail MDS during recovery \(3571\) ============================================== 17:59:29 \(1305680369\)");echo XXRETCODE:$?'
May 17 17:59:29 client-18 kernel: Lustre: DEBUG MARKER: == replay-dual test 16: fail MDS during recovery (3571) ============================================== 17:59:29 (1305680369)
May 17 17:59:29 client-18 xinetd[3390]: EXIT: shell status=0 pid=21303 duration=0(sec)
May 17 17:59:29 client-18 xinetd[3390]: START: shell pid=21338 from=192.168.4.5
May 17 17:59:29 client-18 rshd[21339]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 17:59:29 client-18 xinetd[3390]: EXIT: shell status=0 pid=21338 duration=0(sec)
May 17 17:59:49 client-18 kernel: Lustre: 8885:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf64c7a9 to 0xfff98862bf64d4ad
May 17 18:00:26 client-18 kernel: Lustre: 8885:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf64d4ad to 0xfff98862bf64d62e
May 17 18:00:26 client-18 xinetd[3390]: START: shell pid=21387 from=192.168.4.5
May 17 18:00:26 client-18 rshd[21388]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 18:01:55 client-18 kernel: LustreError: 8885:0:(client.c:2570:ptlrpc_replay_interpret()) @@@ status 0, old was -19  req@ffff810310746800 x1369101282843230/t73014444058(73014444058) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 480/416 e 4 to 0 dl 1305680545 ref 2 fl Interpret:R/ffffffff/ffffffff rc 0/-1
May 17 18:01:55 client-18 xinetd[3390]: EXIT: shell status=0 pid=21387 duration=89(sec)
May 17 18:01:56 client-18 xinetd[3390]: START: shell pid=21434 from=192.168.4.5
May 17 18:01:56 client-18 rshd[21435]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 18:01:56 client-18 xinetd[3390]: EXIT: shell status=0 pid=21434 duration=0(sec)
May 17 18:01:56 client-18 xinetd[3390]: START: shell pid=21470 from=192.168.4.5
May 17 18:01:56 client-18 rshd[21471]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 18:01:56 client-18 xinetd[3390]: EXIT: shell status=0 pid=21470 duration=0(sec)
May 17 18:01:56 client-18 xinetd[3390]: START: shell pid=21507 from=192.168.4.5
May 17 18:01:56 client-18 rshd[21508]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-dual test 17: fail OST during recovery \(3571\) ============================================== 18:01:56 \(1305680516\)");echo XXRETCODE:$?'
May 17 18:01:56 client-18 kernel: Lustre: DEBUG MARKER: == replay-dual test 17: fail OST during recovery (3571) ============================================== 18:01:56 (1305680516)
May 17 18:01:56 client-18 xinetd[3390]: EXIT: shell status=0 pid=21507 duration=0(sec)
May 17 18:01:56 client-18 xinetd[3390]: START: shell pid=21542 from=192.168.4.5
May 17 18:01:56 client-18 rshd[21543]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 18:01:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=21542 duration=1(sec)
May 17 18:02:43 client-18 xinetd[3390]: START: shell pid=21585 from=192.168.4.5
May 17 18:02:43 client-18 rshd[21586]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 18:04:11 client-18 kernel: Lustre: lustre-OST0000-osc-ffff8102f4b82000: Connection restored to service lustre-OST0000 using nid 192.168.4.129@o2ib.
May 17 18:04:11 client-18 kernel: Lustre: Skipped 21 previous similar messages
May 17 18:04:11 client-18 xinetd[3390]: EXIT: shell status=0 pid=21585 duration=88(sec)
May 17 18:04:12 client-18 xinetd[3390]: START: shell pid=21623 from=192.168.4.5
May 17 18:04:12 client-18 rshd[21624]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 18:04:12 client-18 xinetd[3390]: EXIT: shell status=0 pid=21623 duration=0(sec)
May 17 18:04:12 client-18 xinetd[3390]: START: shell pid=21659 from=192.168.4.5
May 17 18:04:12 client-18 rshd[21660]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 18:04:12 client-18 xinetd[3390]: EXIT: shell status=0 pid=21659 duration=0(sec)
May 17 18:04:12 client-18 xinetd[3390]: START: shell pid=21696 from=192.168.4.5
May 17 18:04:12 client-18 rshd[21697]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-dual test 18: ldlm_handle_enqueue succeeds on evicted export \(3822\) ======================== 18:04:12 \(1305680652\)");echo XXRETCODE:$?'
May 17 18:04:12 client-18 kernel: Lustre: DEBUG MARKER: == replay-dual test 18: ldlm_handle_enqueue succeeds on evicted export (3822) ======================== 18:04:12 (1305680652)
May 17 18:04:12 client-18 xinetd[3390]: EXIT: shell status=0 pid=21696 duration=0(sec)
May 17 18:04:53 client-18 xinetd[3390]: START: shell pid=21731 from=192.168.4.5
May 17 18:04:53 client-18 rshd[21732]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 18:04:53 client-18 xinetd[3390]: EXIT: shell status=0 pid=21731 duration=0(sec)
May 17 18:04:53 client-18 xinetd[3390]: START: shell pid=21767 from=192.168.4.5
May 17 18:04:54 client-18 rshd[21768]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 18:04:54 client-18 xinetd[3390]: EXIT: shell status=0 pid=21767 duration=1(sec)
May 17 18:04:54 client-18 xinetd[3390]: START: shell pid=21804 from=192.168.4.5
May 17 18:04:54 client-18 rshd[21805]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-dual test 19: resend of open request ======================================================= 18:04:54 \(1305680694\)");echo XXRETCODE:$?'
May 17 18:04:54 client-18 kernel: Lustre: DEBUG MARKER: == replay-dual test 19: resend of open request ======================================================= 18:04:54 (1305680694)
May 17 18:04:54 client-18 xinetd[3390]: EXIT: shell status=0 pid=21804 duration=0(sec)
May 17 18:04:54 client-18 xinetd[3390]: START: shell pid=21839 from=192.168.4.5
May 17 18:04:54 client-18 rshd[21840]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 18:04:54 client-18 xinetd[3390]: EXIT: shell status=0 pid=21839 duration=0(sec)
May 17 18:06:41 client-18 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The obd_ping operation failed with -107
May 17 18:06:41 client-18 kernel: LustreError: Skipped 27 previous similar messages
May 17 18:06:41 client-18 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail.
May 17 18:06:41 client-18 kernel: LustreError: Skipped 2 previous similar messages
May 17 18:06:41 client-18 xinetd[3390]: START: shell pid=21886 from=192.168.4.5
May 17 18:06:41 client-18 kernel: Lustre: lustre-MDT0000-mdc-ffff8102f4b82000: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete.
May 17 18:06:41 client-18 kernel: Lustre: 8885:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf64d62e to 0xfff98862bf64e9fa
May 17 18:06:41 client-18 kernel: Lustre: Skipped 15 previous similar messages
May 17 18:06:41 client-18 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import
May 17 18:06:41 client-18 kernel: Lustre: Skipped 4 previous similar messages
May 17 18:06:41 client-18 rshd[21889]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 18:06:44 client-18 xinetd[3390]: EXIT: shell status=0 pid=21886 duration=3(sec)
May 17 18:06:44 client-18 xinetd[3390]: START: shell pid=21922 from=192.168.4.5
May 17 18:06:44 client-18 rshd[21923]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 18:06:44 client-18 xinetd[3390]: EXIT: shell status=0 pid=21922 duration=0(sec)
May 17 18:06:44 client-18 xinetd[3390]: START: shell pid=21958 from=192.168.4.5
May 17 18:06:44 client-18 rshd[21959]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 18:06:44 client-18 xinetd[3390]: EXIT: shell status=0 pid=21958 duration=0(sec)
May 17 18:06:44 client-18 xinetd[3390]: START: shell pid=21995 from=192.168.4.5
May 17 18:06:44 client-18 rshd[21996]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-dual test 20: recovery time is not increasing ============================================== 18:06:44 \(1305680804\)");echo XXRETCODE:$?'
May 17 18:06:44 client-18 kernel: Lustre: DEBUG MARKER: == replay-dual test 20: recovery time is not increasing ============================================== 18:06:44 (1305680804)
May 17 18:06:44 client-18 xinetd[3390]: EXIT: shell status=0 pid=21995 duration=0(sec)
May 17 18:06:44 client-18 xinetd[3390]: START: shell pid=22030 from=192.168.4.5
May 17 18:06:44 client-18 rshd[22031]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 18:06:44 client-18 xinetd[3390]: EXIT: shell status=0 pid=22030 duration=0(sec)
May 17 18:07:04 client-18 xinetd[3390]: START: shell pid=22072 from=192.168.4.5
May 17 18:07:04 client-18 rshd[22073]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 18:07:06 client-18 kernel: Lustre: 8885:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf64e9fa to 0xfff98862bf64ed57
May 17 18:07:48 client-18 kernel: Lustre: 8884:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369101282844415 sent from lustre-MDT0000-mdc-ffff810321a69800 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305680786] [real_sent 1305680786] [current 1305680868] [deadline 82s] [delay 0s]  req@ffff81033cfc4800 x1369101282844415/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 192/192 e 0 to 1 dl 1305680868 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
May 17 18:07:48 client-18 kernel: Lustre: 8884:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 30 previous similar messages
May 17 18:09:19 client-18 xinetd[3390]: EXIT: shell status=0 pid=22072 duration=135(sec)
May 17 18:09:19 client-18 xinetd[3390]: START: shell pid=22118 from=192.168.4.5
May 17 18:09:19 client-18 rshd[22119]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 18:09:19 client-18 xinetd[3390]: EXIT: shell status=0 pid=22118 duration=0(sec)
May 17 18:09:38 client-18 kernel: Lustre: 8886:0:(import.c:529:import_select_connection()) MGC192.168.4.128@o2ib: tried all connections, increasing latency to 6s
May 17 18:09:38 client-18 kernel: Lustre: 8886:0:(import.c:529:import_select_connection()) Skipped 6 previous similar messages
May 17 18:09:38 client-18 kernel: Lustre: 8885:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf64ed57 to 0xfff98862bf64f091
May 17 18:09:38 client-18 xinetd[3390]: START: shell pid=22159 from=192.168.4.5
May 17 18:09:38 client-18 rshd[22160]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 18:11:53 client-18 xinetd[3390]: EXIT: shell status=0 pid=22159 duration=135(sec)
May 17 18:11:53 client-18 xinetd[3390]: START: shell pid=22207 from=192.168.4.5
May 17 18:11:53 client-18 rshd[22208]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 18:11:53 client-18 xinetd[3390]: EXIT: shell status=0 pid=22207 duration=0(sec)
May 17 18:11:53 client-18 xinetd[3390]: START: shell pid=22243 from=192.168.4.5
May 17 18:11:53 client-18 rshd[22244]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 18:11:53 client-18 xinetd[3390]: EXIT: shell status=0 pid=22243 duration=0(sec)
May 17 18:11:53 client-18 xinetd[3390]: START: shell pid=22280 from=192.168.4.5
May 17 18:11:53 client-18 rshd[22281]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-dual test 21a: commit on sharing =========================================================== 18:11:53 \(1305681113\)");echo XXRETCODE:$?'
May 17 18:11:53 client-18 kernel: Lustre: DEBUG MARKER: == replay-dual test 21a: commit on sharing =========================================================== 18:11:53 (1305681113)
May 17 18:11:53 client-18 xinetd[3390]: EXIT: shell status=0 pid=22280 duration=0(sec)
May 17 18:12:25 client-18 kernel: Lustre: 8885:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf64f091 to 0xfff98862bf64f449
May 17 18:14:27 client-18 kernel: Lustre: lustre-MDT0000-mdc-ffff8102f4b82000: Connection restored to service lustre-MDT0000 using nid 192.168.4.128@o2ib.
May 17 18:14:27 client-18 kernel: Lustre: Skipped 11 previous similar messages
May 17 18:14:27 client-18 xinetd[3390]: START: shell pid=22328 from=192.168.4.5
May 17 18:14:27 client-18 rshd[22329]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 18:14:27 client-18 xinetd[3390]: EXIT: shell status=0 pid=22328 duration=0(sec)
May 17 18:14:28 client-18 xinetd[3390]: START: shell pid=22364 from=192.168.4.5
May 17 18:14:28 client-18 rshd[22365]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 18:14:28 client-18 xinetd[3390]: EXIT: shell status=0 pid=22364 duration=0(sec)
May 17 18:14:28 client-18 xinetd[3390]: START: shell pid=22401 from=192.168.4.5
May 17 18:14:28 client-18 rshd[22402]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-dual test 21b: commit on sharing, two clients ============================================== 18:14:28 \(1305681268\)");echo XXRETCODE:$?'
May 17 18:14:28 client-18 kernel: Lustre: DEBUG MARKER: == replay-dual test 21b: commit on sharing, two clients ============================================== 18:14:28 (1305681268)
May 17 18:14:28 client-18 xinetd[3390]: EXIT: shell status=0 pid=22401 duration=0(sec)
May 17 18:14:28 client-18 xinetd[3390]: START: shell pid=22436 from=192.168.4.5
May 17 18:14:28 client-18 rshd[22437]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:; lsof -t /mnt/lustre2 || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre2 | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount  /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre2 is still busy, wait one second && sleep 1;     umount  /mnt/lustre2; fi fi");echo XXRETCODE:$?'
May 17 18:14:28 client-18 kernel: LustreError: 22478:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway
May 17 18:14:28 client-18 kernel: LustreError: 22478:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108
May 17 18:14:28 client-18 kernel: Lustre: client ffff810321a69800 umount complete
May 17 18:14:28 client-18 xinetd[3390]: EXIT: shell status=0 pid=22436 duration=0(sec)
May 17 18:14:28 client-18 xinetd[3390]: START: shell pid=22480 from=192.168.4.5
May 17 18:14:28 client-18 rshd[22481]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c " running=\$(mount | grep -c /mnt/lustre' '); rc=0; if [ \$running -eq 0 ] ; then     mkdir -p /mnt/lustre;     mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre;     rc=\$?; fi; exit \$rc");echo XXRETCODE:$?'
May 17 18:14:28 client-18 xinetd[3390]: EXIT: shell status=0 pid=22480 duration=0(sec)
May 17 18:14:28 client-18 xinetd[3390]: START: shell pid=22518 from=192.168.4.5
May 17 18:14:28 client-18 rshd[22519]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "mount | grep -w /mnt/lustre");echo XXRETCODE:$?'
May 17 18:14:28 client-18 xinetd[3390]: EXIT: shell status=0 pid=22518 duration=0(sec)
May 17 18:14:28 client-18 xinetd[3390]: START: shell pid=22555 from=192.168.4.5
May 17 18:14:28 client-18 rshd[22556]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param debug=\"-1\";         lctl set_param subsystem_debug=\"0xffb7e3ff\";         lctl set_param debug_mb=32;");echo XXRETCODE:$?'
May 17 18:14:29 client-18 kernel: Lustre: 22590:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release.
May 17 18:14:29 client-18 kernel: Lustre: 22590:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 2 previous similar messages
May 17 18:14:29 client-18 xinetd[3390]: EXIT: shell status=0 pid=22555 duration=1(sec)
May 17 18:14:31 client-18 xinetd[3390]: START: shell pid=22592 from=192.168.4.5
May 17 18:14:31 client-18 rshd[22593]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mv /mnt/lustre/f21b-1 /mnt/lustre/f21b-2");echo XXRETCODE:$?'
May 17 18:14:31 client-18 xinetd[3390]: EXIT: shell status=0 pid=22592 duration=0(sec)
May 17 18:14:31 client-18 xinetd[3390]: START: shell pid=22627 from=192.168.4.5
May 17 18:14:31 client-18 rshd[22628]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:-f; lsof -t /mnt/lustre || need_kill=no; if [ x-f != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount -f /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre is still busy, wait one second && sleep 1;     umount -f /mnt/lustre; fi fi");echo XXRETCODE:$?'
May 17 18:14:32 client-18 kernel: Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
May 17 18:14:32 client-18 kernel: Lustre: Skipped 6 previous similar messages
May 17 18:14:32 client-18 kernel: Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request
May 17 18:14:32 client-18 kernel: LustreError: 22669:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway
May 17 18:14:32 client-18 kernel: LustreError: 22669:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108
May 17 18:14:32 client-18 kernel: Lustre: client ffff8102f4b82000 umount complete
May 17 18:14:32 client-18 xinetd[3390]: EXIT: shell status=0 pid=22627 duration=1(sec)
May 17 18:17:05 client-18 xinetd[3390]: START: shell pid=22679 from=192.168.4.5
May 17 18:17:05 client-18 rshd[22680]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c " running=\$(mount | grep -c /mnt/lustre' '); rc=0; if [ \$running -eq 0 ] ; then     mkdir -p /mnt/lustre;     mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre;     rc=\$?; fi; exit \$rc");echo XXRETCODE:$?'
May 17 18:17:05 client-18 kernel: Lustre: 22719:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null
May 17 18:17:05 client-18 kernel: Lustre: 22719:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 5 previous similar messages
May 17 18:17:05 client-18 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'.
May 17 18:17:05 client-18 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import
May 17 18:17:05 client-18 kernel: Lustre: Skipped 3 previous similar messages
May 17 18:17:05 client-18 kernel: Lustre: Client lustre-client has started
May 17 18:17:05 client-18 xinetd[3390]: EXIT: shell status=0 pid=22679 duration=0(sec)
May 17 18:17:05 client-18 xinetd[3390]: START: shell pid=22732 from=192.168.4.5
May 17 18:17:05 client-18 rshd[22733]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "mount | grep -w /mnt/lustre");echo XXRETCODE:$?'
May 17 18:17:05 client-18 xinetd[3390]: EXIT: shell status=0 pid=22732 duration=0(sec)
May 17 18:17:05 client-18 xinetd[3390]: START: shell pid=22769 from=192.168.4.5
May 17 18:17:05 client-18 rshd[22770]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param debug=\"-1\";         lctl set_param subsystem_debug=\"0xffb7e3ff\";         lctl set_param debug_mb=32;");echo XXRETCODE:$?'
May 17 18:17:05 client-18 kernel: Lustre: 22804:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release.
May 17 18:17:05 client-18 kernel: Lustre: 22804:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message
May 17 18:17:05 client-18 xinetd[3390]: EXIT: shell status=0 pid=22769 duration=0(sec)
May 17 18:17:06 client-18 xinetd[3390]: START: shell pid=22806 from=192.168.4.5
May 17 18:17:06 client-18 rshd[22807]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mv /mnt/lustre/f21b-1 /mnt/lustre/f21b-2");echo XXRETCODE:$?'
May 17 18:17:06 client-18 xinetd[3390]: EXIT: shell status=0 pid=22806 duration=0(sec)
May 17 18:17:06 client-18 xinetd[3390]: START: shell pid=22841 from=192.168.4.5
May 17 18:17:06 client-18 rshd[22842]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:-f; lsof -t /mnt/lustre || need_kill=no; if [ x-f != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount -f /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre is still busy, wait one second && sleep 1;     umount -f /mnt/lustre; fi fi");echo XXRETCODE:$?'
May 17 18:17:06 client-18 kernel: Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
May 17 18:17:06 client-18 kernel: Lustre: Skipped 5 previous similar messages
May 17 18:17:06 client-18 kernel: LustreError: 22883:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway
May 17 18:17:06 client-18 kernel: LustreError: 22883:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108
May 17 18:17:06 client-18 kernel: Lustre: client ffff810301e6b000 umount complete
May 17 18:17:06 client-18 xinetd[3390]: EXIT: shell status=0 pid=22841 duration=0(sec)
May 17 18:20:55 client-18 xinetd[3390]: START: shell pid=22897 from=192.168.4.5
May 17 18:20:55 client-18 rshd[22898]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c " running=\$(mount | grep -c /mnt/lustre' '); rc=0; if [ \$running -eq 0 ] ; then     mkdir -p /mnt/lustre;     mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre;     rc=\$?; fi; exit \$rc");echo XXRETCODE:$?'
May 17 18:20:55 client-18 kernel: Lustre: 22937:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null
May 17 18:20:55 client-18 kernel: Lustre: 22937:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 7 previous similar messages
May 17 18:20:55 client-18 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'.
May 17 18:20:56 client-18 kernel: Lustre: Client lustre-client has started
May 17 18:20:56 client-18 xinetd[3390]: EXIT: shell status=0 pid=22897 duration=1(sec)
May 17 18:20:56 client-18 xinetd[3390]: START: shell pid=22950 from=192.168.4.5
May 17 18:20:56 client-18 rshd[22951]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "mount | grep -w /mnt/lustre");echo XXRETCODE:$?'
May 17 18:20:56 client-18 xinetd[3390]: EXIT: shell status=0 pid=22950 duration=0(sec)
May 17 18:20:56 client-18 xinetd[3390]: START: shell pid=22987 from=192.168.4.5
May 17 18:20:56 client-18 rshd[22988]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param debug=\"-1\";         lctl set_param subsystem_debug=\"0xffb7e3ff\";         lctl set_param debug_mb=32;");echo XXRETCODE:$?'
May 17 18:20:56 client-18 kernel: Lustre: 23022:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release.
May 17 18:20:56 client-18 kernel: Lustre: 23022:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message
May 17 18:20:56 client-18 xinetd[3390]: EXIT: shell status=0 pid=22987 duration=0(sec)
May 17 18:20:56 client-18 xinetd[3390]: START: shell pid=23024 from=192.168.4.5
May 17 18:20:56 client-18 rshd[23025]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 18:20:56 client-18 xinetd[3390]: EXIT: shell status=0 pid=23024 duration=0(sec)
May 17 18:20:56 client-18 xinetd[3390]: START: shell pid=23060 from=192.168.4.5
May 17 18:20:56 client-18 rshd[23061]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 18:20:56 client-18 xinetd[3390]: EXIT: shell status=0 pid=23060 duration=0(sec)
May 17 18:20:56 client-18 xinetd[3390]: START: shell pid=23097 from=192.168.4.5
May 17 18:20:56 client-18 rshd[23098]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-dual replay-dual.sh test complete, duration 2197 sec ======================================= 18:20:56 \(1305681656\)");echo XXRETCODE:$?'
May 17 18:20:56 client-18 kernel: Lustre: DEBUG MARKER: == replay-dual replay-dual.sh test complete, duration 2197 sec ======================================= 18:20:56 (1305681656)
May 17 18:20:56 client-18 xinetd[3390]: EXIT: shell status=0 pid=23097 duration=0(sec)
May 17 18:20:56 client-18 xinetd[3390]: START: shell pid=23132 from=192.168.4.5
May 17 18:20:56 client-18 rshd[23133]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:-f; lsof -t /mnt/lustre || need_kill=no; if [ x-f != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount -f /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre is still busy, wait one second && sleep 1;     umount -f /mnt/lustre; fi fi");echo XXRETCODE:$?'
May 17 18:20:57 client-18 kernel: Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
May 17 18:20:57 client-18 kernel: Lustre: Skipped 6 previous similar messages
May 17 18:20:57 client-18 kernel: LustreError: 23174:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway
May 17 18:20:57 client-18 kernel: LustreError: 23174:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108
May 17 18:20:57 client-18 kernel: Lustre: client ffff8102ee809800 umount complete
May 17 18:20:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=23132 duration=1(sec)
May 17 18:20:57 client-18 xinetd[3390]: START: shell pid=23176 from=192.168.4.5
May 17 18:20:57 client-18 rshd[23177]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:-f; lsof -t /mnt/lustre2 || need_kill=no; if [ x-f != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre2 | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount -f /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre2 is still busy, wait one second && sleep 1;     umount -f /mnt/lustre2; fi fi");echo XXRETCODE:$?'
May 17 18:20:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=23176 duration=0(sec)
May 17 18:21:43 client-18 xinetd[3390]: START: shell pid=23217 from=192.168.4.5
May 17 18:21:43 client-18 rshd[23218]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark -----============= acceptance-small: replay-vbr ============----- Tue May 17 18:21:43 PDT 2011");echo XXRETCODE:$?'
May 17 18:21:43 client-18 kernel: Lustre: DEBUG MARKER: -----============= acceptance-small: replay-vbr ============----- Tue May 17 18:21:43 PDT 2011
May 17 18:21:43 client-18 xinetd[3390]: EXIT: shell status=0 pid=23217 duration=0(sec)
May 17 18:21:44 client-18 xinetd[3390]: START: shell pid=23252 from=192.168.4.5
May 17 18:21:44 client-18 rshd[23253]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark excepting tests: 12a");echo XXRETCODE:$?'
May 17 18:21:44 client-18 kernel: Lustre: DEBUG MARKER: excepting tests: 12a
May 17 18:21:44 client-18 xinetd[3390]: EXIT: shell status=0 pid=23252 duration=0(sec)
May 17 18:21:44 client-18 xinetd[3390]: START: shell pid=23287 from=192.168.4.5
May 17 18:21:44 client-18 rshd[23288]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:; lsof -t /mnt/lustre || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount  /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre is still busy, wait one second && sleep 1;     umount  /mnt/lustre; fi fi");echo XXRETCODE:$?'
May 17 18:21:44 client-18 xinetd[3390]: EXIT: shell status=0 pid=23287 duration=0(sec)
May 17 18:21:44 client-18 xinetd[3390]: START: shell pid=23324 from=192.168.4.5
May 17 18:21:44 client-18 rshd[23325]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:; lsof -t /mnt/lustre2 || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre2 | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount  /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre2 is still busy, wait one second && sleep 1;     umount  /mnt/lustre2; fi fi");echo XXRETCODE:$?'
May 17 18:21:44 client-18 xinetd[3390]: EXIT: shell status=0 pid=23324 duration=0(sec)
May 17 18:22:44 client-18 xinetd[3390]: START: shell pid=23365 from=192.168.4.5
May 17 18:22:44 client-18 rshd[23366]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); mpts=\$(mount | grep -w -c /mnt/lustre); if [ \$running -ne \$mpts ]; then     echo \$(hostname) env are INSANE!;     exit 1; fi");echo XXRETCODE:$?'
May 17 18:22:44 client-18 xinetd[3390]: EXIT: shell status=0 pid=23365 duration=0(sec)
May 17 18:22:44 client-18 xinetd[3390]: START: shell pid=23405 from=192.168.4.5
May 17 18:22:44 client-18 rshd[23406]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); mpts=\$(mount | grep -w -c /mnt/lustre2); if [ \$running -ne \$mpts ]; then     echo \$(hostname) env are INSANE!;     exit 1; fi");echo XXRETCODE:$?'
May 17 18:22:44 client-18 xinetd[3390]: EXIT: shell status=0 pid=23405 duration=0(sec)
May 17 18:22:53 client-18 xinetd[3390]: START: shell pid=23445 from=192.168.4.5
May 17 18:22:53 client-18 rshd[23446]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c " running=\$(mount | grep -c /mnt/lustre' '); rc=0; if [ \$running -eq 0 ] ; then     mkdir -p /mnt/lustre;     mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre;     rc=\$?; fi; exit \$rc");echo XXRETCODE:$?'
May 17 18:22:53 client-18 kernel: Lustre: 23485:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null
May 17 18:22:53 client-18 kernel: Lustre: 23485:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 7 previous similar messages
May 17 18:22:53 client-18 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'.
May 17 18:22:53 client-18 kernel: Lustre: Client lustre-client has started
May 17 18:22:53 client-18 xinetd[3390]: EXIT: shell status=0 pid=23445 duration=0(sec)
May 17 18:22:53 client-18 xinetd[3390]: START: shell pid=23498 from=192.168.4.5
May 17 18:22:53 client-18 rshd[23499]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "mount | grep -w /mnt/lustre");echo XXRETCODE:$?'
May 17 18:22:53 client-18 xinetd[3390]: EXIT: shell status=0 pid=23498 duration=0(sec)
May 17 18:22:53 client-18 xinetd[3390]: START: shell pid=23535 from=192.168.4.5
May 17 18:22:53 client-18 rshd[23536]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param debug=\"0x33f0406\";         lctl set_param subsystem_debug=\"0xffb7e3ff\";         lctl set_param debug_mb=32;");echo XXRETCODE:$?'
May 17 18:22:54 client-18 kernel: Lustre: 23570:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release.
May 17 18:22:54 client-18 kernel: Lustre: 23570:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message
May 17 18:22:54 client-18 xinetd[3390]: EXIT: shell status=0 pid=23535 duration=1(sec)
May 17 18:22:54 client-18 xinetd[3390]: START: shell pid=23572 from=192.168.4.5
May 17 18:22:54 client-18 rshd[23573]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c " running=\$(mount | grep -c /mnt/lustre2' '); rc=0; if [ \$running -eq 0 ] ; then     mkdir -p /mnt/lustre2;     mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre2;     rc=\$?; fi; exit \$rc");echo XXRETCODE:$?'
May 17 18:22:54 client-18 xinetd[3390]: EXIT: shell status=0 pid=23572 duration=0(sec)
May 17 18:22:54 client-18 xinetd[3390]: START: shell pid=23615 from=192.168.4.5
May 17 18:22:54 client-18 rshd[23616]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "mount | grep -w /mnt/lustre2");echo XXRETCODE:$?'
May 17 18:22:54 client-18 xinetd[3390]: EXIT: shell status=0 pid=23615 duration=0(sec)
May 17 18:22:54 client-18 xinetd[3390]: START: shell pid=23652 from=192.168.4.5
May 17 18:22:54 client-18 rshd[23653]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param debug=\"0x33f0406\";         lctl set_param subsystem_debug=\"0xffb7e3ff\";         lctl set_param debug_mb=32;");echo XXRETCODE:$?'
May 17 18:22:54 client-18 xinetd[3390]: EXIT: shell status=0 pid=23652 duration=0(sec)
May 17 18:22:54 client-18 xinetd[3390]: START: shell pid=23689 from=192.168.4.5
May 17 18:22:54 client-18 rshd[23690]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark Using TIMEOUT=20");echo XXRETCODE:$?'
May 17 18:22:54 client-18 kernel: Lustre: DEBUG MARKER: Using TIMEOUT=20
May 17 18:22:54 client-18 xinetd[3390]: EXIT: shell status=0 pid=23689 duration=0(sec)
May 17 18:23:00 client-18 xinetd[3390]: START: shell pid=23724 from=192.168.4.5
May 17 18:23:00 client-18 rshd[23725]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "cat /proc/mounts");echo XXRETCODE:$?'
May 17 18:23:00 client-18 xinetd[3390]: EXIT: shell status=0 pid=23724 duration=0(sec)
May 17 18:23:00 client-18 xinetd[3390]: START: shell pid=23759 from=192.168.4.5
May 17 18:23:00 client-18 rshd[23760]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl get_param -n osc.*OST*-osc-[^M][^D][^T]*.srpc_info");echo XXRETCODE:$?'
May 17 18:23:00 client-18 xinetd[3390]: EXIT: shell status=0 pid=23759 duration=0(sec)
May 17 18:23:01 client-18 xinetd[3390]: START: shell pid=23794 from=192.168.4.5
May 17 18:23:01 client-18 rshd[23795]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl get_param -n mdc.*-MDT*-mdc-*.srpc_info");echo XXRETCODE:$?'
May 17 18:23:01 client-18 xinetd[3390]: EXIT: shell status=0 pid=23794 duration=0(sec)
May 17 18:23:01 client-18 xinetd[3390]: START: shell pid=23829 from=192.168.4.5
May 17 18:23:01 client-18 rshd[23830]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-vbr test 1a: open and close do not change versions ========================================= 18:23:01 \(1305681781\)");echo XXRETCODE:$?'
May 17 18:23:01 client-18 kernel: Lustre: DEBUG MARKER: == replay-vbr test 1a: open and close do not change versions ========================================= 18:23:01 (1305681781)
May 17 18:23:01 client-18 xinetd[3390]: EXIT: shell status=0 pid=23829 duration=0(sec)
May 17 18:23:01 client-18 xinetd[3390]: START: shell pid=23864 from=192.168.4.5
May 17 18:23:01 client-18 rshd[23865]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 18:23:01 client-18 xinetd[3390]: EXIT: shell status=0 pid=23864 duration=0(sec)
May 17 18:23:01 client-18 xinetd[3390]: START: shell pid=23900 from=192.168.4.5
May 17 18:23:01 client-18 rshd[23901]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 18:23:01 client-18 xinetd[3390]: EXIT: shell status=0 pid=23900 duration=0(sec)
May 17 18:23:02 client-18 xinetd[3390]: START: shell pid=23937 from=192.168.4.5
May 17 18:23:02 client-18 rshd[23938]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-vbr test 1b: open \(O_CREAT\) checks version of parent ======================================= 18:23:02 \(1305681782\)");echo XXRETCODE:$?'
May 17 18:23:02 client-18 kernel: Lustre: DEBUG MARKER: == replay-vbr test 1b: open (O_CREAT) checks version of parent ======================================= 18:23:02 (1305681782)
May 17 18:23:02 client-18 xinetd[3390]: EXIT: shell status=0 pid=23937 duration=0(sec)
May 17 18:23:02 client-18 xinetd[3390]: START: shell pid=23972 from=192.168.4.5
May 17 18:23:02 client-18 rshd[23973]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mkdir -p /mnt/lustre2");echo XXRETCODE:$?'
May 17 18:23:02 client-18 xinetd[3390]: EXIT: shell status=0 pid=23972 duration=0(sec)
May 17 18:23:02 client-18 xinetd[3390]: START: shell pid=24007 from=192.168.4.5
May 17 18:23:02 client-18 rshd[24008]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre2");echo XXRETCODE:$?'
May 17 18:23:02 client-18 xinetd[3390]: EXIT: shell status=0 pid=24007 duration=0(sec)
May 17 18:23:02 client-18 xinetd[3390]: START: shell pid=24043 from=192.168.4.5
May 17 18:23:02 client-18 rshd[24044]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 18:23:02 client-18 xinetd[3390]: EXIT: shell status=0 pid=24043 duration=0(sec)
May 17 18:23:03 client-18 xinetd[3390]: START: shell pid=24082 from=192.168.4.5
May 17 18:23:03 client-18 rshd[24083]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "chmod 777 /mnt/lustre2/d0.replay-vbr/d1");echo XXRETCODE:$?'
May 17 18:23:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=24082 duration=0(sec)
May 17 18:23:03 client-18 xinetd[3390]: START: shell pid=24117 from=192.168.4.5
May 17 18:23:03 client-18 rshd[24118]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "grep -c /mnt/lustre2' ' /proc/mounts");echo XXRETCODE:$?'
May 17 18:23:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=24117 duration=0(sec)
May 17 18:23:03 client-18 xinetd[3390]: START: shell pid=24152 from=192.168.4.5
May 17 18:23:03 client-18 rshd[24153]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lsof -t /mnt/lustre2");echo XXRETCODE:$?'
May 17 18:23:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=24152 duration=0(sec)
May 17 18:23:03 client-18 xinetd[3390]: START: shell pid=24188 from=192.168.4.5
May 17 18:23:03 client-18 rshd[24189]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "umount  /mnt/lustre2 2>&1");echo XXRETCODE:$?'
May 17 18:23:03 client-18 kernel: LustreError: 24223:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway
May 17 18:23:03 client-18 kernel: LustreError: 24223:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108
May 17 18:23:03 client-18 kernel: Lustre: client ffff8102f9802000 umount complete
May 17 18:23:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=24188 duration=0(sec)
May 17 18:23:16 client-18 kernel: Lustre: 8884:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369101282845888 sent from MGC192.168.4.128@o2ib to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305681789] [real_sent 1305681789] [current 1305681796] [deadline 7s] [delay 0s]  req@ffff8102f9d20800 x1369101282845888/t0(0) o-1->MGS@MGC192.168.4.128@o2ib_0:26/25 lens 192/192 e 0 to 1 dl 1305681796 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
May 17 18:23:16 client-18 kernel: Lustre: 8884:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 25 previous similar messages
May 17 18:23:16 client-18 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail.
May 17 18:23:16 client-18 kernel: LustreError: Skipped 3 previous similar messages
May 17 18:23:16 client-18 kernel: Lustre: lustre-MDT0000-mdc-ffff8103028bc400: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete.
May 17 18:23:16 client-18 kernel: Lustre: Skipped 7 previous similar messages
May 17 18:23:26 client-18 kernel: Lustre: 8886:0:(import.c:529:import_select_connection()) MGC192.168.4.128@o2ib: tried all connections, increasing latency to 6s
May 17 18:23:26 client-18 kernel: Lustre: 8886:0:(import.c:529:import_select_connection()) Skipped 1 previous similar message
May 17 18:23:26 client-18 kernel: Lustre: 8885:0:(import.c:885:ptlrpc_connect_interpret()) MGS@MGC192.168.4.128@o2ib_0 changed server handle from 0xfff98862bf64ffcc to 0xfff98862bf6503ca
May 17 18:24:32 client-18 kernel: Lustre: lustre-MDT0000-mdc-ffff8103028bc400: Connection restored to service lustre-MDT0000 using nid 192.168.4.128@o2ib.
May 17 18:24:32 client-18 kernel: Lustre: Skipped 2 previous similar messages
May 17 18:25:03 client-18 xinetd[3390]: START: shell pid=24236 from=192.168.4.5
May 17 18:25:03 client-18 rshd[24237]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 18:25:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=24236 duration=0(sec)
May 17 18:25:03 client-18 xinetd[3390]: START: shell pid=24272 from=192.168.4.5
May 17 18:25:03 client-18 rshd[24273]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 18:25:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=24272 duration=0(sec)
May 17 18:25:03 client-18 xinetd[3390]: START: shell pid=24309 from=192.168.4.5
May 17 18:25:03 client-18 rshd[24310]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-vbr test 1c: open \(non O_CREAT\) does not checks versions =================================== 18:25:03 \(1305681903\)");echo XXRETCODE:$?'
May 17 18:25:03 client-18 kernel: Lustre: DEBUG MARKER: == replay-vbr test 1c: open (non O_CREAT) does not checks versions =================================== 18:25:03 (1305681903)
May 17 18:25:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=24309 duration=0(sec)
May 17 18:25:03 client-18 xinetd[3390]: START: shell pid=24344 from=192.168.4.5
May 17 18:25:03 client-18 rshd[24345]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mkdir -p /mnt/lustre2");echo XXRETCODE:$?'
May 17 18:25:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=24344 duration=1(sec)
May 17 18:25:04 client-18 xinetd[3390]: START: shell pid=24379 from=192.168.4.5
May 17 18:25:04 client-18 rshd[24380]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre2");echo XXRETCODE:$?'
May 17 18:25:04 client-18 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'.
May 17 18:25:04 client-18 kernel: LustreError: Skipped 1 previous similar message
May 17 18:25:04 client-18 kernel: Lustre: 24414:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff8102f8fcd800->192.168.4.128@o2ib netid 50000: select flavor null
May 17 18:25:04 client-18 kernel: Lustre: 24414:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 14 previous similar messages
May 17 18:25:04 client-18 kernel: Lustre: Client lustre-client has started
May 17 18:25:04 client-18 kernel: Lustre: Skipped 1 previous similar message
May 17 18:25:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=24379 duration=0(sec)
May 17 18:25:04 client-18 xinetd[3390]: START: shell pid=24417 from=192.168.4.5
May 17 18:25:04 client-18 rshd[24418]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param debug=\"0x33f0406\";         lctl set_param subsystem_debug=\"0xffb7e3ff\";         lctl set_param debug_mb=32");echo XXRETCODE:$?'
May 17 18:25:04 client-18 kernel: Lustre: 24452:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release.
May 17 18:25:04 client-18 kernel: Lustre: 24452:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 3 previous similar messages
May 17 18:25:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=24417 duration=0(sec)
May 17 18:25:04 client-18 xinetd[3390]: START: shell pid=24454 from=192.168.4.5
May 17 18:25:04 client-18 rshd[24455]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 18:25:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=24454 duration=0(sec)
May 17 18:25:05 client-18 xinetd[3390]: START: shell pid=24493 from=192.168.4.5
May 17 18:25:05 client-18 rshd[24494]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "chmod 0777 /mnt/lustre2/d0.replay-vbr/d1");echo XXRETCODE:$?'
May 17 18:25:05 client-18 xinetd[3390]: EXIT: shell status=0 pid=24493 duration=0(sec)
May 17 18:25:05 client-18 xinetd[3390]: START: shell pid=24528 from=192.168.4.5
May 17 18:25:05 client-18 rshd[24529]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "chmod 0666 /mnt/lustre2/d0.replay-vbr/d1/f1c");echo XXRETCODE:$?'
May 17 18:25:05 client-18 xinetd[3390]: EXIT: shell status=0 pid=24528 duration=0(sec)
May 17 18:25:08 client-18 xinetd[3390]: START: shell pid=24563 from=192.168.4.5
May 17 18:25:08 client-18 rshd[24564]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "grep -c /mnt/lustre2' ' /proc/mounts");echo XXRETCODE:$?'
May 17 18:25:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=24563 duration=0(sec)
May 17 18:25:08 client-18 xinetd[3390]: START: shell pid=24598 from=192.168.4.5
May 17 18:25:08 client-18 rshd[24599]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lsof -t /mnt/lustre2");echo XXRETCODE:$?'
May 17 18:25:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=24598 duration=0(sec)
May 17 18:25:08 client-18 xinetd[3390]: START: shell pid=24634 from=192.168.4.5
May 17 18:25:08 client-18 rshd[24635]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "umount  /mnt/lustre2 2>&1");echo XXRETCODE:$?'
May 17 18:25:08 client-18 kernel: LustreError: 24669:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway
May 17 18:25:08 client-18 kernel: LustreError: 24669:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Skipped 1 previous similar message
May 17 18:25:08 client-18 kernel: LustreError: 24669:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108
May 17 18:25:08 client-18 kernel: LustreError: 24669:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) Skipped 1 previous similar message
May 17 18:25:08 client-18 kernel: Lustre: client ffff8102f8fcd800 umount complete
May 17 18:25:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=24634 duration=0(sec)
May 17 18:25:26 client-18 kernel: Lustre: 8885:0:(import.c:885:ptlrpc_connect_interpret()) MGS@MGC192.168.4.128@o2ib_0 changed server handle from 0xfff98862bf6503ca to 0xfff98862bf65077b
May 17 18:25:31 client-18 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The obd_ping operation failed with -107
May 17 18:25:31 client-18 kernel: LustreError: Skipped 9 previous similar messages
May 17 18:26:52 client-18 xinetd[3390]: START: shell pid=24678 from=192.168.4.5
May 17 18:26:52 client-18 rshd[24679]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 18:26:52 client-18 xinetd[3390]: EXIT: shell status=0 pid=24678 duration=0(sec)
May 17 18:26:52 client-18 xinetd[3390]: START: shell pid=24714 from=192.168.4.5
May 17 18:26:52 client-18 rshd[24715]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 18:26:52 client-18 xinetd[3390]: EXIT: shell status=0 pid=24714 duration=0(sec)
May 17 18:26:52 client-18 xinetd[3390]: START: shell pid=24751 from=192.168.4.5
May 17 18:26:52 client-18 rshd[24752]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-vbr test 2a: create operations doesn\'t change version of parent ============================ 18:26:52 \(1305682012\)");echo XXRETCODE:$?'
May 17 18:26:52 client-18 kernel: Lustre: DEBUG MARKER: == replay-vbr test 2a: create operations doesn't change version of parent ============================ 18:26:52 (1305682012)
May 17 18:26:52 client-18 xinetd[3390]: EXIT: shell status=0 pid=24751 duration=0(sec)
May 17 18:26:53 client-18 xinetd[3390]: START: shell pid=24786 from=192.168.4.5
May 17 18:26:53 client-18 rshd[24787]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 18:26:54 client-18 xinetd[3390]: EXIT: shell status=0 pid=24786 duration=1(sec)
May 17 18:26:54 client-18 xinetd[3390]: START: shell pid=24822 from=192.168.4.5
May 17 18:26:54 client-18 rshd[24823]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 18:26:54 client-18 xinetd[3390]: EXIT: shell status=0 pid=24822 duration=0(sec)
May 17 18:26:54 client-18 xinetd[3390]: START: shell pid=24859 from=192.168.4.5
May 17 18:26:54 client-18 rshd[24860]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-vbr test 2b: create checks version of parent =============================================== 18:26:54 \(1305682014\)");echo XXRETCODE:$?'
May 17 18:26:54 client-18 kernel: Lustre: DEBUG MARKER: == replay-vbr test 2b: create checks version of parent =============================================== 18:26:54 (1305682014)
May 17 18:26:54 client-18 xinetd[3390]: EXIT: shell status=0 pid=24859 duration=0(sec)
May 17 18:26:54 client-18 xinetd[3390]: START: shell pid=24894 from=192.168.4.5
May 17 18:26:54 client-18 rshd[24895]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mkdir -p /mnt/lustre2");echo XXRETCODE:$?'
May 17 18:26:54 client-18 xinetd[3390]: EXIT: shell status=0 pid=24894 duration=0(sec)
May 17 18:26:54 client-18 xinetd[3390]: START: shell pid=24929 from=192.168.4.5
May 17 18:26:54 client-18 rshd[24930]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre2");echo XXRETCODE:$?'
May 17 18:26:54 client-18 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'.
May 17 18:26:54 client-18 kernel: Lustre: 24964:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff81033f772000->192.168.4.128@o2ib netid 50000: select flavor null
May 17 18:26:54 client-18 kernel: Lustre: 24964:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 6 previous similar messages
May 17 18:26:54 client-18 kernel: Lustre: Client lustre-client has started
May 17 18:26:54 client-18 xinetd[3390]: EXIT: shell status=0 pid=24929 duration=0(sec)
May 17 18:26:54 client-18 xinetd[3390]: START: shell pid=24967 from=192.168.4.5
May 17 18:26:54 client-18 rshd[24968]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param debug=\"0x33f0406\";         lctl set_param subsystem_debug=\"0xffb7e3ff\";         lctl set_param debug_mb=32");echo XXRETCODE:$?'
May 17 18:26:54 client-18 kernel: Lustre: 25002:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release.
May 17 18:26:54 client-18 kernel: Lustre: 25002:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message
May 17 18:26:54 client-18 xinetd[3390]: EXIT: shell status=0 pid=24967 duration=0(sec)
May 17 18:26:55 client-18 xinetd[3390]: START: shell pid=25004 from=192.168.4.5
May 17 18:26:55 client-18 rshd[25005]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 18:26:55 client-18 xinetd[3390]: EXIT: shell status=0 pid=25004 duration=0(sec)
May 17 18:26:55 client-18 xinetd[3390]: START: shell pid=25043 from=192.168.4.5
May 17 18:26:55 client-18 rshd[25044]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "chmod 777 /mnt/lustre2/d0.replay-vbr/d2");echo XXRETCODE:$?'
May 17 18:26:55 client-18 xinetd[3390]: EXIT: shell status=0 pid=25043 duration=0(sec)
May 17 18:26:55 client-18 xinetd[3390]: START: shell pid=25078 from=192.168.4.5
May 17 18:26:55 client-18 rshd[25079]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "grep -c /mnt/lustre2' ' /proc/mounts");echo XXRETCODE:$?'
May 17 18:26:55 client-18 xinetd[3390]: EXIT: shell status=0 pid=25078 duration=0(sec)
May 17 18:26:55 client-18 xinetd[3390]: START: shell pid=25113 from=192.168.4.5
May 17 18:26:55 client-18 rshd[25114]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lsof -t /mnt/lustre2");echo XXRETCODE:$?'
May 17 18:26:55 client-18 xinetd[3390]: EXIT: shell status=0 pid=25113 duration=0(sec)
May 17 18:26:55 client-18 xinetd[3390]: START: shell pid=25149 from=192.168.4.5
May 17 18:26:55 client-18 rshd[25150]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "umount  /mnt/lustre2 2>&1");echo XXRETCODE:$?'
May 17 18:26:56 client-18 kernel: LustreError: 25184:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway
May 17 18:26:56 client-18 kernel: LustreError: 25184:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Skipped 2 previous similar messages
May 17 18:26:56 client-18 kernel: LustreError: 25184:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108
May 17 18:26:56 client-18 kernel: LustreError: 25184:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) Skipped 2 previous similar messages
May 17 18:26:56 client-18 kernel: Lustre: client ffff81033f772000 umount complete
May 17 18:26:56 client-18 xinetd[3390]: EXIT: shell status=0 pid=25149 duration=1(sec)
May 17 18:27:26 client-18 kernel: Lustre: 8885:0:(import.c:885:ptlrpc_connect_interpret()) MGS@MGC192.168.4.128@o2ib_0 changed server handle from 0xfff98862bf65077b to 0xfff98862bf650cc9
May 17 18:27:26 client-18 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import
May 17 18:27:26 client-18 kernel: Lustre: Skipped 4 previous similar messages
May 17 18:29:22 client-18 xinetd[3390]: START: shell pid=25197 from=192.168.4.5
May 17 18:29:22 client-18 rshd[25198]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 18:29:23 client-18 xinetd[3390]: EXIT: shell status=0 pid=25197 duration=1(sec)
May 17 18:29:23 client-18 xinetd[3390]: START: shell pid=25233 from=192.168.4.5
May 17 18:29:23 client-18 rshd[25234]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 18:29:23 client-18 xinetd[3390]: EXIT: shell status=0 pid=25233 duration=0(sec)
May 17 18:29:23 client-18 xinetd[3390]: START: shell pid=25270 from=192.168.4.5
May 17 18:29:23 client-18 rshd[25271]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-vbr test 3a: unlink doesn\'t change version of parent ======================================= 18:29:23 \(1305682163\)");echo XXRETCODE:$?'
May 17 18:29:23 client-18 kernel: Lustre: DEBUG MARKER: == replay-vbr test 3a: unlink doesn't change version of parent ======================================= 18:29:23 (1305682163)
May 17 18:29:23 client-18 xinetd[3390]: EXIT: shell status=0 pid=25270 duration=0(sec)
May 17 18:29:23 client-18 xinetd[3390]: START: shell pid=25305 from=192.168.4.5
May 17 18:29:23 client-18 rshd[25306]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 18:29:23 client-18 xinetd[3390]: EXIT: shell status=0 pid=25305 duration=0(sec)
May 17 18:29:23 client-18 xinetd[3390]: START: shell pid=25341 from=192.168.4.5
May 17 18:29:23 client-18 rshd[25342]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 18:29:23 client-18 xinetd[3390]: EXIT: shell status=0 pid=25341 duration=0(sec)
May 17 18:29:23 client-18 xinetd[3390]: START: shell pid=25378 from=192.168.4.5
May 17 18:29:23 client-18 rshd[25379]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-vbr test 3b: unlink checks version of parent =============================================== 18:29:23 \(1305682163\)");echo XXRETCODE:$?'
May 17 18:29:24 client-18 kernel: Lustre: DEBUG MARKER: == replay-vbr test 3b: unlink checks version of parent =============================================== 18:29:23 (1305682163)
May 17 18:29:24 client-18 xinetd[3390]: EXIT: shell status=0 pid=25378 duration=1(sec)
May 17 18:29:24 client-18 xinetd[3390]: START: shell pid=25413 from=192.168.4.5
May 17 18:29:24 client-18 rshd[25414]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mkdir -p /mnt/lustre2");echo XXRETCODE:$?'
May 17 18:29:24 client-18 xinetd[3390]: EXIT: shell status=0 pid=25413 duration=0(sec)
May 17 18:29:24 client-18 xinetd[3390]: START: shell pid=25448 from=192.168.4.5
May 17 18:29:24 client-18 rshd[25449]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre2");echo XXRETCODE:$?'
May 17 18:29:24 client-18 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'.
May 17 18:29:24 client-18 kernel: Lustre: 25483:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff810310746c00->192.168.4.128@o2ib netid 50000: select flavor null
May 17 18:29:24 client-18 kernel: Lustre: 25483:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 6 previous similar messages
May 17 18:29:24 client-18 kernel: Lustre: Client lustre-client has started
May 17 18:29:24 client-18 xinetd[3390]: EXIT: shell status=0 pid=25448 duration=0(sec)
May 17 18:29:24 client-18 xinetd[3390]: START: shell pid=25486 from=192.168.4.5
May 17 18:29:24 client-18 rshd[25487]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param debug=\"0x33f0406\";         lctl set_param subsystem_debug=\"0xffb7e3ff\";         lctl set_param debug_mb=32");echo XXRETCODE:$?'
May 17 18:29:24 client-18 kernel: Lustre: 25521:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release.
May 17 18:29:24 client-18 kernel: Lustre: 25521:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message
May 17 18:29:24 client-18 xinetd[3390]: EXIT: shell status=0 pid=25486 duration=0(sec)
May 17 18:29:24 client-18 xinetd[3390]: START: shell pid=25523 from=192.168.4.5
May 17 18:29:24 client-18 rshd[25524]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 18:29:24 client-18 xinetd[3390]: EXIT: shell status=0 pid=25523 duration=0(sec)
May 17 18:29:25 client-18 xinetd[3390]: START: shell pid=25562 from=192.168.4.5
May 17 18:29:25 client-18 rshd[25563]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "chmod 777 /mnt/lustre2/d0.replay-vbr/d3");echo XXRETCODE:$?'
May 17 18:29:25 client-18 xinetd[3390]: EXIT: shell status=0 pid=25562 duration=0(sec)
May 17 18:29:25 client-18 xinetd[3390]: START: shell pid=25597 from=192.168.4.5
May 17 18:29:25 client-18 rshd[25598]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "grep -c /mnt/lustre2' ' /proc/mounts");echo XXRETCODE:$?'
May 17 18:29:25 client-18 xinetd[3390]: EXIT: shell status=0 pid=25597 duration=0(sec)
May 17 18:29:25 client-18 xinetd[3390]: START: shell pid=25632 from=192.168.4.5
May 17 18:29:25 client-18 rshd[25633]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lsof -t /mnt/lustre2");echo XXRETCODE:$?'
May 17 18:29:25 client-18 xinetd[3390]: EXIT: shell status=0 pid=25632 duration=0(sec)
May 17 18:29:25 client-18 xinetd[3390]: START: shell pid=25668 from=192.168.4.5
May 17 18:29:25 client-18 rshd[25669]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "umount  /mnt/lustre2 2>&1");echo XXRETCODE:$?'
May 17 18:29:25 client-18 kernel: LustreError: 25703:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway
May 17 18:29:25 client-18 kernel: LustreError: 25703:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Skipped 1 previous similar message
May 17 18:29:25 client-18 kernel: LustreError: 25703:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108
May 17 18:29:25 client-18 kernel: LustreError: 25703:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) Skipped 1 previous similar message
May 17 18:29:25 client-18 kernel: Lustre: client ffff810310746c00 umount complete
May 17 18:29:25 client-18 xinetd[3390]: EXIT: shell status=0 pid=25668 duration=0(sec)
May 17 18:29:44 client-18 kernel: Lustre: 8885:0:(import.c:885:ptlrpc_connect_interpret()) MGS@MGC192.168.4.128@o2ib_0 changed server handle from 0xfff98862bf650cc9 to 0xfff98862bf6510d5
May 17 18:31:53 client-18 xinetd[3390]: START: shell pid=25716 from=192.168.4.5
May 17 18:31:53 client-18 rshd[25717]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 18:31:53 client-18 xinetd[3390]: EXIT: shell status=0 pid=25716 duration=0(sec)
May 17 18:31:53 client-18 xinetd[3390]: START: shell pid=25752 from=192.168.4.5
May 17 18:31:53 client-18 rshd[25753]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 18:31:53 client-18 xinetd[3390]: EXIT: shell status=0 pid=25752 duration=0(sec)
May 17 18:31:53 client-18 xinetd[3390]: START: shell pid=25789 from=192.168.4.5
May 17 18:31:53 client-18 rshd[25790]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-vbr test 4a: setattr of UID changes versions =============================================== 18:31:53 \(1305682313\)");echo XXRETCODE:$?'
May 17 18:31:53 client-18 kernel: Lustre: DEBUG MARKER: == replay-vbr test 4a: setattr of UID changes versions =============================================== 18:31:53 (1305682313)
May 17 18:31:53 client-18 xinetd[3390]: EXIT: shell status=0 pid=25789 duration=0(sec)
May 17 18:31:54 client-18 xinetd[3390]: START: shell pid=25824 from=192.168.4.5
May 17 18:31:54 client-18 rshd[25825]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 18:31:54 client-18 xinetd[3390]: EXIT: shell status=0 pid=25824 duration=0(sec)
May 17 18:31:54 client-18 xinetd[3390]: START: shell pid=25860 from=192.168.4.5
May 17 18:31:54 client-18 rshd[25861]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 18:31:54 client-18 xinetd[3390]: EXIT: shell status=0 pid=25860 duration=0(sec)
May 17 18:31:54 client-18 xinetd[3390]: START: shell pid=25897 from=192.168.4.5
May 17 18:31:54 client-18 rshd[25898]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-vbr test 4b: setattr of GID changes versions =============================================== 18:31:54 \(1305682314\)");echo XXRETCODE:$?'
May 17 18:31:54 client-18 kernel: Lustre: DEBUG MARKER: == replay-vbr test 4b: setattr of GID changes versions =============================================== 18:31:54 (1305682314)
May 17 18:31:54 client-18 xinetd[3390]: EXIT: shell status=0 pid=25897 duration=0(sec)
May 17 18:31:54 client-18 xinetd[3390]: START: shell pid=25932 from=192.168.4.5
May 17 18:31:54 client-18 rshd[25933]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 18:31:55 client-18 xinetd[3390]: EXIT: shell status=0 pid=25932 duration=1(sec)
May 17 18:31:55 client-18 xinetd[3390]: START: shell pid=25968 from=192.168.4.5
May 17 18:31:55 client-18 rshd[25969]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 18:31:55 client-18 xinetd[3390]: EXIT: shell status=0 pid=25968 duration=0(sec)
May 17 18:31:55 client-18 xinetd[3390]: START: shell pid=26005 from=192.168.4.5
May 17 18:31:55 client-18 rshd[26006]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-vbr test 4c: setattr of UID checks versions ================================================ 18:31:55 \(1305682315\)");echo XXRETCODE:$?'
May 17 18:31:55 client-18 kernel: Lustre: DEBUG MARKER: == replay-vbr test 4c: setattr of UID checks versions ================================================ 18:31:55 (1305682315)
May 17 18:31:55 client-18 xinetd[3390]: EXIT: shell status=0 pid=26005 duration=0(sec)
May 17 18:31:55 client-18 xinetd[3390]: START: shell pid=26040 from=192.168.4.5
May 17 18:31:55 client-18 rshd[26041]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mkdir -p /mnt/lustre2");echo XXRETCODE:$?'
May 17 18:31:55 client-18 xinetd[3390]: START: shell pid=26076 from=192.168.4.5
May 17 18:31:55 client-18 xinetd[3390]: EXIT: shell status=0 pid=26040 duration=0(sec)
May 17 18:31:55 client-18 rshd[26077]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre2");echo XXRETCODE:$?'
May 17 18:31:55 client-18 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'.
May 17 18:31:55 client-18 kernel: Lustre: 26111:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff8102ec27e000->192.168.4.128@o2ib netid 50000: select flavor null
May 17 18:31:55 client-18 kernel: Lustre: 26111:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 6 previous similar messages
May 17 18:31:55 client-18 kernel: Lustre: Client lustre-client has started
May 17 18:31:55 client-18 xinetd[3390]: START: shell pid=26114 from=192.168.4.5
May 17 18:31:55 client-18 xinetd[3390]: EXIT: shell status=0 pid=26076 duration=0(sec)
May 17 18:31:55 client-18 rshd[26115]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param debug=\"0x33f0406\";         lctl set_param subsystem_debug=\"0xffb7e3ff\";         lctl set_param debug_mb=32");echo XXRETCODE:$?'
May 17 18:31:55 client-18 kernel: Lustre: 26149:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release.
May 17 18:31:55 client-18 kernel: Lustre: 26149:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message
May 17 18:31:55 client-18 xinetd[3390]: EXIT: shell status=0 pid=26114 duration=0(sec)
May 17 18:31:56 client-18 xinetd[3390]: START: shell pid=26151 from=192.168.4.5
May 17 18:31:56 client-18 rshd[26152]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 18:31:56 client-18 xinetd[3390]: EXIT: shell status=0 pid=26151 duration=0(sec)
May 17 18:31:56 client-18 xinetd[3390]: START: shell pid=26190 from=192.168.4.5
May 17 18:31:56 client-18 rshd[26191]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "chgrp 500 /mnt/lustre2/f4c");echo XXRETCODE:$?'
May 17 18:31:56 client-18 xinetd[3390]: EXIT: shell status=0 pid=26190 duration=0(sec)
May 17 18:31:56 client-18 xinetd[3390]: START: shell pid=26225 from=192.168.4.5
May 17 18:31:56 client-18 rshd[26226]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "grep -c /mnt/lustre2' ' /proc/mounts");echo XXRETCODE:$?'
May 17 18:31:56 client-18 xinetd[3390]: EXIT: shell status=0 pid=26225 duration=0(sec)
May 17 18:31:56 client-18 xinetd[3390]: START: shell pid=26260 from=192.168.4.5
May 17 18:31:56 client-18 rshd[26261]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lsof -t /mnt/lustre2");echo XXRETCODE:$?'
May 17 18:31:56 client-18 xinetd[3390]: EXIT: shell status=0 pid=26260 duration=0(sec)
May 17 18:31:56 client-18 xinetd[3390]: START: shell pid=26296 from=192.168.4.5
May 17 18:31:56 client-18 rshd[26297]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "umount  /mnt/lustre2 2>&1");echo XXRETCODE:$?'
May 17 18:31:57 client-18 kernel: LustreError: 26331:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway
May 17 18:31:57 client-18 kernel: LustreError: 26331:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Skipped 1 previous similar message
May 17 18:31:57 client-18 kernel: LustreError: 26331:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108
May 17 18:31:57 client-18 kernel: LustreError: 26331:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) Skipped 1 previous similar message
May 17 18:31:57 client-18 kernel: Lustre: client ffff8102ec27e000 umount complete
May 17 18:31:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=26296 duration=1(sec)
May 17 18:32:15 client-18 kernel: Lustre: 8885:0:(import.c:885:ptlrpc_connect_interpret()) MGS@MGC192.168.4.128@o2ib_0 changed server handle from 0xfff98862bf6510d5 to 0xfff98862bf65155f
May 17 18:34:19 client-18 xinetd[3390]: START: shell pid=26344 from=192.168.4.5
May 17 18:34:19 client-18 rshd[26345]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 18:34:19 client-18 xinetd[3390]: EXIT: shell status=0 pid=26344 duration=0(sec)
May 17 18:34:19 client-18 xinetd[3390]: START: shell pid=26380 from=192.168.4.5
May 17 18:34:20 client-18 rshd[26381]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 18:34:20 client-18 xinetd[3390]: EXIT: shell status=0 pid=26380 duration=1(sec)
May 17 18:34:20 client-18 xinetd[3390]: START: shell pid=26417 from=192.168.4.5
May 17 18:34:20 client-18 rshd[26418]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-vbr test 4d: setattr of GID checks versions ================================================ 18:34:20 \(1305682460\)");echo XXRETCODE:$?'
May 17 18:34:20 client-18 kernel: Lustre: DEBUG MARKER: == replay-vbr test 4d: setattr of GID checks versions ================================================ 18:34:20 (1305682460)
May 17 18:34:20 client-18 xinetd[3390]: EXIT: shell status=0 pid=26417 duration=0(sec)
May 17 18:34:20 client-18 xinetd[3390]: START: shell pid=26452 from=192.168.4.5
May 17 18:34:20 client-18 rshd[26453]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mkdir -p /mnt/lustre2");echo XXRETCODE:$?'
May 17 18:34:20 client-18 xinetd[3390]: START: shell pid=26487 from=192.168.4.5
May 17 18:34:20 client-18 xinetd[3390]: EXIT: shell status=0 pid=26452 duration=0(sec)
May 17 18:34:20 client-18 rshd[26488]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre2");echo XXRETCODE:$?'
May 17 18:34:20 client-18 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'.
May 17 18:34:20 client-18 kernel: Lustre: 26522:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff810322ef5400->192.168.4.128@o2ib netid 50000: select flavor null
May 17 18:34:20 client-18 kernel: Lustre: 26522:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 6 previous similar messages
May 17 18:34:20 client-18 kernel: Lustre: Client lustre-client has started
May 17 18:34:20 client-18 xinetd[3390]: EXIT: shell status=0 pid=26487 duration=0(sec)
May 17 18:34:20 client-18 xinetd[3390]: START: shell pid=26525 from=192.168.4.5
May 17 18:34:20 client-18 rshd[26526]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param debug=\"0x33f0406\";         lctl set_param subsystem_debug=\"0xffb7e3ff\";         lctl set_param debug_mb=32");echo XXRETCODE:$?'
May 17 18:34:20 client-18 kernel: Lustre: 26560:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release.
May 17 18:34:20 client-18 kernel: Lustre: 26560:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message
May 17 18:34:20 client-18 xinetd[3390]: EXIT: shell status=0 pid=26525 duration=0(sec)
May 17 18:34:21 client-18 xinetd[3390]: START: shell pid=26562 from=192.168.4.5
May 17 18:34:21 client-18 rshd[26563]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 18:34:21 client-18 xinetd[3390]: EXIT: shell status=0 pid=26562 duration=0(sec)
May 17 18:34:21 client-18 xinetd[3390]: START: shell pid=26601 from=192.168.4.5
May 17 18:34:21 client-18 rshd[26602]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "chown 500:500 /mnt/lustre2/f4d");echo XXRETCODE:$?'
May 17 18:34:21 client-18 xinetd[3390]: EXIT: shell status=0 pid=26601 duration=0(sec)
May 17 18:34:21 client-18 xinetd[3390]: START: shell pid=26636 from=192.168.4.5
May 17 18:34:21 client-18 rshd[26637]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "grep -c /mnt/lustre2' ' /proc/mounts");echo XXRETCODE:$?'
May 17 18:34:21 client-18 xinetd[3390]: START: shell pid=26671 from=192.168.4.5
May 17 18:34:21 client-18 xinetd[3390]: EXIT: shell status=0 pid=26636 duration=0(sec)
May 17 18:34:21 client-18 rshd[26672]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lsof -t /mnt/lustre2");echo XXRETCODE:$?'
May 17 18:34:21 client-18 xinetd[3390]: EXIT: shell status=0 pid=26671 duration=0(sec)
May 17 18:34:21 client-18 xinetd[3390]: START: shell pid=26707 from=192.168.4.5
May 17 18:34:21 client-18 rshd[26708]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "umount  /mnt/lustre2 2>&1");echo XXRETCODE:$?'
May 17 18:34:22 client-18 kernel: LustreError: 26742:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway
May 17 18:34:22 client-18 kernel: LustreError: 26742:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108
May 17 18:34:22 client-18 kernel: Lustre: client ffff810322ef5400 umount complete
May 17 18:34:22 client-18 xinetd[3390]: EXIT: shell status=0 pid=26707 duration=1(sec)
May 17 18:34:40 client-18 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail.
May 17 18:34:40 client-18 kernel: LustreError: Skipped 4 previous similar messages
May 17 18:34:40 client-18 kernel: Lustre: lustre-MDT0000-mdc-ffff8103028bc400: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete.
May 17 18:34:40 client-18 kernel: Lustre: 8885:0:(import.c:885:ptlrpc_connect_interpret()) MGS@MGC192.168.4.128@o2ib_0 changed server handle from 0xfff98862bf65155f to 0xfff98862bf6518bc
May 17 18:34:40 client-18 kernel: Lustre: Skipped 4 previous similar messages
May 17 18:34:40 client-18 kernel: Lustre: MGC192.168.4.128@o2ib: Connection restored to service MGS using nid 192.168.4.128@o2ib.
May 17 18:34:40 client-18 kernel: Lustre: Skipped 8 previous similar messages
May 17 18:35:03 client-18 kernel: Lustre: 8884:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369101282847067 sent from lustre-MDT0000-mdc-ffff8103028bc400 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305682470] [real_sent 1305682470] [current 1305682503] [deadline 33s] [delay 0s]  req@ffff8102eb8e2c00 x1369101282847067/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 192/192 e 0 to 1 dl 1305682503 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
May 17 18:35:03 client-18 kernel: Lustre: 8884:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 16 previous similar messages
May 17 18:36:44 client-18 xinetd[3390]: START: shell pid=26755 from=192.168.4.5
May 17 18:36:44 client-18 rshd[26756]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 18:36:44 client-18 xinetd[3390]: EXIT: shell status=0 pid=26755 duration=0(sec)
May 17 18:36:44 client-18 xinetd[3390]: START: shell pid=26791 from=192.168.4.5
May 17 18:36:44 client-18 rshd[26792]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 18:36:44 client-18 xinetd[3390]: EXIT: shell status=0 pid=26791 duration=0(sec)
May 17 18:36:44 client-18 xinetd[3390]: START: shell pid=26828 from=192.168.4.5
May 17 18:36:44 client-18 rshd[26829]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-vbr test 4e: setattr of permission changes versions ======================================== 18:36:44 \(1305682604\)");echo XXRETCODE:$?'
May 17 18:36:44 client-18 kernel: Lustre: DEBUG MARKER: == replay-vbr test 4e: setattr of permission changes versions ======================================== 18:36:44 (1305682604)
May 17 18:36:44 client-18 xinetd[3390]: EXIT: shell status=0 pid=26828 duration=0(sec)
May 17 18:36:45 client-18 xinetd[3390]: START: shell pid=26863 from=192.168.4.5
May 17 18:36:45 client-18 rshd[26864]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 18:36:45 client-18 xinetd[3390]: EXIT: shell status=0 pid=26863 duration=0(sec)
May 17 18:36:45 client-18 xinetd[3390]: START: shell pid=26899 from=192.168.4.5
May 17 18:36:45 client-18 rshd[26900]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 18:36:45 client-18 xinetd[3390]: EXIT: shell status=0 pid=26899 duration=0(sec)
May 17 18:36:45 client-18 xinetd[3390]: START: shell pid=26936 from=192.168.4.5
May 17 18:36:45 client-18 rshd[26937]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-vbr test 4f: setattr of permission checks versions ========================================= 18:36:45 \(1305682605\)");echo XXRETCODE:$?'
May 17 18:36:45 client-18 kernel: Lustre: DEBUG MARKER: == replay-vbr test 4f: setattr of permission checks versions ========================================= 18:36:45 (1305682605)
May 17 18:36:45 client-18 xinetd[3390]: EXIT: shell status=0 pid=26936 duration=0(sec)
May 17 18:36:45 client-18 xinetd[3390]: START: shell pid=26971 from=192.168.4.5
May 17 18:36:45 client-18 rshd[26972]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mkdir -p /mnt/lustre2");echo XXRETCODE:$?'
May 17 18:36:45 client-18 xinetd[3390]: START: shell pid=27006 from=192.168.4.5
May 17 18:36:45 client-18 xinetd[3390]: EXIT: shell status=0 pid=26971 duration=0(sec)
May 17 18:36:45 client-18 rshd[27007]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre2");echo XXRETCODE:$?'
May 17 18:36:45 client-18 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'.
May 17 18:36:45 client-18 kernel: Lustre: 27041:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff81031c4a0400->192.168.4.128@o2ib netid 50000: select flavor null
May 17 18:36:45 client-18 kernel: Lustre: 27041:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 6 previous similar messages
May 17 18:36:45 client-18 kernel: Lustre: Client lustre-client has started
May 17 18:36:45 client-18 xinetd[3390]: EXIT: shell status=0 pid=27006 duration=0(sec)
May 17 18:36:45 client-18 xinetd[3390]: START: shell pid=27044 from=192.168.4.5
May 17 18:36:45 client-18 rshd[27045]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param debug=\"0x33f0406\";         lctl set_param subsystem_debug=\"0xffb7e3ff\";         lctl set_param debug_mb=32");echo XXRETCODE:$?'
May 17 18:36:46 client-18 kernel: Lustre: 27079:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release.
May 17 18:36:46 client-18 kernel: Lustre: 27079:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message
May 17 18:36:46 client-18 xinetd[3390]: EXIT: shell status=0 pid=27044 duration=1(sec)
May 17 18:36:46 client-18 xinetd[3390]: START: shell pid=27081 from=192.168.4.5
May 17 18:36:46 client-18 rshd[27082]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 18:36:46 client-18 xinetd[3390]: EXIT: shell status=0 pid=27081 duration=0(sec)
May 17 18:36:46 client-18 xinetd[3390]: START: shell pid=27120 from=192.168.4.5
May 17 18:36:46 client-18 rshd[27121]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "chgrp 500 /mnt/lustre2/f4f");echo XXRETCODE:$?'
May 17 18:36:46 client-18 xinetd[3390]: EXIT: shell status=0 pid=27120 duration=0(sec)
May 17 18:36:46 client-18 xinetd[3390]: START: shell pid=27155 from=192.168.4.5
May 17 18:36:47 client-18 rshd[27156]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "grep -c /mnt/lustre2' ' /proc/mounts");echo XXRETCODE:$?'
May 17 18:36:47 client-18 xinetd[3390]: EXIT: shell status=0 pid=27155 duration=1(sec)
May 17 18:36:47 client-18 xinetd[3390]: START: shell pid=27190 from=192.168.4.5
May 17 18:36:47 client-18 rshd[27191]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lsof -t /mnt/lustre2");echo XXRETCODE:$?'
May 17 18:36:47 client-18 xinetd[3390]: EXIT: shell status=0 pid=27190 duration=0(sec)
May 17 18:36:47 client-18 xinetd[3390]: START: shell pid=27226 from=192.168.4.5
May 17 18:36:47 client-18 rshd[27227]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "umount  /mnt/lustre2 2>&1");echo XXRETCODE:$?'
May 17 18:36:47 client-18 kernel: LustreError: 27261:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway
May 17 18:36:47 client-18 kernel: LustreError: 27261:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108
May 17 18:36:47 client-18 kernel: Lustre: client ffff81031c4a0400 umount complete
May 17 18:36:47 client-18 xinetd[3390]: EXIT: shell status=0 pid=27226 duration=0(sec)
May 17 18:37:07 client-18 kernel: Lustre: 8886:0:(import.c:529:import_select_connection()) MGC192.168.4.128@o2ib: tried all connections, increasing latency to 6s
May 17 18:37:07 client-18 kernel: Lustre: 8886:0:(import.c:529:import_select_connection()) Skipped 2 previous similar messages
May 17 18:37:07 client-18 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The obd_ping operation failed with -107
May 17 18:37:07 client-18 kernel: Lustre: 8885:0:(import.c:885:ptlrpc_connect_interpret()) MGS@MGC192.168.4.128@o2ib_0 changed server handle from 0xfff98862bf6518bc to 0xfff98862bf651c9e
May 17 18:37:07 client-18 kernel: LustreError: Skipped 7 previous similar messages
May 17 18:39:10 client-18 xinetd[3390]: START: shell pid=27274 from=192.168.4.5
May 17 18:39:10 client-18 rshd[27275]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 18:39:10 client-18 xinetd[3390]: EXIT: shell status=0 pid=27274 duration=0(sec)
May 17 18:39:10 client-18 xinetd[3390]: START: shell pid=27310 from=192.168.4.5
May 17 18:39:10 client-18 rshd[27311]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 18:39:10 client-18 xinetd[3390]: EXIT: shell status=0 pid=27310 duration=0(sec)
May 17 18:39:10 client-18 xinetd[3390]: START: shell pid=27347 from=192.168.4.5
May 17 18:39:10 client-18 rshd[27348]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-vbr test 4g: setattr of flags changes versions ============================================= 18:39:10 \(1305682750\)");echo XXRETCODE:$?'
May 17 18:39:10 client-18 kernel: Lustre: DEBUG MARKER: == replay-vbr test 4g: setattr of flags changes versions ============================================= 18:39:10 (1305682750)
May 17 18:39:10 client-18 xinetd[3390]: EXIT: shell status=0 pid=27347 duration=0(sec)
May 17 18:39:11 client-18 xinetd[3390]: START: shell pid=27382 from=192.168.4.5
May 17 18:39:11 client-18 rshd[27383]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 18:39:11 client-18 xinetd[3390]: EXIT: shell status=0 pid=27382 duration=0(sec)
May 17 18:39:11 client-18 xinetd[3390]: START: shell pid=27418 from=192.168.4.5
May 17 18:39:11 client-18 rshd[27419]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 18:39:11 client-18 xinetd[3390]: EXIT: shell status=0 pid=27418 duration=0(sec)
May 17 18:39:11 client-18 xinetd[3390]: START: shell pid=27455 from=192.168.4.5
May 17 18:39:11 client-18 rshd[27456]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-vbr test 4h: setattr of flags checks versions ============================================== 18:39:11 \(1305682751\)");echo XXRETCODE:$?'
May 17 18:39:11 client-18 kernel: Lustre: DEBUG MARKER: == replay-vbr test 4h: setattr of flags checks versions ============================================== 18:39:11 (1305682751)
May 17 18:39:11 client-18 xinetd[3390]: EXIT: shell status=0 pid=27455 duration=0(sec)
May 17 18:39:11 client-18 xinetd[3390]: START: shell pid=27490 from=192.168.4.5
May 17 18:39:11 client-18 rshd[27491]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mkdir -p /mnt/lustre2");echo XXRETCODE:$?'
May 17 18:39:11 client-18 xinetd[3390]: START: shell pid=27526 from=192.168.4.5
May 17 18:39:11 client-18 xinetd[3390]: EXIT: shell status=0 pid=27490 duration=0(sec)
May 17 18:39:11 client-18 rshd[27527]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre2");echo XXRETCODE:$?'
May 17 18:39:11 client-18 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'.
May 17 18:39:11 client-18 kernel: Lustre: 27561:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff81033feb0400->192.168.4.128@o2ib netid 50000: select flavor null
May 17 18:39:11 client-18 kernel: Lustre: 27561:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 6 previous similar messages
May 17 18:39:11 client-18 kernel: Lustre: Client lustre-client has started
May 17 18:39:11 client-18 xinetd[3390]: START: shell pid=27565 from=192.168.4.5
May 17 18:39:11 client-18 xinetd[3390]: EXIT: shell status=0 pid=27526 duration=0(sec)
May 17 18:39:11 client-18 rshd[27566]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param debug=\"0x33f0406\";         lctl set_param subsystem_debug=\"0xffb7e3ff\";         lctl set_param debug_mb=32");echo XXRETCODE:$?'
May 17 18:39:11 client-18 kernel: Lustre: 27600:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release.
May 17 18:39:11 client-18 kernel: Lustre: 27600:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message
May 17 18:39:11 client-18 xinetd[3390]: EXIT: shell status=0 pid=27565 duration=0(sec)
May 17 18:39:12 client-18 xinetd[3390]: START: shell pid=27602 from=192.168.4.5
May 17 18:39:12 client-18 rshd[27603]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 18:39:12 client-18 xinetd[3390]: EXIT: shell status=0 pid=27602 duration=0(sec)
May 17 18:39:12 client-18 xinetd[3390]: START: shell pid=27641 from=192.168.4.5
May 17 18:39:12 client-18 rshd[27642]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "chmod 666 /mnt/lustre2/f4h");echo XXRETCODE:$?'
May 17 18:39:12 client-18 xinetd[3390]: EXIT: shell status=0 pid=27641 duration=0(sec)
May 17 18:39:12 client-18 xinetd[3390]: START: shell pid=27676 from=192.168.4.5
May 17 18:39:12 client-18 rshd[27677]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "grep -c /mnt/lustre2' ' /proc/mounts");echo XXRETCODE:$?'
May 17 18:39:12 client-18 xinetd[3390]: EXIT: shell status=0 pid=27676 duration=0(sec)
May 17 18:39:12 client-18 xinetd[3390]: START: shell pid=27711 from=192.168.4.5
May 17 18:39:12 client-18 rshd[27712]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lsof -t /mnt/lustre2");echo XXRETCODE:$?'
May 17 18:39:13 client-18 xinetd[3390]: EXIT: shell status=0 pid=27711 duration=1(sec)
May 17 18:39:13 client-18 xinetd[3390]: START: shell pid=27747 from=192.168.4.5
May 17 18:39:13 client-18 rshd[27748]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "umount  /mnt/lustre2 2>&1");echo XXRETCODE:$?'
May 17 18:39:13 client-18 xinetd[3390]: EXIT: shell status=0 pid=27747 duration=0(sec)
May 17 18:39:43 client-18 kernel: Lustre: 8885:0:(import.c:885:ptlrpc_connect_interpret()) MGS@MGC192.168.4.128@o2ib_0 changed server handle from 0xfff98862bf651c9e to 0xfff98862bf652136
May 17 18:39:43 client-18 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import
May 17 18:39:43 client-18 kernel: Lustre: Skipped 4 previous similar messages
May 17 18:41:36 client-18 xinetd[3390]: START: shell pid=27795 from=192.168.4.5
May 17 18:41:36 client-18 rshd[27796]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 18:41:36 client-18 xinetd[3390]: EXIT: shell status=0 pid=27795 duration=0(sec)
May 17 18:41:36 client-18 xinetd[3390]: START: shell pid=27831 from=192.168.4.5
May 17 18:41:36 client-18 rshd[27832]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 18:41:36 client-18 xinetd[3390]: EXIT: shell status=0 pid=27831 duration=0(sec)
May 17 18:41:36 client-18 xinetd[3390]: START: shell pid=27868 from=192.168.4.5
May 17 18:41:36 client-18 rshd[27869]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-vbr test 4i: setattr of times does not change versions ===================================== 18:41:36 \(1305682896\)");echo XXRETCODE:$?'
May 17 18:41:36 client-18 kernel: Lustre: DEBUG MARKER: == replay-vbr test 4i: setattr of times does not change versions ===================================== 18:41:36 (1305682896)
May 17 18:41:36 client-18 xinetd[3390]: EXIT: shell status=0 pid=27868 duration=0(sec)
May 17 18:41:37 client-18 xinetd[3390]: START: shell pid=27903 from=192.168.4.5
May 17 18:41:37 client-18 rshd[27904]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 18:41:37 client-18 xinetd[3390]: EXIT: shell status=0 pid=27903 duration=0(sec)
May 17 18:41:37 client-18 xinetd[3390]: START: shell pid=27939 from=192.168.4.5
May 17 18:41:37 client-18 rshd[27940]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 18:41:37 client-18 xinetd[3390]: EXIT: shell status=0 pid=27939 duration=0(sec)
May 17 18:41:37 client-18 xinetd[3390]: START: shell pid=27976 from=192.168.4.5
May 17 18:41:37 client-18 rshd[27977]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-vbr test 4j: setattr of size does not change versions ====================================== 18:41:37 \(1305682897\)");echo XXRETCODE:$?'
May 17 18:41:37 client-18 kernel: Lustre: DEBUG MARKER: == replay-vbr test 4j: setattr of size does not change versions ====================================== 18:41:37 (1305682897)
May 17 18:41:37 client-18 xinetd[3390]: EXIT: shell status=0 pid=27976 duration=0(sec)
May 17 18:41:38 client-18 xinetd[3390]: START: shell pid=28011 from=192.168.4.5
May 17 18:41:38 client-18 rshd[28012]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 18:41:38 client-18 xinetd[3390]: EXIT: shell status=0 pid=28011 duration=0(sec)
May 17 18:41:38 client-18 xinetd[3390]: START: shell pid=28047 from=192.168.4.5
May 17 18:41:38 client-18 rshd[28048]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 18:41:38 client-18 xinetd[3390]: EXIT: shell status=0 pid=28047 duration=0(sec)
May 17 18:41:38 client-18 xinetd[3390]: START: shell pid=28084 from=192.168.4.5
May 17 18:41:38 client-18 rshd[28085]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-vbr test 4k: setattr of times and size does not check versions ============================= 18:41:38 \(1305682898\)");echo XXRETCODE:$?'
May 17 18:41:38 client-18 kernel: Lustre: DEBUG MARKER: == replay-vbr test 4k: setattr of times and size does not check versions ============================= 18:41:38 (1305682898)
May 17 18:41:38 client-18 xinetd[3390]: EXIT: shell status=0 pid=28084 duration=0(sec)
May 17 18:41:38 client-18 xinetd[3390]: START: shell pid=28119 from=192.168.4.5
May 17 18:41:38 client-18 rshd[28120]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mkdir -p /mnt/lustre2");echo XXRETCODE:$?'
May 17 18:41:38 client-18 xinetd[3390]: EXIT: shell status=0 pid=28119 duration=0(sec)
May 17 18:41:38 client-18 xinetd[3390]: START: shell pid=28154 from=192.168.4.5
May 17 18:41:38 client-18 rshd[28155]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre2");echo XXRETCODE:$?'
May 17 18:41:38 client-18 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'.
May 17 18:41:38 client-18 kernel: Lustre: 28189:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff810327f61400->192.168.4.128@o2ib netid 50000: select flavor null
May 17 18:41:38 client-18 kernel: Lustre: 28189:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 6 previous similar messages
May 17 18:41:38 client-18 kernel: Lustre: Client lustre-client has started
May 17 18:41:38 client-18 xinetd[3390]: EXIT: shell status=0 pid=28154 duration=0(sec)
May 17 18:41:38 client-18 xinetd[3390]: START: shell pid=28192 from=192.168.4.5
May 17 18:41:38 client-18 rshd[28193]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param debug=\"0x33f0406\";         lctl set_param subsystem_debug=\"0xffb7e3ff\";         lctl set_param debug_mb=32");echo XXRETCODE:$?'
May 17 18:41:38 client-18 xinetd[3390]: EXIT: shell status=0 pid=28192 duration=0(sec)
May 17 18:41:39 client-18 xinetd[3390]: START: shell pid=28229 from=192.168.4.5
May 17 18:41:39 client-18 rshd[28230]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 18:41:39 client-18 xinetd[3390]: EXIT: shell status=0 pid=28229 duration=0(sec)
May 17 18:41:39 client-18 xinetd[3390]: START: shell pid=28268 from=192.168.4.5
May 17 18:41:39 client-18 rshd[28269]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "chmod 666 /mnt/lustre2/f4k");echo XXRETCODE:$?'
May 17 18:41:39 client-18 xinetd[3390]: EXIT: shell status=0 pid=28268 duration=0(sec)
May 17 18:41:41 client-18 xinetd[3390]: START: shell pid=28303 from=192.168.4.5
May 17 18:41:41 client-18 rshd[28304]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "grep -c /mnt/lustre2' ' /proc/mounts");echo XXRETCODE:$?'
May 17 18:41:41 client-18 xinetd[3390]: START: shell pid=28338 from=192.168.4.5
May 17 18:41:41 client-18 xinetd[3390]: EXIT: shell status=0 pid=28303 duration=0(sec)
May 17 18:41:41 client-18 rshd[28339]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lsof -t /mnt/lustre2");echo XXRETCODE:$?'
May 17 18:41:42 client-18 xinetd[3390]: EXIT: shell status=0 pid=28338 duration=1(sec)
May 17 18:41:42 client-18 xinetd[3390]: START: shell pid=28374 from=192.168.4.5
May 17 18:41:42 client-18 rshd[28375]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "umount  /mnt/lustre2 2>&1");echo XXRETCODE:$?'
May 17 18:41:42 client-18 kernel: LustreError: 28409:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway
May 17 18:41:42 client-18 kernel: LustreError: 28409:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Skipped 1 previous similar message
May 17 18:41:42 client-18 kernel: LustreError: 28409:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108
May 17 18:41:42 client-18 kernel: LustreError: 28409:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) Skipped 1 previous similar message
May 17 18:41:42 client-18 kernel: Lustre: client ffff810327f61400 umount complete
May 17 18:41:42 client-18 kernel: Lustre: Skipped 1 previous similar message
May 17 18:41:42 client-18 xinetd[3390]: EXIT: shell status=0 pid=28374 duration=0(sec)
May 17 18:42:00 client-18 kernel: Lustre: 8885:0:(import.c:885:ptlrpc_connect_interpret()) MGS@MGC192.168.4.128@o2ib_0 changed server handle from 0xfff98862bf652136 to 0xfff98862bf65263e
May 17 18:43:32 client-18 xinetd[3390]: START: shell pid=28418 from=192.168.4.5
May 17 18:43:32 client-18 rshd[28419]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 18:43:32 client-18 xinetd[3390]: EXIT: shell status=0 pid=28418 duration=0(sec)
May 17 18:43:32 client-18 xinetd[3390]: START: shell pid=28454 from=192.168.4.5
May 17 18:43:32 client-18 rshd[28455]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 18:43:32 client-18 xinetd[3390]: EXIT: shell status=0 pid=28454 duration=0(sec)
May 17 18:43:32 client-18 xinetd[3390]: START: shell pid=28491 from=192.168.4.5
May 17 18:43:32 client-18 rshd[28492]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-vbr test 5a: link changes versions of source but not target parent ========================= 18:43:32 \(1305683012\)");echo XXRETCODE:$?'
May 17 18:43:32 client-18 kernel: Lustre: DEBUG MARKER: == replay-vbr test 5a: link changes versions of source but not target parent ========================= 18:43:32 (1305683012)
May 17 18:43:32 client-18 xinetd[3390]: EXIT: shell status=0 pid=28491 duration=0(sec)
May 17 18:43:33 client-18 xinetd[3390]: START: shell pid=28526 from=192.168.4.5
May 17 18:43:33 client-18 rshd[28527]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 18:43:33 client-18 xinetd[3390]: EXIT: shell status=0 pid=28526 duration=0(sec)
May 17 18:43:33 client-18 xinetd[3390]: START: shell pid=28562 from=192.168.4.5
May 17 18:43:33 client-18 rshd[28563]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 18:43:33 client-18 xinetd[3390]: EXIT: shell status=0 pid=28562 duration=0(sec)
May 17 18:43:33 client-18 xinetd[3390]: START: shell pid=28599 from=192.168.4.5
May 17 18:43:33 client-18 rshd[28600]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-vbr test 5b: link checks version of target parent ========================================== 18:43:33 \(1305683013\)");echo XXRETCODE:$?'
May 17 18:43:33 client-18 kernel: Lustre: DEBUG MARKER: == replay-vbr test 5b: link checks version of target parent ========================================== 18:43:33 (1305683013)
May 17 18:43:33 client-18 xinetd[3390]: EXIT: shell status=0 pid=28599 duration=0(sec)
May 17 18:43:33 client-18 xinetd[3390]: START: shell pid=28634 from=192.168.4.5
May 17 18:43:33 client-18 rshd[28635]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mkdir -p /mnt/lustre2");echo XXRETCODE:$?'
May 17 18:43:33 client-18 xinetd[3390]: START: shell pid=28669 from=192.168.4.5
May 17 18:43:33 client-18 xinetd[3390]: EXIT: shell status=0 pid=28634 duration=0(sec)
May 17 18:43:33 client-18 rshd[28670]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre2");echo XXRETCODE:$?'
May 17 18:43:34 client-18 xinetd[3390]: START: shell pid=28707 from=192.168.4.5
May 17 18:43:34 client-18 xinetd[3390]: EXIT: shell status=0 pid=28669 duration=1(sec)
May 17 18:43:34 client-18 rshd[28708]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param debug=\"0x33f0406\";         lctl set_param subsystem_debug=\"0xffb7e3ff\";         lctl set_param debug_mb=32");echo XXRETCODE:$?'
May 17 18:43:34 client-18 kernel: Lustre: 28742:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release.
May 17 18:43:34 client-18 kernel: Lustre: 28742:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 3 previous similar messages
May 17 18:43:34 client-18 xinetd[3390]: EXIT: shell status=0 pid=28707 duration=0(sec)
May 17 18:43:34 client-18 xinetd[3390]: START: shell pid=28744 from=192.168.4.5
May 17 18:43:34 client-18 rshd[28745]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 18:43:34 client-18 xinetd[3390]: EXIT: shell status=0 pid=28744 duration=0(sec)
May 17 18:43:34 client-18 xinetd[3390]: START: shell pid=28783 from=192.168.4.5
May 17 18:43:35 client-18 rshd[28784]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "chmod 777 /mnt/lustre2/d0.replay-vbr/d5");echo XXRETCODE:$?'
May 17 18:43:35 client-18 xinetd[3390]: EXIT: shell status=0 pid=28783 duration=1(sec)
May 17 18:43:35 client-18 xinetd[3390]: START: shell pid=28818 from=192.168.4.5
May 17 18:43:35 client-18 rshd[28819]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "grep -c /mnt/lustre2' ' /proc/mounts");echo XXRETCODE:$?'
May 17 18:43:35 client-18 xinetd[3390]: EXIT: shell status=0 pid=28818 duration=0(sec)
May 17 18:43:35 client-18 xinetd[3390]: START: shell pid=28853 from=192.168.4.5
May 17 18:43:35 client-18 rshd[28854]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lsof -t /mnt/lustre2");echo XXRETCODE:$?'
May 17 18:43:35 client-18 xinetd[3390]: START: shell pid=28889 from=192.168.4.5
May 17 18:43:35 client-18 xinetd[3390]: EXIT: shell status=0 pid=28853 duration=0(sec)
May 17 18:43:35 client-18 rshd[28890]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "umount  /mnt/lustre2 2>&1");echo XXRETCODE:$?'
May 17 18:43:35 client-18 xinetd[3390]: EXIT: shell status=0 pid=28889 duration=0(sec)
May 17 18:43:54 client-18 kernel: Lustre: 8885:0:(import.c:885:ptlrpc_connect_interpret()) MGS@MGC192.168.4.128@o2ib_0 changed server handle from 0xfff98862bf65263e to 0xfff98862bf652b3f
May 17 18:45:25 client-18 kernel: Lustre: lustre-MDT0000-mdc-ffff8103028bc400: Connection restored to service lustre-MDT0000 using nid 192.168.4.128@o2ib.
May 17 18:45:25 client-18 kernel: Lustre: Skipped 8 previous similar messages
May 17 18:45:58 client-18 xinetd[3390]: START: shell pid=28937 from=192.168.4.5
May 17 18:45:58 client-18 rshd[28938]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 18:45:58 client-18 xinetd[3390]: EXIT: shell status=0 pid=28937 duration=0(sec)
May 17 18:45:58 client-18 xinetd[3390]: START: shell pid=28973 from=192.168.4.5
May 17 18:45:58 client-18 rshd[28974]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 18:45:58 client-18 xinetd[3390]: EXIT: shell status=0 pid=28973 duration=0(sec)
May 17 18:45:58 client-18 xinetd[3390]: START: shell pid=29010 from=192.168.4.5
May 17 18:45:58 client-18 rshd[29011]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-vbr test 5c: link checks version of source ================================================= 18:45:58 \(1305683158\)");echo XXRETCODE:$?'
May 17 18:45:58 client-18 kernel: Lustre: DEBUG MARKER: == replay-vbr test 5c: link checks version of source ================================================= 18:45:58 (1305683158)
May 17 18:45:58 client-18 xinetd[3390]: EXIT: shell status=0 pid=29010 duration=0(sec)
May 17 18:45:58 client-18 xinetd[3390]: START: shell pid=29045 from=192.168.4.5
May 17 18:45:58 client-18 rshd[29046]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mkdir -p /mnt/lustre2");echo XXRETCODE:$?'
May 17 18:45:58 client-18 xinetd[3390]: START: shell pid=29080 from=192.168.4.5
May 17 18:45:58 client-18 xinetd[3390]: EXIT: shell status=0 pid=29045 duration=0(sec)
May 17 18:45:58 client-18 rshd[29081]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre2");echo XXRETCODE:$?'
May 17 18:45:58 client-18 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'.
May 17 18:45:58 client-18 kernel: LustreError: Skipped 1 previous similar message
May 17 18:45:58 client-18 kernel: Lustre: 29115:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff8102ee4a0000->192.168.4.128@o2ib netid 50000: select flavor null
May 17 18:45:58 client-18 kernel: Lustre: 29115:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 13 previous similar messages
May 17 18:45:58 client-18 kernel: Lustre: Client lustre-client has started
May 17 18:45:58 client-18 kernel: Lustre: Skipped 1 previous similar message
May 17 18:45:58 client-18 xinetd[3390]: START: shell pid=29118 from=192.168.4.5
May 17 18:45:58 client-18 xinetd[3390]: EXIT: shell status=0 pid=29080 duration=0(sec)
May 17 18:45:58 client-18 rshd[29119]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param debug=\"0x33f0406\";         lctl set_param subsystem_debug=\"0xffb7e3ff\";         lctl set_param debug_mb=32");echo XXRETCODE:$?'
May 17 18:45:59 client-18 xinetd[3390]: EXIT: shell status=0 pid=29118 duration=1(sec)
May 17 18:45:59 client-18 xinetd[3390]: START: shell pid=29155 from=192.168.4.5
May 17 18:45:59 client-18 rshd[29156]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 18:45:59 client-18 xinetd[3390]: EXIT: shell status=0 pid=29155 duration=0(sec)
May 17 18:45:59 client-18 xinetd[3390]: START: shell pid=29194 from=192.168.4.5
May 17 18:45:59 client-18 rshd[29195]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "chmod 666 /mnt/lustre2/f5c");echo XXRETCODE:$?'
May 17 18:45:59 client-18 xinetd[3390]: EXIT: shell status=0 pid=29194 duration=0(sec)
May 17 18:45:59 client-18 xinetd[3390]: START: shell pid=29229 from=192.168.4.5
May 17 18:45:59 client-18 rshd[29230]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "grep -c /mnt/lustre2' ' /proc/mounts");echo XXRETCODE:$?'
May 17 18:46:00 client-18 xinetd[3390]: START: shell pid=29264 from=192.168.4.5
May 17 18:46:00 client-18 xinetd[3390]: EXIT: shell status=0 pid=29229 duration=1(sec)
May 17 18:46:00 client-18 rshd[29265]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lsof -t /mnt/lustre2");echo XXRETCODE:$?'
May 17 18:46:00 client-18 xinetd[3390]: EXIT: shell status=0 pid=29264 duration=0(sec)
May 17 18:46:00 client-18 xinetd[3390]: START: shell pid=29300 from=192.168.4.5
May 17 18:46:00 client-18 rshd[29301]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "umount  /mnt/lustre2 2>&1");echo XXRETCODE:$?'
May 17 18:46:00 client-18 xinetd[3390]: EXIT: shell status=0 pid=29300 duration=0(sec)
May 17 18:46:18 client-18 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail.
May 17 18:46:18 client-18 kernel: LustreError: Skipped 4 previous similar messages
May 17 18:46:18 client-18 kernel: Lustre: lustre-MDT0000-mdc-ffff8103028bc400: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete.
May 17 18:46:18 client-18 kernel: Lustre: 8885:0:(import.c:885:ptlrpc_connect_interpret()) MGS@MGC192.168.4.128@o2ib_0 changed server handle from 0xfff98862bf652b3f to 0xfff98862bf652ee9
May 17 18:46:18 client-18 kernel: Lustre: Skipped 4 previous similar messages
May 17 18:46:41 client-18 kernel: Lustre: 8884:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369101282848262 sent from lustre-MDT0000-mdc-ffff8103028bc400 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305683168] [real_sent 1305683168] [current 1305683201] [deadline 33s] [delay 0s]  req@ffff81030659f400 x1369101282848262/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 192/192 e 0 to 1 dl 1305683201 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
May 17 18:46:41 client-18 kernel: Lustre: 8884:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 15 previous similar messages
May 17 18:48:22 client-18 xinetd[3390]: START: shell pid=29348 from=192.168.4.5
May 17 18:48:22 client-18 rshd[29349]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 18:48:22 client-18 xinetd[3390]: EXIT: shell status=0 pid=29348 duration=0(sec)
May 17 18:48:22 client-18 xinetd[3390]: START: shell pid=29384 from=192.168.4.5
May 17 18:48:22 client-18 rshd[29385]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 18:48:22 client-18 xinetd[3390]: EXIT: shell status=0 pid=29384 duration=0(sec)
May 17 18:48:22 client-18 xinetd[3390]: START: shell pid=29421 from=192.168.4.5
May 17 18:48:22 client-18 rshd[29422]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-vbr test 6a: rename doesn\'t change versions of source parent and target parent ============= 18:48:22 \(1305683302\)");echo XXRETCODE:$?'
May 17 18:48:23 client-18 kernel: Lustre: DEBUG MARKER: == replay-vbr test 6a: rename doesn't change versions of source parent and target parent ============= 18:48:22 (1305683302)
May 17 18:48:23 client-18 xinetd[3390]: EXIT: shell status=0 pid=29421 duration=1(sec)
May 17 18:48:23 client-18 xinetd[3390]: START: shell pid=29456 from=192.168.4.5
May 17 18:48:23 client-18 rshd[29457]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 18:48:23 client-18 xinetd[3390]: EXIT: shell status=0 pid=29456 duration=0(sec)
May 17 18:48:23 client-18 xinetd[3390]: START: shell pid=29492 from=192.168.4.5
May 17 18:48:23 client-18 rshd[29493]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 18:48:23 client-18 xinetd[3390]: EXIT: shell status=0 pid=29492 duration=0(sec)
May 17 18:48:23 client-18 xinetd[3390]: START: shell pid=29529 from=192.168.4.5
May 17 18:48:23 client-18 rshd[29530]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-vbr test 6b: rename within same dir doesn\'t change version of parent ======================= 18:48:23 \(1305683303\)");echo XXRETCODE:$?'
May 17 18:48:24 client-18 kernel: Lustre: DEBUG MARKER: == replay-vbr test 6b: rename within same dir doesn't change version of parent ======================= 18:48:23 (1305683303)
May 17 18:48:24 client-18 xinetd[3390]: EXIT: shell status=0 pid=29529 duration=1(sec)
May 17 18:48:24 client-18 xinetd[3390]: START: shell pid=29564 from=192.168.4.5
May 17 18:48:24 client-18 rshd[29565]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 18:48:24 client-18 xinetd[3390]: EXIT: shell status=0 pid=29564 duration=0(sec)
May 17 18:48:24 client-18 xinetd[3390]: START: shell pid=29600 from=192.168.4.5
May 17 18:48:24 client-18 rshd[29601]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 18:48:24 client-18 xinetd[3390]: EXIT: shell status=0 pid=29600 duration=0(sec)
May 17 18:48:24 client-18 xinetd[3390]: START: shell pid=29637 from=192.168.4.5
May 17 18:48:24 client-18 rshd[29638]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-vbr test 6c: rename checks version of source parent ======================================== 18:48:24 \(1305683304\)");echo XXRETCODE:$?'
May 17 18:48:24 client-18 kernel: Lustre: DEBUG MARKER: == replay-vbr test 6c: rename checks version of source parent ======================================== 18:48:24 (1305683304)
May 17 18:48:24 client-18 xinetd[3390]: EXIT: shell status=0 pid=29637 duration=0(sec)
May 17 18:48:24 client-18 xinetd[3390]: START: shell pid=29672 from=192.168.4.5
May 17 18:48:24 client-18 rshd[29673]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mkdir -p /mnt/lustre2");echo XXRETCODE:$?'
May 17 18:48:24 client-18 xinetd[3390]: EXIT: shell status=0 pid=29672 duration=0(sec)
May 17 18:48:24 client-18 xinetd[3390]: START: shell pid=29707 from=192.168.4.5
May 17 18:48:24 client-18 rshd[29708]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre2");echo XXRETCODE:$?'
May 17 18:48:25 client-18 xinetd[3390]: EXIT: shell status=0 pid=29707 duration=1(sec)
May 17 18:48:25 client-18 xinetd[3390]: START: shell pid=29745 from=192.168.4.5
May 17 18:48:25 client-18 rshd[29746]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param debug=\"0x33f0406\";         lctl set_param subsystem_debug=\"0xffb7e3ff\";         lctl set_param debug_mb=32");echo XXRETCODE:$?'
May 17 18:48:25 client-18 xinetd[3390]: EXIT: shell status=0 pid=29745 duration=0(sec)
May 17 18:48:25 client-18 xinetd[3390]: START: shell pid=29782 from=192.168.4.5
May 17 18:48:25 client-18 rshd[29783]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 18:48:25 client-18 xinetd[3390]: EXIT: shell status=0 pid=29782 duration=0(sec)
May 17 18:48:25 client-18 xinetd[3390]: START: shell pid=29821 from=192.168.4.5
May 17 18:48:25 client-18 rshd[29822]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "chmod 777 /mnt/lustre2");echo XXRETCODE:$?'
May 17 18:48:25 client-18 xinetd[3390]: EXIT: shell status=0 pid=29821 duration=0(sec)
May 17 18:48:26 client-18 xinetd[3390]: START: shell pid=29856 from=192.168.4.5
May 17 18:48:26 client-18 rshd[29857]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "grep -c /mnt/lustre2' ' /proc/mounts");echo XXRETCODE:$?'
May 17 18:48:26 client-18 xinetd[3390]: EXIT: shell status=0 pid=29856 duration=0(sec)
May 17 18:48:26 client-18 xinetd[3390]: START: shell pid=29891 from=192.168.4.5
May 17 18:48:26 client-18 rshd[29892]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lsof -t /mnt/lustre2");echo XXRETCODE:$?'
May 17 18:48:26 client-18 xinetd[3390]: EXIT: shell status=0 pid=29891 duration=0(sec)
May 17 18:48:26 client-18 xinetd[3390]: START: shell pid=29927 from=192.168.4.5
May 17 18:48:26 client-18 rshd[29928]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "umount  /mnt/lustre2 2>&1");echo XXRETCODE:$?'
May 17 18:48:26 client-18 xinetd[3390]: EXIT: shell status=0 pid=29927 duration=0(sec)
May 17 18:48:46 client-18 kernel: Lustre: 8886:0:(import.c:529:import_select_connection()) MGC192.168.4.128@o2ib: tried all connections, increasing latency to 6s
May 17 18:48:46 client-18 kernel: Lustre: 8886:0:(import.c:529:import_select_connection()) Skipped 1 previous similar message
May 17 18:48:47 client-18 kernel: Lustre: 8885:0:(import.c:885:ptlrpc_connect_interpret()) MGS@MGC192.168.4.128@o2ib_0 changed server handle from 0xfff98862bf652ee9 to 0xfff98862bf653499
May 17 18:48:47 client-18 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The obd_ping operation failed with -107
May 17 18:48:47 client-18 kernel: LustreError: Skipped 6 previous similar messages
May 17 18:50:49 client-18 xinetd[3390]: START: shell pid=29975 from=192.168.4.5
May 17 18:50:49 client-18 rshd[29976]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 18:50:49 client-18 xinetd[3390]: EXIT: shell status=0 pid=29975 duration=0(sec)
May 17 18:50:49 client-18 xinetd[3390]: START: shell pid=30011 from=192.168.4.5
May 17 18:50:49 client-18 rshd[30012]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 18:50:49 client-18 xinetd[3390]: EXIT: shell status=0 pid=30011 duration=0(sec)
May 17 18:50:49 client-18 xinetd[3390]: START: shell pid=30048 from=192.168.4.5
May 17 18:50:49 client-18 rshd[30049]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-vbr test 6d: rename checks version of target parent ======================================== 18:50:49 \(1305683449\)");echo XXRETCODE:$?'
May 17 18:50:49 client-18 kernel: Lustre: DEBUG MARKER: == replay-vbr test 6d: rename checks version of target parent ======================================== 18:50:49 (1305683449)
May 17 18:50:49 client-18 xinetd[3390]: EXIT: shell status=0 pid=30048 duration=0(sec)
May 17 18:50:49 client-18 xinetd[3390]: START: shell pid=30083 from=192.168.4.5
May 17 18:50:49 client-18 rshd[30084]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mkdir -p /mnt/lustre2");echo XXRETCODE:$?'
May 17 18:50:49 client-18 xinetd[3390]: START: shell pid=30118 from=192.168.4.5
May 17 18:50:49 client-18 xinetd[3390]: EXIT: shell status=0 pid=30083 duration=0(sec)
May 17 18:50:49 client-18 rshd[30119]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre2");echo XXRETCODE:$?'
May 17 18:50:50 client-18 xinetd[3390]: EXIT: shell status=0 pid=30118 duration=1(sec)
May 17 18:50:50 client-18 xinetd[3390]: START: shell pid=30156 from=192.168.4.5
May 17 18:50:50 client-18 rshd[30157]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param debug=\"0x33f0406\";         lctl set_param subsystem_debug=\"0xffb7e3ff\";         lctl set_param debug_mb=32");echo XXRETCODE:$?'
May 17 18:50:50 client-18 xinetd[3390]: EXIT: shell status=0 pid=30156 duration=0(sec)
May 17 18:50:50 client-18 xinetd[3390]: START: shell pid=30193 from=192.168.4.5
May 17 18:50:50 client-18 rshd[30194]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 18:50:50 client-18 xinetd[3390]: EXIT: shell status=0 pid=30193 duration=0(sec)
May 17 18:50:50 client-18 xinetd[3390]: START: shell pid=30232 from=192.168.4.5
May 17 18:50:50 client-18 rshd[30233]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "chmod 777 /mnt/lustre2/d0.replay-vbr/d6");echo XXRETCODE:$?'
May 17 18:50:50 client-18 xinetd[3390]: EXIT: shell status=0 pid=30232 duration=0(sec)
May 17 18:50:50 client-18 xinetd[3390]: START: shell pid=30267 from=192.168.4.5
May 17 18:50:50 client-18 rshd[30268]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "grep -c /mnt/lustre2' ' /proc/mounts");echo XXRETCODE:$?'
May 17 18:50:51 client-18 xinetd[3390]: EXIT: shell status=0 pid=30267 duration=1(sec)
May 17 18:50:51 client-18 xinetd[3390]: START: shell pid=30302 from=192.168.4.5
May 17 18:50:51 client-18 rshd[30303]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lsof -t /mnt/lustre2");echo XXRETCODE:$?'
May 17 18:50:51 client-18 xinetd[3390]: EXIT: shell status=0 pid=30302 duration=0(sec)
May 17 18:50:51 client-18 xinetd[3390]: START: shell pid=30338 from=192.168.4.5
May 17 18:50:51 client-18 rshd[30339]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "umount  /mnt/lustre2 2>&1");echo XXRETCODE:$?'
May 17 18:50:51 client-18 kernel: LustreError: 30373:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway
May 17 18:50:51 client-18 kernel: LustreError: 30373:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Skipped 4 previous similar messages
May 17 18:50:51 client-18 kernel: LustreError: 30373:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108
May 17 18:50:51 client-18 kernel: LustreError: 30373:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) Skipped 4 previous similar messages
May 17 18:50:51 client-18 kernel: Lustre: client ffff8102ea018c00 umount complete
May 17 18:50:51 client-18 kernel: Lustre: Skipped 3 previous similar messages
May 17 18:50:51 client-18 xinetd[3390]: EXIT: shell status=0 pid=30338 duration=0(sec)
May 17 18:51:21 client-18 kernel: Lustre: 8885:0:(import.c:885:ptlrpc_connect_interpret()) MGS@MGC192.168.4.128@o2ib_0 changed server handle from 0xfff98862bf653499 to 0xfff98862bf6538cf
May 17 18:51:21 client-18 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import
May 17 18:51:21 client-18 kernel: Lustre: Skipped 4 previous similar messages
May 17 18:53:14 client-18 xinetd[3390]: START: shell pid=30386 from=192.168.4.5
May 17 18:53:14 client-18 rshd[30387]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 18:53:14 client-18 xinetd[3390]: EXIT: shell status=0 pid=30386 duration=0(sec)
May 17 18:53:14 client-18 xinetd[3390]: START: shell pid=30422 from=192.168.4.5
May 17 18:53:14 client-18 rshd[30423]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 18:53:14 client-18 xinetd[3390]: EXIT: shell status=0 pid=30422 duration=0(sec)
May 17 18:53:14 client-18 xinetd[3390]: START: shell pid=30459 from=192.168.4.5
May 17 18:53:14 client-18 rshd[30460]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-vbr test 7a: create, {lost}, create ======================================================== 18:53:14 \(1305683594\)");echo XXRETCODE:$?'
May 17 18:53:14 client-18 kernel: Lustre: DEBUG MARKER: == replay-vbr test 7a: create, {lost}, create ======================================================== 18:53:14 (1305683594)
May 17 18:53:14 client-18 xinetd[3390]: EXIT: shell status=0 pid=30459 duration=0(sec)
May 17 18:53:14 client-18 xinetd[3390]: START: shell pid=30494 from=192.168.4.5
May 17 18:53:14 client-18 rshd[30495]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mkdir -p /mnt/lustre2");echo XXRETCODE:$?'
May 17 18:53:14 client-18 xinetd[3390]: START: shell pid=30530 from=192.168.4.5
May 17 18:53:14 client-18 xinetd[3390]: EXIT: shell status=0 pid=30494 duration=0(sec)
May 17 18:53:14 client-18 rshd[30531]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre2");echo XXRETCODE:$?'
May 17 18:53:15 client-18 xinetd[3390]: EXIT: shell status=0 pid=30530 duration=1(sec)
May 17 18:53:15 client-18 xinetd[3390]: START: shell pid=30568 from=192.168.4.5
May 17 18:53:15 client-18 rshd[30569]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param debug=\"0x33f0406\";         lctl set_param subsystem_debug=\"0xffb7e3ff\";         lctl set_param debug_mb=32");echo XXRETCODE:$?'
May 17 18:53:15 client-18 kernel: Lustre: 30603:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release.
May 17 18:53:15 client-18 kernel: Lustre: 30603:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 7 previous similar messages
May 17 18:53:15 client-18 xinetd[3390]: EXIT: shell status=0 pid=30568 duration=0(sec)
May 17 18:53:15 client-18 xinetd[3390]: START: shell pid=30605 from=192.168.4.5
May 17 18:53:15 client-18 rshd[30606]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 18:53:15 client-18 xinetd[3390]: EXIT: shell status=0 pid=30605 duration=0(sec)
May 17 18:53:15 client-18 xinetd[3390]: START: shell pid=30644 from=192.168.4.5
May 17 18:53:15 client-18 rshd[30645]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "rm /mnt/lustre2/d0.replay-vbr/d7/f7a-0");echo XXRETCODE:$?'
May 17 18:53:16 client-18 xinetd[3390]: EXIT: shell status=0 pid=30644 duration=1(sec)
May 17 18:53:16 client-18 xinetd[3390]: START: shell pid=30679 from=192.168.4.5
May 17 18:53:16 client-18 rshd[30680]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "grep -c /mnt/lustre2' ' /proc/mounts");echo XXRETCODE:$?'
May 17 18:53:16 client-18 xinetd[3390]: EXIT: shell status=0 pid=30679 duration=0(sec)
May 17 18:53:16 client-18 xinetd[3390]: START: shell pid=30714 from=192.168.4.5
May 17 18:53:16 client-18 rshd[30715]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lsof -t /mnt/lustre2");echo XXRETCODE:$?'
May 17 18:53:16 client-18 xinetd[3390]: EXIT: shell status=0 pid=30714 duration=0(sec)
May 17 18:53:16 client-18 xinetd[3390]: START: shell pid=30750 from=192.168.4.5
May 17 18:53:16 client-18 rshd[30751]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "umount  /mnt/lustre2 2>&1");echo XXRETCODE:$?'
May 17 18:53:16 client-18 xinetd[3390]: EXIT: shell status=0 pid=30750 duration=0(sec)
May 17 18:53:34 client-18 kernel: Lustre: 8885:0:(import.c:885:ptlrpc_connect_interpret()) MGS@MGC192.168.4.128@o2ib_0 changed server handle from 0xfff98862bf6538cf to 0xfff98862bf653ce9
May 17 18:55:41 client-18 xinetd[3390]: START: shell pid=30798 from=192.168.4.5
May 17 18:55:41 client-18 rshd[30799]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mkdir -p /mnt/lustre2");echo XXRETCODE:$?'
May 17 18:55:41 client-18 xinetd[3390]: START: shell pid=30833 from=192.168.4.5
May 17 18:55:41 client-18 xinetd[3390]: EXIT: shell status=0 pid=30798 duration=0(sec)
May 17 18:55:41 client-18 rshd[30834]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre2");echo XXRETCODE:$?'
May 17 18:55:41 client-18 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'.
May 17 18:55:41 client-18 kernel: LustreError: Skipped 3 previous similar messages
May 17 18:55:41 client-18 kernel: Lustre: 30868:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff810321dd9400->192.168.4.128@o2ib netid 50000: select flavor null
May 17 18:55:41 client-18 kernel: Lustre: 30868:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 27 previous similar messages
May 17 18:55:41 client-18 kernel: Lustre: Client lustre-client has started
May 17 18:55:41 client-18 kernel: Lustre: Skipped 3 previous similar messages
May 17 18:55:41 client-18 xinetd[3390]: EXIT: shell status=0 pid=30833 duration=0(sec)
May 17 18:55:41 client-18 xinetd[3390]: START: shell pid=30871 from=192.168.4.5
May 17 18:55:41 client-18 rshd[30872]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param debug=\"0x33f0406\";         lctl set_param subsystem_debug=\"0xffb7e3ff\";         lctl set_param debug_mb=32");echo XXRETCODE:$?'
May 17 18:55:41 client-18 xinetd[3390]: EXIT: shell status=0 pid=30871 duration=0(sec)
May 17 18:55:42 client-18 xinetd[3390]: START: shell pid=30908 from=192.168.4.5
May 17 18:55:42 client-18 rshd[30909]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 18:55:42 client-18 xinetd[3390]: EXIT: shell status=0 pid=30908 duration=0(sec)
May 17 18:55:42 client-18 xinetd[3390]: START: shell pid=30947 from=192.168.4.5
May 17 18:55:42 client-18 rshd[30948]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "rm /mnt/lustre2/d0.replay-vbr/d7/f7a-0");echo XXRETCODE:$?'
May 17 18:55:42 client-18 xinetd[3390]: EXIT: shell status=0 pid=30947 duration=0(sec)
May 17 18:55:42 client-18 xinetd[3390]: START: shell pid=30982 from=192.168.4.5
May 17 18:55:42 client-18 rshd[30983]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "grep -c /mnt/lustre2' ' /proc/mounts");echo XXRETCODE:$?'
May 17 18:55:42 client-18 xinetd[3390]: EXIT: shell status=0 pid=30982 duration=0(sec)
May 17 18:55:42 client-18 xinetd[3390]: START: shell pid=31017 from=192.168.4.5
May 17 18:55:42 client-18 rshd[31018]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lsof -t /mnt/lustre2");echo XXRETCODE:$?'
May 17 18:55:43 client-18 xinetd[3390]: EXIT: shell status=0 pid=31017 duration=1(sec)
May 17 18:55:43 client-18 xinetd[3390]: START: shell pid=31053 from=192.168.4.5
May 17 18:55:43 client-18 rshd[31054]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "umount  /mnt/lustre2 2>&1");echo XXRETCODE:$?'
May 17 18:55:43 client-18 xinetd[3390]: EXIT: shell status=0 pid=31053 duration=0(sec)
May 17 18:56:01 client-18 kernel: Lustre: 8885:0:(import.c:885:ptlrpc_connect_interpret()) MGS@MGC192.168.4.128@o2ib_0 changed server handle from 0xfff98862bf653ce9 to 0xfff98862bf654196
May 17 18:56:01 client-18 kernel: Lustre: MGC192.168.4.128@o2ib: Connection restored to service MGS using nid 192.168.4.128@o2ib.
May 17 18:56:01 client-18 kernel: Lustre: Skipped 8 previous similar messages
May 17 18:58:08 client-18 xinetd[3390]: START: shell pid=31101 from=192.168.4.5
May 17 18:58:08 client-18 rshd[31102]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mkdir -p /mnt/lustre2");echo XXRETCODE:$?'
May 17 18:58:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=31101 duration=0(sec)
May 17 18:58:08 client-18 xinetd[3390]: START: shell pid=31136 from=192.168.4.5
May 17 18:58:08 client-18 rshd[31137]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre2");echo XXRETCODE:$?'
May 17 18:58:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=31136 duration=0(sec)
May 17 18:58:08 client-18 xinetd[3390]: START: shell pid=31174 from=192.168.4.5
May 17 18:58:08 client-18 rshd[31175]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param debug=\"0x33f0406\";         lctl set_param subsystem_debug=\"0xffb7e3ff\";         lctl set_param debug_mb=32");echo XXRETCODE:$?'
May 17 18:58:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=31174 duration=0(sec)
May 17 18:58:08 client-18 xinetd[3390]: START: shell pid=31211 from=192.168.4.5
May 17 18:58:08 client-18 rshd[31212]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 18:58:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=31211 duration=0(sec)
May 17 18:58:09 client-18 xinetd[3390]: START: shell pid=31250 from=192.168.4.5
May 17 18:58:09 client-18 rshd[31251]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mv /mnt/lustre2/d0.replay-vbr/d7/f7a-0 /mnt/lustre2/d0.replay-vbr/d7/f7a-1");echo XXRETCODE:$?'
May 17 18:58:09 client-18 xinetd[3390]: EXIT: shell status=0 pid=31250 duration=0(sec)
May 17 18:58:09 client-18 xinetd[3390]: START: shell pid=31285 from=192.168.4.5
May 17 18:58:09 client-18 rshd[31286]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "grep -c /mnt/lustre2' ' /proc/mounts");echo XXRETCODE:$?'
May 17 18:58:09 client-18 xinetd[3390]: EXIT: shell status=0 pid=31285 duration=0(sec)
May 17 18:58:09 client-18 xinetd[3390]: START: shell pid=31320 from=192.168.4.5
May 17 18:58:09 client-18 rshd[31321]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lsof -t /mnt/lustre2");echo XXRETCODE:$?'
May 17 18:58:09 client-18 xinetd[3390]: EXIT: shell status=0 pid=31320 duration=0(sec)
May 17 18:58:09 client-18 xinetd[3390]: START: shell pid=31356 from=192.168.4.5
May 17 18:58:09 client-18 rshd[31357]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "umount  /mnt/lustre2 2>&1");echo XXRETCODE:$?'
May 17 18:58:10 client-18 xinetd[3390]: EXIT: shell status=0 pid=31356 duration=1(sec)
May 17 18:58:28 client-18 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail.
May 17 18:58:28 client-18 kernel: LustreError: Skipped 4 previous similar messages
May 17 18:58:28 client-18 kernel: Lustre: lustre-MDT0000-mdc-ffff8103028bc400: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete.
May 17 18:58:28 client-18 kernel: Lustre: 8885:0:(import.c:885:ptlrpc_connect_interpret()) MGS@MGC192.168.4.128@o2ib_0 changed server handle from 0xfff98862bf654196 to 0xfff98862bf654689
May 17 18:58:28 client-18 kernel: Lustre: Skipped 4 previous similar messages
May 17 18:58:51 client-18 kernel: Lustre: 8884:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369101282849527 sent from lustre-MDT0000-mdc-ffff8103028bc400 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305683898] [real_sent 1305683898] [current 1305683931] [deadline 33s] [delay 0s]  req@ffff8102ee809400 x1369101282849527/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 192/192 e 0 to 1 dl 1305683931 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
May 17 18:58:51 client-18 kernel: Lustre: 8884:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 13 previous similar messages
May 17 19:00:34 client-18 xinetd[3390]: START: shell pid=31404 from=192.168.4.5
May 17 19:00:34 client-18 rshd[31405]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 19:00:35 client-18 xinetd[3390]: EXIT: shell status=0 pid=31404 duration=1(sec)
May 17 19:00:35 client-18 xinetd[3390]: START: shell pid=31440 from=192.168.4.5
May 17 19:00:35 client-18 rshd[31441]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 19:00:35 client-18 xinetd[3390]: EXIT: shell status=0 pid=31440 duration=0(sec)
May 17 19:00:35 client-18 xinetd[3390]: START: shell pid=31477 from=192.168.4.5
May 17 19:00:35 client-18 rshd[31478]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-vbr test 7b: create, {lost}, unlink ======================================================== 19:00:35 \(1305684035\)");echo XXRETCODE:$?'
May 17 19:00:35 client-18 kernel: Lustre: DEBUG MARKER: == replay-vbr test 7b: create, {lost}, unlink ======================================================== 19:00:35 (1305684035)
May 17 19:00:35 client-18 xinetd[3390]: EXIT: shell status=0 pid=31477 duration=0(sec)
May 17 19:00:35 client-18 xinetd[3390]: START: shell pid=31512 from=192.168.4.5
May 17 19:00:35 client-18 rshd[31513]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mkdir -p /mnt/lustre2");echo XXRETCODE:$?'
May 17 19:00:35 client-18 xinetd[3390]: EXIT: shell status=0 pid=31512 duration=0(sec)
May 17 19:00:35 client-18 xinetd[3390]: START: shell pid=31547 from=192.168.4.5
May 17 19:00:35 client-18 rshd[31548]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre2");echo XXRETCODE:$?'
May 17 19:00:35 client-18 xinetd[3390]: START: shell pid=31585 from=192.168.4.5
May 17 19:00:35 client-18 xinetd[3390]: EXIT: shell status=0 pid=31547 duration=0(sec)
May 17 19:00:35 client-18 rshd[31586]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param debug=\"0x33f0406\";         lctl set_param subsystem_debug=\"0xffb7e3ff\";         lctl set_param debug_mb=32");echo XXRETCODE:$?'
May 17 19:00:35 client-18 xinetd[3390]: EXIT: shell status=0 pid=31585 duration=0(sec)
May 17 19:00:36 client-18 xinetd[3390]: START: shell pid=31622 from=192.168.4.5
May 17 19:00:36 client-18 rshd[31623]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 19:00:36 client-18 xinetd[3390]: EXIT: shell status=0 pid=31622 duration=0(sec)
May 17 19:00:36 client-18 xinetd[3390]: START: shell pid=31661 from=192.168.4.5
May 17 19:00:36 client-18 rshd[31662]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "rm /mnt/lustre2/d0.replay-vbr/d7/f7b-0; createmany -o /mnt/lustre2/d0.replay-vbr/d7/f7b- 1");echo XXRETCODE:$?'
May 17 19:00:36 client-18 xinetd[3390]: EXIT: shell status=0 pid=31661 duration=0(sec)
May 17 19:00:36 client-18 xinetd[3390]: START: shell pid=31698 from=192.168.4.5
May 17 19:00:36 client-18 rshd[31699]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "grep -c /mnt/lustre2' ' /proc/mounts");echo XXRETCODE:$?'
May 17 19:00:36 client-18 xinetd[3390]: START: shell pid=31733 from=192.168.4.5
May 17 19:00:36 client-18 xinetd[3390]: EXIT: shell status=0 pid=31698 duration=0(sec)
May 17 19:00:36 client-18 rshd[31734]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lsof -t /mnt/lustre2");echo XXRETCODE:$?'
May 17 19:00:36 client-18 xinetd[3390]: EXIT: shell status=0 pid=31733 duration=0(sec)
May 17 19:00:36 client-18 xinetd[3390]: START: shell pid=31769 from=192.168.4.5
May 17 19:00:37 client-18 rshd[31770]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "umount  /mnt/lustre2 2>&1");echo XXRETCODE:$?'
May 17 19:00:37 client-18 xinetd[3390]: EXIT: shell status=0 pid=31769 duration=1(sec)
May 17 19:00:57 client-18 kernel: Lustre: 8886:0:(import.c:529:import_select_connection()) MGC192.168.4.128@o2ib: tried all connections, increasing latency to 6s
May 17 19:00:57 client-18 kernel: Lustre: 8886:0:(import.c:529:import_select_connection()) Skipped 1 previous similar message
May 17 19:00:57 client-18 kernel: Lustre: 8885:0:(import.c:885:ptlrpc_connect_interpret()) MGS@MGC192.168.4.128@o2ib_0 changed server handle from 0xfff98862bf654689 to 0xfff98862bf654b98
May 17 19:00:57 client-18 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The obd_ping operation failed with -107
May 17 19:00:57 client-18 kernel: LustreError: Skipped 7 previous similar messages
May 17 19:03:01 client-18 xinetd[3390]: START: shell pid=31823 from=192.168.4.5
May 17 19:03:01 client-18 rshd[31824]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mkdir -p /mnt/lustre2");echo XXRETCODE:$?'
May 17 19:03:01 client-18 xinetd[3390]: START: shell pid=31858 from=192.168.4.5
May 17 19:03:01 client-18 xinetd[3390]: EXIT: shell status=0 pid=31823 duration=0(sec)
May 17 19:03:01 client-18 rshd[31859]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre2");echo XXRETCODE:$?'
May 17 19:03:02 client-18 xinetd[3390]: EXIT: shell status=0 pid=31858 duration=1(sec)
May 17 19:03:02 client-18 xinetd[3390]: START: shell pid=31896 from=192.168.4.5
May 17 19:03:02 client-18 rshd[31897]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param debug=\"0x33f0406\";         lctl set_param subsystem_debug=\"0xffb7e3ff\";         lctl set_param debug_mb=32");echo XXRETCODE:$?'
May 17 19:03:02 client-18 xinetd[3390]: EXIT: shell status=0 pid=31896 duration=0(sec)
May 17 19:03:02 client-18 xinetd[3390]: START: shell pid=31933 from=192.168.4.5
May 17 19:03:02 client-18 rshd[31934]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 19:03:02 client-18 xinetd[3390]: EXIT: shell status=0 pid=31933 duration=0(sec)
May 17 19:03:02 client-18 xinetd[3390]: START: shell pid=31972 from=192.168.4.5
May 17 19:03:02 client-18 rshd[31973]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "touch /mnt/lustre2/d0.replay-vbr/d7/f7b; mv /mnt/lustre2/d0.replay-vbr/d7/f7b /mnt/lustre2/d0.replay-vbr/d7/f7b-0");echo XXRETCODE:$?'
May 17 19:03:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=31972 duration=1(sec)
May 17 19:03:03 client-18 xinetd[3390]: START: shell pid=32009 from=192.168.4.5
May 17 19:03:03 client-18 rshd[32010]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "grep -c /mnt/lustre2' ' /proc/mounts");echo XXRETCODE:$?'
May 17 19:03:03 client-18 xinetd[3390]: START: shell pid=32044 from=192.168.4.5
May 17 19:03:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=32009 duration=0(sec)
May 17 19:03:03 client-18 rshd[32045]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lsof -t /mnt/lustre2");echo XXRETCODE:$?'
May 17 19:03:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=32044 duration=0(sec)
May 17 19:03:03 client-18 xinetd[3390]: START: shell pid=32080 from=192.168.4.5
May 17 19:03:03 client-18 rshd[32081]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "umount  /mnt/lustre2 2>&1");echo XXRETCODE:$?'
May 17 19:03:03 client-18 kernel: LustreError: 32115:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway
May 17 19:03:03 client-18 kernel: LustreError: 32115:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Skipped 13 previous similar messages
May 17 19:03:03 client-18 kernel: LustreError: 32115:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108
May 17 19:03:03 client-18 kernel: LustreError: 32115:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) Skipped 13 previous similar messages
May 17 19:03:03 client-18 kernel: Lustre: client ffff8102e68e8c00 umount complete
May 17 19:03:03 client-18 kernel: Lustre: Skipped 4 previous similar messages
May 17 19:03:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=32080 duration=0(sec)
May 17 19:03:34 client-18 kernel: Lustre: 8885:0:(import.c:885:ptlrpc_connect_interpret()) MGS@MGC192.168.4.128@o2ib_0 changed server handle from 0xfff98862bf654b98 to 0xfff98862bf6550b5
May 17 19:03:34 client-18 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import
May 17 19:03:34 client-18 kernel: Lustre: Skipped 4 previous similar messages
May 17 19:05:28 client-18 xinetd[3390]: START: shell pid=32128 from=192.168.4.5
May 17 19:05:28 client-18 rshd[32129]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mkdir -p /mnt/lustre2");echo XXRETCODE:$?'
May 17 19:05:28 client-18 xinetd[3390]: START: shell pid=32163 from=192.168.4.5
May 17 19:05:28 client-18 xinetd[3390]: EXIT: shell status=0 pid=32128 duration=0(sec)
May 17 19:05:28 client-18 rshd[32164]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre2");echo XXRETCODE:$?'
May 17 19:05:28 client-18 xinetd[3390]: START: shell pid=32201 from=192.168.4.5
May 17 19:05:28 client-18 xinetd[3390]: EXIT: shell status=0 pid=32163 duration=0(sec)
May 17 19:05:28 client-18 rshd[32202]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param debug=\"0x33f0406\";         lctl set_param subsystem_debug=\"0xffb7e3ff\";         lctl set_param debug_mb=32");echo XXRETCODE:$?'
May 17 19:05:29 client-18 kernel: Lustre: 32236:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release.
May 17 19:05:29 client-18 kernel: Lustre: 32236:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 9 previous similar messages
May 17 19:05:29 client-18 xinetd[3390]: EXIT: shell status=0 pid=32201 duration=1(sec)
May 17 19:05:29 client-18 xinetd[3390]: START: shell pid=32238 from=192.168.4.5
May 17 19:05:29 client-18 rshd[32239]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 19:05:29 client-18 xinetd[3390]: EXIT: shell status=0 pid=32238 duration=0(sec)
May 17 19:05:29 client-18 xinetd[3390]: START: shell pid=32277 from=192.168.4.5
May 17 19:05:29 client-18 rshd[32278]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "rm /mnt/lustre2/d0.replay-vbr/d7/f7b-0; mkdir /mnt/lustre2/d0.replay-vbr/d7/f7b-0");echo XXRETCODE:$?'
May 17 19:05:29 client-18 xinetd[3390]: EXIT: shell status=0 pid=32277 duration=0(sec)
May 17 19:05:29 client-18 xinetd[3390]: START: shell pid=32314 from=192.168.4.5
May 17 19:05:29 client-18 rshd[32315]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "grep -c /mnt/lustre2' ' /proc/mounts");echo XXRETCODE:$?'
May 17 19:05:30 client-18 xinetd[3390]: EXIT: shell status=0 pid=32314 duration=1(sec)
May 17 19:05:30 client-18 xinetd[3390]: START: shell pid=32349 from=192.168.4.5
May 17 19:05:30 client-18 rshd[32350]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lsof -t /mnt/lustre2");echo XXRETCODE:$?'
May 17 19:05:30 client-18 xinetd[3390]: EXIT: shell status=0 pid=32349 duration=0(sec)
May 17 19:05:30 client-18 xinetd[3390]: START: shell pid=32385 from=192.168.4.5
May 17 19:05:30 client-18 rshd[32386]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "umount  /mnt/lustre2 2>&1");echo XXRETCODE:$?'
May 17 19:05:30 client-18 xinetd[3390]: EXIT: shell status=0 pid=32385 duration=0(sec)
May 17 19:05:48 client-18 kernel: Lustre: 8885:0:(import.c:885:ptlrpc_connect_interpret()) MGS@MGC192.168.4.128@o2ib_0 changed server handle from 0xfff98862bf6550b5 to 0xfff98862bf655554
May 17 19:07:20 client-18 kernel: Lustre: lustre-MDT0000-mdc-ffff8103028bc400: Connection restored to service lustre-MDT0000 using nid 192.168.4.128@o2ib.
May 17 19:07:20 client-18 kernel: Lustre: Skipped 8 previous similar messages
May 17 19:07:55 client-18 xinetd[3390]: START: shell pid=32433 from=192.168.4.5
May 17 19:07:55 client-18 rshd[32434]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 19:07:55 client-18 xinetd[3390]: EXIT: shell status=0 pid=32433 duration=0(sec)
May 17 19:07:55 client-18 xinetd[3390]: START: shell pid=32469 from=192.168.4.5
May 17 19:07:55 client-18 rshd[32470]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 19:07:55 client-18 xinetd[3390]: EXIT: shell status=0 pid=32469 duration=0(sec)
May 17 19:07:55 client-18 xinetd[3390]: START: shell pid=32506 from=192.168.4.5
May 17 19:07:55 client-18 rshd[32507]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-vbr test 7c: create, {lost}, rename ======================================================== 19:07:55 \(1305684475\)");echo XXRETCODE:$?'
May 17 19:07:55 client-18 kernel: Lustre: DEBUG MARKER: == replay-vbr test 7c: create, {lost}, rename ======================================================== 19:07:55 (1305684475)
May 17 19:07:55 client-18 xinetd[3390]: EXIT: shell status=0 pid=32506 duration=0(sec)
May 17 19:07:55 client-18 xinetd[3390]: START: shell pid=32541 from=192.168.4.5
May 17 19:07:55 client-18 rshd[32542]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mkdir -p /mnt/lustre2");echo XXRETCODE:$?'
May 17 19:07:56 client-18 xinetd[3390]: EXIT: shell status=0 pid=32541 duration=1(sec)
May 17 19:07:56 client-18 xinetd[3390]: START: shell pid=32576 from=192.168.4.5
May 17 19:07:56 client-18 rshd[32577]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre2");echo XXRETCODE:$?'
May 17 19:07:56 client-18 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'.
May 17 19:07:56 client-18 kernel: LustreError: Skipped 4 previous similar messages
May 17 19:07:56 client-18 kernel: Lustre: 32611:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff8102edee1c00->192.168.4.128@o2ib netid 50000: select flavor null
May 17 19:07:56 client-18 kernel: Lustre: 32611:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 34 previous similar messages
May 17 19:07:56 client-18 kernel: Lustre: Client lustre-client has started
May 17 19:07:56 client-18 kernel: Lustre: Skipped 4 previous similar messages
May 17 19:07:56 client-18 xinetd[3390]: START: shell pid=32614 from=192.168.4.5
May 17 19:07:56 client-18 xinetd[3390]: EXIT: shell status=0 pid=32576 duration=0(sec)
May 17 19:07:56 client-18 rshd[32615]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param debug=\"0x33f0406\";         lctl set_param subsystem_debug=\"0xffb7e3ff\";         lctl set_param debug_mb=32");echo XXRETCODE:$?'
May 17 19:07:56 client-18 xinetd[3390]: EXIT: shell status=0 pid=32614 duration=0(sec)
May 17 19:07:56 client-18 xinetd[3390]: START: shell pid=32651 from=192.168.4.5
May 17 19:07:56 client-18 rshd[32652]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 19:07:56 client-18 xinetd[3390]: EXIT: shell status=0 pid=32651 duration=0(sec)
May 17 19:07:57 client-18 xinetd[3390]: START: shell pid=32690 from=192.168.4.5
May 17 19:07:57 client-18 rshd[32691]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "rm /mnt/lustre2/d0.replay-vbr/d7/f7c-0; createmany -o /mnt/lustre2/d0.replay-vbr/d7/f7c- 1");echo XXRETCODE:$?'
May 17 19:07:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=32690 duration=0(sec)
May 17 19:07:57 client-18 xinetd[3390]: START: shell pid=32727 from=192.168.4.5
May 17 19:07:57 client-18 rshd[32728]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "grep -c /mnt/lustre2' ' /proc/mounts");echo XXRETCODE:$?'
May 17 19:07:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=32727 duration=0(sec)
May 17 19:07:57 client-18 xinetd[3390]: START: shell pid=32762 from=192.168.4.5
May 17 19:07:57 client-18 rshd[32763]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lsof -t /mnt/lustre2");echo XXRETCODE:$?'
May 17 19:07:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=32762 duration=0(sec)
May 17 19:07:57 client-18 xinetd[3390]: START: shell pid=332 from=192.168.4.5
May 17 19:07:57 client-18 rshd[333]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "umount  /mnt/lustre2 2>&1");echo XXRETCODE:$?'
May 17 19:07:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=332 duration=0(sec)
May 17 19:08:16 client-18 kernel: Lustre: 8885:0:(import.c:885:ptlrpc_connect_interpret()) MGS@MGC192.168.4.128@o2ib_0 changed server handle from 0xfff98862bf655554 to 0xfff98862bf655a94
May 17 19:10:22 client-18 xinetd[3390]: START: shell pid=380 from=192.168.4.5
May 17 19:10:22 client-18 rshd[381]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mkdir -p /mnt/lustre2");echo XXRETCODE:$?'
May 17 19:10:22 client-18 xinetd[3390]: EXIT: shell status=0 pid=380 duration=0(sec)
May 17 19:10:22 client-18 xinetd[3390]: START: shell pid=423 from=192.168.4.5
May 17 19:10:22 client-18 rshd[424]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre2");echo XXRETCODE:$?'
May 17 19:10:22 client-18 xinetd[3390]: EXIT: shell status=0 pid=423 duration=0(sec)
May 17 19:10:22 client-18 xinetd[3390]: START: shell pid=461 from=192.168.4.5
May 17 19:10:22 client-18 rshd[462]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param debug=\"0x33f0406\";         lctl set_param subsystem_debug=\"0xffb7e3ff\";         lctl set_param debug_mb=32");echo XXRETCODE:$?'
May 17 19:10:22 client-18 xinetd[3390]: EXIT: shell status=0 pid=461 duration=0(sec)
May 17 19:10:23 client-18 xinetd[3390]: START: shell pid=498 from=192.168.4.5
May 17 19:10:23 client-18 rshd[499]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 19:10:23 client-18 xinetd[3390]: EXIT: shell status=0 pid=498 duration=0(sec)
May 17 19:10:23 client-18 xinetd[3390]: START: shell pid=537 from=192.168.4.5
May 17 19:10:23 client-18 rshd[538]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "rm /mnt/lustre2/d0.replay-vbr/d7/f7c-0; mkdir /mnt/lustre2/d0.replay-vbr/d7/f7c-0");echo XXRETCODE:$?'
May 17 19:10:23 client-18 xinetd[3390]: EXIT: shell status=0 pid=537 duration=0(sec)
May 17 19:10:23 client-18 xinetd[3390]: START: shell pid=575 from=192.168.4.5
May 17 19:10:23 client-18 rshd[576]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "grep -c /mnt/lustre2' ' /proc/mounts");echo XXRETCODE:$?'
May 17 19:10:24 client-18 xinetd[3390]: EXIT: shell status=0 pid=575 duration=1(sec)
May 17 19:10:24 client-18 xinetd[3390]: START: shell pid=621 from=192.168.4.5
May 17 19:10:24 client-18 rshd[622]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lsof -t /mnt/lustre2");echo XXRETCODE:$?'
May 17 19:10:24 client-18 xinetd[3390]: EXIT: shell status=0 pid=621 duration=0(sec)
May 17 19:10:24 client-18 xinetd[3390]: START: shell pid=659 from=192.168.4.5
May 17 19:10:24 client-18 rshd[660]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "umount  /mnt/lustre2 2>&1");echo XXRETCODE:$?'
May 17 19:10:24 client-18 xinetd[3390]: EXIT: shell status=0 pid=659 duration=0(sec)
May 17 19:10:42 client-18 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail.
May 17 19:10:42 client-18 kernel: LustreError: Skipped 4 previous similar messages
May 17 19:10:42 client-18 kernel: Lustre: lustre-MDT0000-mdc-ffff8103028bc400: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete.
May 17 19:10:42 client-18 kernel: Lustre: 8885:0:(import.c:885:ptlrpc_connect_interpret()) MGS@MGC192.168.4.128@o2ib_0 changed server handle from 0xfff98862bf655a94 to 0xfff98862bf655fdb
May 17 19:10:42 client-18 kernel: Lustre: Skipped 4 previous similar messages
May 17 19:11:05 client-18 kernel: Lustre: 8884:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369101282850820 sent from lustre-MDT0000-mdc-ffff8103028bc400 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305684632] [real_sent 1305684632] [current 1305684665] [deadline 33s] [delay 0s]  req@ffff8102ed9d0c00 x1369101282850820/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 192/192 e 0 to 1 dl 1305684665 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
May 17 19:11:05 client-18 kernel: Lustre: 8884:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 13 previous similar messages
May 17 19:12:49 client-18 xinetd[3390]: START: shell pid=712 from=192.168.4.5
May 17 19:12:49 client-18 rshd[713]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mkdir -p /mnt/lustre2");echo XXRETCODE:$?'
May 17 19:12:49 client-18 xinetd[3390]: START: shell pid=747 from=192.168.4.5
May 17 19:12:49 client-18 xinetd[3390]: EXIT: shell status=0 pid=712 duration=0(sec)
May 17 19:12:49 client-18 rshd[748]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre2");echo XXRETCODE:$?'
May 17 19:12:49 client-18 xinetd[3390]: START: shell pid=785 from=192.168.4.5
May 17 19:12:49 client-18 xinetd[3390]: EXIT: shell status=0 pid=747 duration=0(sec)
May 17 19:12:49 client-18 rshd[786]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param debug=\"0x33f0406\";         lctl set_param subsystem_debug=\"0xffb7e3ff\";         lctl set_param debug_mb=32");echo XXRETCODE:$?'
May 17 19:12:49 client-18 xinetd[3390]: EXIT: shell status=0 pid=785 duration=0(sec)
May 17 19:12:50 client-18 xinetd[3390]: START: shell pid=822 from=192.168.4.5
May 17 19:12:50 client-18 rshd[823]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 19:12:50 client-18 xinetd[3390]: EXIT: shell status=0 pid=822 duration=0(sec)
May 17 19:12:50 client-18 xinetd[3390]: START: shell pid=861 from=192.168.4.5
May 17 19:12:50 client-18 rshd[862]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "rmdir /mnt/lustre2/d0.replay-vbr/d7/f7c-1-0; createmany -o /mnt/lustre2/d0.replay-vbr/d7/f7c-1- 1");echo XXRETCODE:$?'
May 17 19:12:50 client-18 xinetd[3390]: EXIT: shell status=0 pid=861 duration=0(sec)
May 17 19:12:50 client-18 xinetd[3390]: START: shell pid=898 from=192.168.4.5
May 17 19:12:50 client-18 rshd[899]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "grep -c /mnt/lustre2' ' /proc/mounts");echo XXRETCODE:$?'
May 17 19:12:50 client-18 xinetd[3390]: EXIT: shell status=0 pid=898 duration=0(sec)
May 17 19:12:50 client-18 xinetd[3390]: START: shell pid=933 from=192.168.4.5
May 17 19:12:50 client-18 rshd[934]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lsof -t /mnt/lustre2");echo XXRETCODE:$?'
May 17 19:12:50 client-18 xinetd[3390]: EXIT: shell status=0 pid=933 duration=0(sec)
May 17 19:12:50 client-18 xinetd[3390]: START: shell pid=969 from=192.168.4.5
May 17 19:12:50 client-18 rshd[970]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "umount  /mnt/lustre2 2>&1");echo XXRETCODE:$?'
May 17 19:12:51 client-18 xinetd[3390]: EXIT: shell status=0 pid=969 duration=1(sec)
May 17 19:13:09 client-18 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The obd_ping operation failed with -107
May 17 19:13:09 client-18 kernel: LustreError: Skipped 7 previous similar messages
May 17 19:13:09 client-18 kernel: Lustre: 8885:0:(import.c:885:ptlrpc_connect_interpret()) MGS@MGC192.168.4.128@o2ib_0 changed server handle from 0xfff98862bf655fdb to 0xfff98862bf65654c
May 17 19:15:15 client-18 xinetd[3390]: START: shell pid=1017 from=192.168.4.5
May 17 19:15:16 client-18 rshd[1018]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mkdir -p /mnt/lustre2");echo XXRETCODE:$?'
May 17 19:15:16 client-18 xinetd[3390]: START: shell pid=1052 from=192.168.4.5
May 17 19:15:16 client-18 xinetd[3390]: EXIT: shell status=0 pid=1017 duration=1(sec)
May 17 19:15:16 client-18 rshd[1053]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre2");echo XXRETCODE:$?'
May 17 19:15:16 client-18 xinetd[3390]: EXIT: shell status=0 pid=1052 duration=0(sec)
May 17 19:15:16 client-18 xinetd[3390]: START: shell pid=1090 from=192.168.4.5
May 17 19:15:16 client-18 rshd[1091]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param debug=\"0x33f0406\";         lctl set_param subsystem_debug=\"0xffb7e3ff\";         lctl set_param debug_mb=32");echo XXRETCODE:$?'
May 17 19:15:16 client-18 xinetd[3390]: EXIT: shell status=0 pid=1090 duration=0(sec)
May 17 19:15:16 client-18 xinetd[3390]: START: shell pid=1127 from=192.168.4.5
May 17 19:15:16 client-18 rshd[1128]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 19:15:16 client-18 xinetd[3390]: EXIT: shell status=0 pid=1127 duration=0(sec)
May 17 19:15:17 client-18 xinetd[3390]: START: shell pid=1166 from=192.168.4.5
May 17 19:15:17 client-18 rshd[1167]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mv /mnt/lustre2/d0.replay-vbr/d7/f7c-0 /mnt/lustre2/d0.replay-vbr/d7/f7c");echo XXRETCODE:$?'
May 17 19:15:17 client-18 xinetd[3390]: EXIT: shell status=0 pid=1166 duration=0(sec)
May 17 19:15:17 client-18 xinetd[3390]: START: shell pid=1201 from=192.168.4.5
May 17 19:15:17 client-18 rshd[1202]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "grep -c /mnt/lustre2' ' /proc/mounts");echo XXRETCODE:$?'
May 17 19:15:17 client-18 xinetd[3390]: EXIT: shell status=0 pid=1201 duration=0(sec)
May 17 19:15:17 client-18 xinetd[3390]: START: shell pid=1236 from=192.168.4.5
May 17 19:15:17 client-18 rshd[1237]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lsof -t /mnt/lustre2");echo XXRETCODE:$?'
May 17 19:15:17 client-18 xinetd[3390]: EXIT: shell status=0 pid=1236 duration=0(sec)
May 17 19:15:17 client-18 xinetd[3390]: START: shell pid=1272 from=192.168.4.5
May 17 19:15:17 client-18 rshd[1273]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "umount  /mnt/lustre2 2>&1");echo XXRETCODE:$?'
May 17 19:15:17 client-18 kernel: LustreError: 1307:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway
May 17 19:15:17 client-18 kernel: LustreError: 1307:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Skipped 14 previous similar messages
May 17 19:15:17 client-18 kernel: LustreError: 1307:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108
May 17 19:15:17 client-18 kernel: LustreError: 1307:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) Skipped 14 previous similar messages
May 17 19:15:17 client-18 kernel: Lustre: client ffff8102fe352000 umount complete
May 17 19:15:17 client-18 kernel: Lustre: Skipped 4 previous similar messages
May 17 19:15:17 client-18 xinetd[3390]: EXIT: shell status=0 pid=1272 duration=0(sec)
May 17 19:15:38 client-18 kernel: Lustre: 8886:0:(import.c:529:import_select_connection()) MGC192.168.4.128@o2ib: tried all connections, increasing latency to 6s
May 17 19:15:38 client-18 kernel: Lustre: 8886:0:(import.c:529:import_select_connection()) Skipped 1 previous similar message
May 17 19:15:38 client-18 kernel: Lustre: 8885:0:(import.c:885:ptlrpc_connect_interpret()) MGS@MGC192.168.4.128@o2ib_0 changed server handle from 0xfff98862bf65654c to 0xfff98862bf656b57
May 17 19:15:38 client-18 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import
May 17 19:15:38 client-18 kernel: Lustre: Skipped 4 previous similar messages
May 17 19:17:42 client-18 xinetd[3390]: START: shell pid=1320 from=192.168.4.5
May 17 19:17:43 client-18 rshd[1321]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 19:17:43 client-18 xinetd[3390]: EXIT: shell status=0 pid=1320 duration=1(sec)
May 17 19:17:43 client-18 xinetd[3390]: START: shell pid=1356 from=192.168.4.5
May 17 19:17:43 client-18 rshd[1357]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 19:17:43 client-18 xinetd[3390]: EXIT: shell status=0 pid=1356 duration=0(sec)
May 17 19:17:43 client-18 xinetd[3390]: START: shell pid=1393 from=192.168.4.5
May 17 19:17:43 client-18 rshd[1394]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-vbr test 7d: unlink, {lost}, create ======================================================== 19:17:43 \(1305685063\)");echo XXRETCODE:$?'
May 17 19:17:43 client-18 kernel: Lustre: DEBUG MARKER: == replay-vbr test 7d: unlink, {lost}, create ======================================================== 19:17:43 (1305685063)
May 17 19:17:43 client-18 xinetd[3390]: EXIT: shell status=0 pid=1393 duration=0(sec)
May 17 19:17:43 client-18 xinetd[3390]: START: shell pid=1428 from=192.168.4.5
May 17 19:17:43 client-18 rshd[1429]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mkdir -p /mnt/lustre2");echo XXRETCODE:$?'
May 17 19:17:43 client-18 xinetd[3390]: EXIT: shell status=0 pid=1428 duration=0(sec)
May 17 19:17:43 client-18 xinetd[3390]: START: shell pid=1464 from=192.168.4.5
May 17 19:17:43 client-18 rshd[1465]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre2");echo XXRETCODE:$?'
May 17 19:17:43 client-18 xinetd[3390]: START: shell pid=1502 from=192.168.4.5
May 17 19:17:43 client-18 xinetd[3390]: EXIT: shell status=0 pid=1464 duration=0(sec)
May 17 19:17:43 client-18 rshd[1503]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param debug=\"0x33f0406\";         lctl set_param subsystem_debug=\"0xffb7e3ff\";         lctl set_param debug_mb=32");echo XXRETCODE:$?'
May 17 19:17:43 client-18 kernel: Lustre: 1537:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release.
May 17 19:17:43 client-18 kernel: Lustre: 1537:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 9 previous similar messages
May 17 19:17:43 client-18 xinetd[3390]: EXIT: shell status=0 pid=1502 duration=0(sec)
May 17 19:17:44 client-18 xinetd[3390]: START: shell pid=1539 from=192.168.4.5
May 17 19:17:44 client-18 rshd[1540]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 19:17:44 client-18 xinetd[3390]: EXIT: shell status=0 pid=1539 duration=0(sec)
May 17 19:17:44 client-18 xinetd[3390]: START: shell pid=1578 from=192.168.4.5
May 17 19:17:44 client-18 rshd[1579]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "createmany -o /mnt/lustre2/d0.replay-vbr/d7/f7d- 1; rm /mnt/lustre2/d0.replay-vbr/d7/f7d-0");echo XXRETCODE:$?'
May 17 19:17:44 client-18 xinetd[3390]: EXIT: shell status=0 pid=1578 duration=0(sec)
May 17 19:17:44 client-18 xinetd[3390]: START: shell pid=1615 from=192.168.4.5
May 17 19:17:44 client-18 rshd[1616]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "grep -c /mnt/lustre2' ' /proc/mounts");echo XXRETCODE:$?'
May 17 19:17:44 client-18 xinetd[3390]: EXIT: shell status=0 pid=1615 duration=0(sec)
May 17 19:17:44 client-18 xinetd[3390]: START: shell pid=1650 from=192.168.4.5
May 17 19:17:44 client-18 rshd[1652]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lsof -t /mnt/lustre2");echo XXRETCODE:$?'
May 17 19:17:44 client-18 xinetd[3390]: EXIT: shell status=0 pid=1650 duration=0(sec)
May 17 19:17:44 client-18 xinetd[3390]: START: shell pid=1689 from=192.168.4.5
May 17 19:17:44 client-18 rshd[1690]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "umount  /mnt/lustre2 2>&1");echo XXRETCODE:$?'
May 17 19:17:45 client-18 xinetd[3390]: EXIT: shell status=0 pid=1689 duration=1(sec)
May 17 19:18:15 client-18 kernel: Lustre: 8885:0:(import.c:885:ptlrpc_connect_interpret()) MGS@MGC192.168.4.128@o2ib_0 changed server handle from 0xfff98862bf656b57 to 0xfff98862bf6570d6
May 17 19:18:15 client-18 kernel: Lustre: MGC192.168.4.128@o2ib: Connection restored to service MGS using nid 192.168.4.128@o2ib.
May 17 19:18:15 client-18 kernel: Lustre: Skipped 8 previous similar messages
May 17 19:19:37 client-18 xinetd[3390]: START: shell pid=1733 from=192.168.4.5
May 17 19:19:37 client-18 rshd[1734]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mkdir -p /mnt/lustre2");echo XXRETCODE:$?'
May 17 19:19:37 client-18 xinetd[3390]: EXIT: shell status=0 pid=1733 duration=0(sec)
May 17 19:19:37 client-18 xinetd[3390]: START: shell pid=1768 from=192.168.4.5
May 17 19:19:37 client-18 rshd[1769]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre2");echo XXRETCODE:$?'
May 17 19:19:37 client-18 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'.
May 17 19:19:37 client-18 kernel: LustreError: Skipped 4 previous similar messages
May 17 19:19:37 client-18 kernel: Lustre: 1803:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff810325357400->192.168.4.128@o2ib netid 50000: select flavor null
May 17 19:19:37 client-18 kernel: Lustre: 1803:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 34 previous similar messages
May 17 19:19:37 client-18 kernel: Lustre: Client lustre-client has started
May 17 19:19:37 client-18 kernel: Lustre: Skipped 4 previous similar messages
May 17 19:19:37 client-18 xinetd[3390]: EXIT: shell status=0 pid=1768 duration=0(sec)
May 17 19:19:37 client-18 xinetd[3390]: START: shell pid=1806 from=192.168.4.5
May 17 19:19:37 client-18 rshd[1807]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param debug=\"0x33f0406\";         lctl set_param subsystem_debug=\"0xffb7e3ff\";         lctl set_param debug_mb=32");echo XXRETCODE:$?'
May 17 19:19:37 client-18 xinetd[3390]: EXIT: shell status=0 pid=1806 duration=0(sec)
May 17 19:19:38 client-18 xinetd[3390]: START: shell pid=1843 from=192.168.4.5
May 17 19:19:38 client-18 rshd[1844]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 19:19:38 client-18 xinetd[3390]: EXIT: shell status=0 pid=1843 duration=0(sec)
May 17 19:19:38 client-18 xinetd[3390]: START: shell pid=1882 from=192.168.4.5
May 17 19:19:38 client-18 rshd[1883]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mkdir /mnt/lustre2/d0.replay-vbr/d7/f7d-0; rmdir /mnt/lustre2/d0.replay-vbr/d7/f7d-0");echo XXRETCODE:$?'
May 17 19:19:38 client-18 xinetd[3390]: EXIT: shell status=0 pid=1882 duration=0(sec)
May 17 19:19:38 client-18 xinetd[3390]: START: shell pid=1924 from=192.168.4.5
May 17 19:19:38 client-18 rshd[1925]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "grep -c /mnt/lustre2' ' /proc/mounts");echo XXRETCODE:$?'
May 17 19:19:38 client-18 xinetd[3390]: EXIT: shell status=0 pid=1924 duration=0(sec)
May 17 19:19:38 client-18 xinetd[3390]: START: shell pid=1959 from=192.168.4.5
May 17 19:19:39 client-18 rshd[1960]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lsof -t /mnt/lustre2");echo XXRETCODE:$?'
May 17 19:19:39 client-18 xinetd[3390]: EXIT: shell status=0 pid=1959 duration=1(sec)
May 17 19:19:39 client-18 xinetd[3390]: START: shell pid=1995 from=192.168.4.5
May 17 19:19:39 client-18 rshd[1996]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "umount  /mnt/lustre2 2>&1");echo XXRETCODE:$?'
May 17 19:19:39 client-18 xinetd[3390]: EXIT: shell status=0 pid=1995 duration=0(sec)
May 17 19:19:57 client-18 kernel: Lustre: 8885:0:(import.c:885:ptlrpc_connect_interpret()) MGS@MGC192.168.4.128@o2ib_0 changed server handle from 0xfff98862bf6570d6 to 0xfff98862bf657560
May 17 19:21:31 client-18 xinetd[3390]: START: shell pid=2044 from=192.168.4.5
May 17 19:21:31 client-18 rshd[2045]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mkdir -p /mnt/lustre2");echo XXRETCODE:$?'
May 17 19:21:31 client-18 xinetd[3390]: START: shell pid=2079 from=192.168.4.5
May 17 19:21:31 client-18 xinetd[3390]: EXIT: shell status=0 pid=2044 duration=0(sec)
May 17 19:21:31 client-18 rshd[2080]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre2");echo XXRETCODE:$?'
May 17 19:21:31 client-18 xinetd[3390]: START: shell pid=2118 from=192.168.4.5
May 17 19:21:31 client-18 xinetd[3390]: EXIT: shell status=0 pid=2079 duration=0(sec)
May 17 19:21:31 client-18 rshd[2119]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param debug=\"0x33f0406\";         lctl set_param subsystem_debug=\"0xffb7e3ff\";         lctl set_param debug_mb=32");echo XXRETCODE:$?'
May 17 19:21:31 client-18 xinetd[3390]: EXIT: shell status=0 pid=2118 duration=0(sec)
May 17 19:21:31 client-18 xinetd[3390]: START: shell pid=2156 from=192.168.4.5
May 17 19:21:31 client-18 rshd[2157]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 19:21:32 client-18 xinetd[3390]: EXIT: shell status=0 pid=2156 duration=1(sec)
May 17 19:21:32 client-18 xinetd[3390]: START: shell pid=2199 from=192.168.4.5
May 17 19:21:32 client-18 rshd[2200]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "createmany -o /mnt/lustre2/d0.replay-vbr/d7/f7d- 1; mv /mnt/lustre2/d0.replay-vbr/d7/f7d-0 /mnt/lustre2/d0.replay-vbr/d7/f7d-1");echo XXRETCODE:$?'
May 17 19:21:32 client-18 xinetd[3390]: EXIT: shell status=0 pid=2199 duration=0(sec)
May 17 19:21:32 client-18 xinetd[3390]: START: shell pid=2236 from=192.168.4.5
May 17 19:21:32 client-18 rshd[2237]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "grep -c /mnt/lustre2' ' /proc/mounts");echo XXRETCODE:$?'
May 17 19:21:32 client-18 xinetd[3390]: EXIT: shell status=0 pid=2236 duration=0(sec)
May 17 19:21:32 client-18 xinetd[3390]: START: shell pid=2279 from=192.168.4.5
May 17 19:21:32 client-18 rshd[2280]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lsof -t /mnt/lustre2");echo XXRETCODE:$?'
May 17 19:21:32 client-18 xinetd[3390]: EXIT: shell status=0 pid=2279 duration=0(sec)
May 17 19:21:32 client-18 xinetd[3390]: START: shell pid=2315 from=192.168.4.5
May 17 19:21:32 client-18 rshd[2316]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "umount  /mnt/lustre2 2>&1");echo XXRETCODE:$?'
May 17 19:21:33 client-18 xinetd[3390]: EXIT: shell status=0 pid=2315 duration=1(sec)
May 17 19:21:51 client-18 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail.
May 17 19:21:51 client-18 kernel: LustreError: Skipped 4 previous similar messages
May 17 19:21:51 client-18 kernel: Lustre: lustre-MDT0000-mdc-ffff8103028bc400: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete.
May 17 19:21:51 client-18 kernel: Lustre: 8885:0:(import.c:885:ptlrpc_connect_interpret()) MGS@MGC192.168.4.128@o2ib_0 changed server handle from 0xfff98862bf657560 to 0xfff98862bf657ad8
May 17 19:21:51 client-18 kernel: Lustre: Skipped 4 previous similar messages
May 17 19:22:14 client-18 kernel: Lustre: 8884:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369101282852018 sent from lustre-MDT0000-mdc-ffff8103028bc400 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305685301] [real_sent 1305685301] [current 1305685334] [deadline 33s] [delay 0s]  req@ffff8102f5e21400 x1369101282852018/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 192/192 e 0 to 1 dl 1305685334 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
May 17 19:22:14 client-18 kernel: Lustre: 8884:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 13 previous similar messages
May 17 19:23:25 client-18 xinetd[3390]: START: shell pid=2359 from=192.168.4.5
May 17 19:23:25 client-18 rshd[2360]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 19:23:25 client-18 xinetd[3390]: EXIT: shell status=0 pid=2359 duration=0(sec)
May 17 19:23:25 client-18 xinetd[3390]: START: shell pid=2398 from=192.168.4.5
May 17 19:23:25 client-18 rshd[2399]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 19:23:25 client-18 xinetd[3390]: EXIT: shell status=0 pid=2398 duration=0(sec)
May 17 19:23:25 client-18 xinetd[3390]: START: shell pid=2443 from=192.168.4.5
May 17 19:23:25 client-18 rshd[2444]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-vbr test 7e: unlink, {lost}, unlink ======================================================== 19:23:25 \(1305685405\)");echo XXRETCODE:$?'
May 17 19:23:25 client-18 kernel: Lustre: DEBUG MARKER: == replay-vbr test 7e: unlink, {lost}, unlink ======================================================== 19:23:25 (1305685405)
May 17 19:23:25 client-18 xinetd[3390]: EXIT: shell status=0 pid=2443 duration=0(sec)
May 17 19:23:25 client-18 xinetd[3390]: START: shell pid=2479 from=192.168.4.5
May 17 19:23:25 client-18 rshd[2480]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mkdir -p /mnt/lustre2");echo XXRETCODE:$?'
May 17 19:23:25 client-18 xinetd[3390]: START: shell pid=2514 from=192.168.4.5
May 17 19:23:25 client-18 xinetd[3390]: EXIT: shell status=0 pid=2479 duration=0(sec)
May 17 19:23:25 client-18 rshd[2515]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre2");echo XXRETCODE:$?'
May 17 19:23:25 client-18 xinetd[3390]: EXIT: shell status=0 pid=2514 duration=0(sec)
May 17 19:23:25 client-18 xinetd[3390]: START: shell pid=2557 from=192.168.4.5
May 17 19:23:25 client-18 rshd[2558]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param debug=\"0x33f0406\";         lctl set_param subsystem_debug=\"0xffb7e3ff\";         lctl set_param debug_mb=32");echo XXRETCODE:$?'
May 17 19:23:25 client-18 xinetd[3390]: EXIT: shell status=0 pid=2557 duration=0(sec)
May 17 19:23:26 client-18 xinetd[3390]: START: shell pid=2594 from=192.168.4.5
May 17 19:23:26 client-18 rshd[2595]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 19:23:26 client-18 xinetd[3390]: EXIT: shell status=0 pid=2594 duration=0(sec)
May 17 19:23:26 client-18 xinetd[3390]: START: shell pid=2633 from=192.168.4.5
May 17 19:23:26 client-18 rshd[2634]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "createmany -o /mnt/lustre2/d0.replay-vbr/d7/f7e- 1; rm /mnt/lustre2/d0.replay-vbr/d7/f7e-0;createmany -o /mnt/lustre2/d0.replay-vbr/d7/f7e- 1");echo XXRETCODE:$?'
May 17 19:23:26 client-18 xinetd[3390]: EXIT: shell status=0 pid=2633 duration=0(sec)
May 17 19:23:26 client-18 xinetd[3390]: START: shell pid=2671 from=192.168.4.5
May 17 19:23:26 client-18 rshd[2672]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "grep -c /mnt/lustre2' ' /proc/mounts");echo XXRETCODE:$?'
May 17 19:23:26 client-18 xinetd[3390]: EXIT: shell status=0 pid=2671 duration=0(sec)
May 17 19:23:26 client-18 xinetd[3390]: START: shell pid=2706 from=192.168.4.5
May 17 19:23:26 client-18 rshd[2707]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lsof -t /mnt/lustre2");echo XXRETCODE:$?'
May 17 19:23:26 client-18 xinetd[3390]: EXIT: shell status=0 pid=2706 duration=0(sec)
May 17 19:23:26 client-18 xinetd[3390]: START: shell pid=2742 from=192.168.4.5
May 17 19:23:26 client-18 rshd[2743]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "umount  /mnt/lustre2 2>&1");echo XXRETCODE:$?'
May 17 19:23:27 client-18 xinetd[3390]: EXIT: shell status=0 pid=2742 duration=1(sec)
May 17 19:23:45 client-18 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The obd_ping operation failed with -107
May 17 19:23:45 client-18 kernel: LustreError: Skipped 7 previous similar messages
May 17 19:23:45 client-18 kernel: Lustre: 8885:0:(import.c:885:ptlrpc_connect_interpret()) MGS@MGC192.168.4.128@o2ib_0 changed server handle from 0xfff98862bf657ad8 to 0xfff98862bf65800a
May 17 19:25:52 client-18 xinetd[3390]: START: shell pid=2790 from=192.168.4.5
May 17 19:25:52 client-18 rshd[2791]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mkdir -p /mnt/lustre2");echo XXRETCODE:$?'
May 17 19:25:52 client-18 xinetd[3390]: START: shell pid=2825 from=192.168.4.5
May 17 19:25:52 client-18 xinetd[3390]: EXIT: shell status=0 pid=2790 duration=0(sec)
May 17 19:25:52 client-18 rshd[2826]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre2");echo XXRETCODE:$?'
May 17 19:25:52 client-18 xinetd[3390]: EXIT: shell status=0 pid=2825 duration=0(sec)
May 17 19:25:52 client-18 xinetd[3390]: START: shell pid=2863 from=192.168.4.5
May 17 19:25:52 client-18 rshd[2864]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param debug=\"0x33f0406\";         lctl set_param subsystem_debug=\"0xffb7e3ff\";         lctl set_param debug_mb=32");echo XXRETCODE:$?'
May 17 19:25:52 client-18 xinetd[3390]: EXIT: shell status=0 pid=2863 duration=0(sec)
May 17 19:25:52 client-18 xinetd[3390]: START: shell pid=2900 from=192.168.4.5
May 17 19:25:52 client-18 rshd[2901]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 19:25:52 client-18 xinetd[3390]: EXIT: shell status=0 pid=2900 duration=0(sec)
May 17 19:25:53 client-18 xinetd[3390]: START: shell pid=2939 from=192.168.4.5
May 17 19:25:53 client-18 rshd[2940]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mkdir /mnt/lustre2/d0.replay-vbr/d7/f7e-0; rmdir /mnt/lustre2/d0.replay-vbr/d7/f7e-0; mkdir /mnt/lustre2/d0.replay-vbr/d7/f7e-0");echo XXRETCODE:$?'
May 17 19:25:53 client-18 xinetd[3390]: EXIT: shell status=0 pid=2939 duration=0(sec)
May 17 19:25:53 client-18 xinetd[3390]: START: shell pid=2977 from=192.168.4.5
May 17 19:25:53 client-18 rshd[2978]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "grep -c /mnt/lustre2' ' /proc/mounts");echo XXRETCODE:$?'
May 17 19:25:53 client-18 xinetd[3390]: EXIT: shell status=0 pid=2977 duration=0(sec)
May 17 19:25:53 client-18 xinetd[3390]: START: shell pid=3016 from=192.168.4.5
May 17 19:25:53 client-18 rshd[3018]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lsof -t /mnt/lustre2");echo XXRETCODE:$?'
May 17 19:25:53 client-18 xinetd[3390]: EXIT: shell status=0 pid=3016 duration=0(sec)
May 17 19:25:53 client-18 xinetd[3390]: START: shell pid=3054 from=192.168.4.5
May 17 19:25:53 client-18 rshd[3055]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "umount  /mnt/lustre2 2>&1");echo XXRETCODE:$?'
May 17 19:25:54 client-18 kernel: LustreError: 3094:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway
May 17 19:25:54 client-18 kernel: LustreError: 3094:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Skipped 14 previous similar messages
May 17 19:25:54 client-18 kernel: LustreError: 3094:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108
May 17 19:25:54 client-18 kernel: LustreError: 3094:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) Skipped 14 previous similar messages
May 17 19:25:54 client-18 kernel: Lustre: client ffff8102f87be400 umount complete
May 17 19:25:54 client-18 kernel: Lustre: Skipped 4 previous similar messages
May 17 19:25:54 client-18 xinetd[3390]: EXIT: shell status=0 pid=3054 duration=1(sec)
May 17 19:26:12 client-18 kernel: Lustre: 8885:0:(import.c:885:ptlrpc_connect_interpret()) MGS@MGC192.168.4.128@o2ib_0 changed server handle from 0xfff98862bf65800a to 0xfff98862bf6584ef
May 17 19:26:12 client-18 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import
May 17 19:26:12 client-18 kernel: Lustre: Skipped 4 previous similar messages
May 17 19:28:18 client-18 xinetd[3390]: START: shell pid=3108 from=192.168.4.5
May 17 19:28:18 client-18 rshd[3109]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mkdir -p /mnt/lustre2");echo XXRETCODE:$?'
May 17 19:28:18 client-18 xinetd[3390]: START: shell pid=3148 from=192.168.4.5
May 17 19:28:18 client-18 xinetd[3390]: EXIT: shell status=0 pid=3108 duration=0(sec)
May 17 19:28:18 client-18 rshd[3149]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre2");echo XXRETCODE:$?'
May 17 19:28:19 client-18 xinetd[3390]: EXIT: shell status=0 pid=3148 duration=1(sec)
May 17 19:28:19 client-18 xinetd[3390]: START: shell pid=3189 from=192.168.4.5
May 17 19:28:19 client-18 rshd[3191]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param debug=\"0x33f0406\";         lctl set_param subsystem_debug=\"0xffb7e3ff\";         lctl set_param debug_mb=32");echo XXRETCODE:$?'
May 17 19:28:19 client-18 kernel: Lustre: 3226:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release.
May 17 19:28:19 client-18 kernel: Lustre: 3226:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 9 previous similar messages
May 17 19:28:19 client-18 xinetd[3390]: EXIT: shell status=0 pid=3189 duration=0(sec)
May 17 19:28:19 client-18 xinetd[3390]: START: shell pid=3228 from=192.168.4.5
May 17 19:28:19 client-18 rshd[3229]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 19:28:19 client-18 xinetd[3390]: EXIT: shell status=0 pid=3228 duration=0(sec)
May 17 19:28:19 client-18 xinetd[3390]: START: shell pid=3269 from=192.168.4.5
May 17 19:28:19 client-18 rshd[3270]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mkdir /mnt/lustre2/d0.replay-vbr/d7/f7e-0");echo XXRETCODE:$?'
May 17 19:28:20 client-18 xinetd[3390]: EXIT: shell status=0 pid=3269 duration=1(sec)
May 17 19:28:20 client-18 xinetd[3390]: START: shell pid=3309 from=192.168.4.5
May 17 19:28:20 client-18 rshd[3310]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "grep -c /mnt/lustre2' ' /proc/mounts");echo XXRETCODE:$?'
May 17 19:28:20 client-18 xinetd[3390]: EXIT: shell status=0 pid=3309 duration=0(sec)
May 17 19:28:20 client-18 xinetd[3390]: START: shell pid=3349 from=192.168.4.5
May 17 19:28:20 client-18 rshd[3350]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lsof -t /mnt/lustre2");echo XXRETCODE:$?'
May 17 19:28:20 client-18 xinetd[3390]: EXIT: shell status=0 pid=3349 duration=0(sec)
May 17 19:28:20 client-18 xinetd[3390]: START: shell pid=3394 from=192.168.4.5
May 17 19:28:20 client-18 rshd[3395]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "umount  /mnt/lustre2 2>&1");echo XXRETCODE:$?'
May 17 19:28:20 client-18 xinetd[3390]: EXIT: shell status=0 pid=3394 duration=0(sec)
May 17 19:28:41 client-18 kernel: Lustre: 8886:0:(import.c:529:import_select_connection()) MGC192.168.4.128@o2ib: tried all connections, increasing latency to 6s
May 17 19:28:41 client-18 kernel: Lustre: 8886:0:(import.c:529:import_select_connection()) Skipped 1 previous similar message
May 17 19:28:41 client-18 kernel: Lustre: 8885:0:(import.c:885:ptlrpc_connect_interpret()) MGS@MGC192.168.4.128@o2ib_0 changed server handle from 0xfff98862bf6584ef to 0xfff98862bf6589b1
May 17 19:28:41 client-18 kernel: Lustre: MGC192.168.4.128@o2ib: Connection restored to service MGS using nid 192.168.4.128@o2ib.
May 17 19:28:41 client-18 kernel: Lustre: Skipped 9 previous similar messages
May 17 19:30:46 client-18 xinetd[3390]: START: shell pid=3446 from=192.168.4.5
May 17 19:30:46 client-18 rshd[3447]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mkdir -p /mnt/lustre2");echo XXRETCODE:$?'
May 17 19:30:46 client-18 xinetd[3390]: EXIT: shell status=0 pid=3446 duration=0(sec)
May 17 19:30:46 client-18 xinetd[3390]: START: shell pid=3484 from=192.168.4.5
May 17 19:30:46 client-18 rshd[3485]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre2");echo XXRETCODE:$?'
May 17 19:30:46 client-18 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'.
May 17 19:30:46 client-18 kernel: LustreError: Skipped 4 previous similar messages
May 17 19:30:46 client-18 kernel: Lustre: 3528:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff8102fb5b4800->192.168.4.128@o2ib netid 50000: select flavor null
May 17 19:30:46 client-18 kernel: Lustre: 3528:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 34 previous similar messages
May 17 19:30:46 client-18 kernel: Lustre: Client lustre-client has started
May 17 19:30:46 client-18 kernel: Lustre: Skipped 4 previous similar messages
May 17 19:30:46 client-18 xinetd[3390]: EXIT: shell status=0 pid=3484 duration=0(sec)
May 17 19:30:46 client-18 xinetd[3390]: START: shell pid=3531 from=192.168.4.5
May 17 19:30:46 client-18 rshd[3532]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param debug=\"0x33f0406\";         lctl set_param subsystem_debug=\"0xffb7e3ff\";         lctl set_param debug_mb=32");echo XXRETCODE:$?'
May 17 19:30:46 client-18 xinetd[3390]: EXIT: shell status=0 pid=3531 duration=0(sec)
May 17 19:30:46 client-18 xinetd[3390]: START: shell pid=3568 from=192.168.4.5
May 17 19:30:46 client-18 rshd[3569]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 19:30:46 client-18 xinetd[3390]: EXIT: shell status=0 pid=3568 duration=0(sec)
May 17 19:30:47 client-18 xinetd[3390]: START: shell pid=3610 from=192.168.4.5
May 17 19:30:47 client-18 rshd[3611]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "createmany -o /mnt/lustre2/d0.replay-vbr/d7/f7e- 1");echo XXRETCODE:$?'
May 17 19:30:47 client-18 xinetd[3390]: EXIT: shell status=0 pid=3610 duration=0(sec)
May 17 19:30:47 client-18 xinetd[3390]: START: shell pid=3645 from=192.168.4.5
May 17 19:30:47 client-18 rshd[3646]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "grep -c /mnt/lustre2' ' /proc/mounts");echo XXRETCODE:$?'
May 17 19:30:47 client-18 xinetd[3390]: EXIT: shell status=0 pid=3645 duration=0(sec)
May 17 19:30:47 client-18 xinetd[3390]: START: shell pid=3680 from=192.168.4.5
May 17 19:30:47 client-18 rshd[3681]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lsof -t /mnt/lustre2");echo XXRETCODE:$?'
May 17 19:30:47 client-18 xinetd[3390]: EXIT: shell status=0 pid=3680 duration=0(sec)
May 17 19:30:47 client-18 xinetd[3390]: START: shell pid=3716 from=192.168.4.5
May 17 19:30:47 client-18 rshd[3717]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "umount  /mnt/lustre2 2>&1");echo XXRETCODE:$?'
May 17 19:30:48 client-18 xinetd[3390]: EXIT: shell status=0 pid=3716 duration=1(sec)
May 17 19:31:18 client-18 kernel: Lustre: 8885:0:(import.c:885:ptlrpc_connect_interpret()) MGS@MGC192.168.4.128@o2ib_0 changed server handle from 0xfff98862bf6589b1 to 0xfff98862bf658ece
May 17 19:33:14 client-18 xinetd[3390]: START: shell pid=3764 from=192.168.4.5
May 17 19:33:14 client-18 rshd[3765]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mkdir -p /mnt/lustre2");echo XXRETCODE:$?'
May 17 19:33:14 client-18 xinetd[3390]: START: shell pid=3799 from=192.168.4.5
May 17 19:33:14 client-18 xinetd[3390]: EXIT: shell status=0 pid=3764 duration=0(sec)
May 17 19:33:14 client-18 rshd[3800]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre2");echo XXRETCODE:$?'
May 17 19:33:15 client-18 xinetd[3390]: EXIT: shell status=0 pid=3799 duration=1(sec)
May 17 19:33:15 client-18 xinetd[3390]: START: shell pid=3837 from=192.168.4.5
May 17 19:33:15 client-18 rshd[3838]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param debug=\"0x33f0406\";         lctl set_param subsystem_debug=\"0xffb7e3ff\";         lctl set_param debug_mb=32");echo XXRETCODE:$?'
May 17 19:33:15 client-18 xinetd[3390]: EXIT: shell status=0 pid=3837 duration=0(sec)
May 17 19:33:15 client-18 xinetd[3390]: START: shell pid=3874 from=192.168.4.5
May 17 19:33:15 client-18 rshd[3875]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 19:33:15 client-18 xinetd[3390]: EXIT: shell status=0 pid=3874 duration=0(sec)
May 17 19:33:15 client-18 xinetd[3390]: START: shell pid=3913 from=192.168.4.5
May 17 19:33:16 client-18 rshd[3914]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mv /mnt/lustre2/d0.replay-vbr/d7/f7e-1 /mnt/lustre2/d0.replay-vbr/d7/f7e-0");echo XXRETCODE:$?'
May 17 19:33:16 client-18 xinetd[3390]: EXIT: shell status=0 pid=3913 duration=1(sec)
May 17 19:33:16 client-18 xinetd[3390]: START: shell pid=3948 from=192.168.4.5
May 17 19:33:16 client-18 rshd[3949]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "grep -c /mnt/lustre2' ' /proc/mounts");echo XXRETCODE:$?'
May 17 19:33:16 client-18 xinetd[3390]: EXIT: shell status=0 pid=3948 duration=0(sec)
May 17 19:33:16 client-18 xinetd[3390]: START: shell pid=3983 from=192.168.4.5
May 17 19:33:16 client-18 rshd[3984]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lsof -t /mnt/lustre2");echo XXRETCODE:$?'
May 17 19:33:16 client-18 xinetd[3390]: EXIT: shell status=0 pid=3983 duration=0(sec)
May 17 19:33:16 client-18 xinetd[3390]: START: shell pid=4019 from=192.168.4.5
May 17 19:33:16 client-18 rshd[4020]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "umount  /mnt/lustre2 2>&1");echo XXRETCODE:$?'
May 17 19:33:16 client-18 xinetd[3390]: EXIT: shell status=0 pid=4019 duration=0(sec)
May 17 19:33:35 client-18 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail.
May 17 19:33:35 client-18 kernel: LustreError: Skipped 4 previous similar messages
May 17 19:33:35 client-18 kernel: Lustre: lustre-MDT0000-mdc-ffff8103028bc400: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete.
May 17 19:33:35 client-18 kernel: Lustre: 8885:0:(import.c:885:ptlrpc_connect_interpret()) MGS@MGC192.168.4.128@o2ib_0 changed server handle from 0xfff98862bf658ece to 0xfff98862bf659397
May 17 19:33:35 client-18 kernel: Lustre: Skipped 4 previous similar messages
May 17 19:33:58 client-18 kernel: Lustre: 8884:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369101282853263 sent from lustre-MDT0000-mdc-ffff8103028bc400 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305686005] [real_sent 1305686005] [current 1305686038] [deadline 33s] [delay 0s]  req@ffff81031ab22800 x1369101282853263/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 192/192 e 0 to 1 dl 1305686038 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
May 17 19:33:58 client-18 kernel: Lustre: 8884:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 13 previous similar messages
May 17 19:35:41 client-18 xinetd[3390]: START: shell pid=4067 from=192.168.4.5
May 17 19:35:41 client-18 rshd[4068]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 19:35:41 client-18 xinetd[3390]: EXIT: shell status=0 pid=4067 duration=0(sec)
May 17 19:35:41 client-18 xinetd[3390]: START: shell pid=4103 from=192.168.4.5
May 17 19:35:41 client-18 rshd[4104]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 19:35:41 client-18 xinetd[3390]: EXIT: shell status=0 pid=4103 duration=0(sec)
May 17 19:35:41 client-18 xinetd[3390]: START: shell pid=4140 from=192.168.4.5
May 17 19:35:41 client-18 rshd[4141]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-vbr test 7f: unlink, {lost}, rename ======================================================== 19:35:41 \(1305686141\)");echo XXRETCODE:$?'
May 17 19:35:42 client-18 kernel: Lustre: DEBUG MARKER: == replay-vbr test 7f: unlink, {lost}, rename ======================================================== 19:35:41 (1305686141)
May 17 19:35:42 client-18 xinetd[3390]: EXIT: shell status=0 pid=4140 duration=1(sec)
May 17 19:35:42 client-18 xinetd[3390]: START: shell pid=4175 from=192.168.4.5
May 17 19:35:42 client-18 rshd[4176]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mkdir -p /mnt/lustre2");echo XXRETCODE:$?'
May 17 19:35:42 client-18 xinetd[3390]: START: shell pid=4210 from=192.168.4.5
May 17 19:35:42 client-18 xinetd[3390]: EXIT: shell status=0 pid=4175 duration=0(sec)
May 17 19:35:42 client-18 rshd[4211]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre2");echo XXRETCODE:$?'
May 17 19:35:42 client-18 xinetd[3390]: START: shell pid=4248 from=192.168.4.5
May 17 19:35:42 client-18 xinetd[3390]: EXIT: shell status=0 pid=4210 duration=0(sec)
May 17 19:35:42 client-18 rshd[4249]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param debug=\"0x33f0406\";         lctl set_param subsystem_debug=\"0xffb7e3ff\";         lctl set_param debug_mb=32");echo XXRETCODE:$?'
May 17 19:35:42 client-18 xinetd[3390]: EXIT: shell status=0 pid=4248 duration=0(sec)
May 17 19:35:42 client-18 xinetd[3390]: START: shell pid=4285 from=192.168.4.5
May 17 19:35:42 client-18 rshd[4286]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 19:35:42 client-18 xinetd[3390]: EXIT: shell status=0 pid=4285 duration=0(sec)
May 17 19:35:43 client-18 xinetd[3390]: START: shell pid=4324 from=192.168.4.5
May 17 19:35:43 client-18 rshd[4325]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "createmany -o /mnt/lustre2/d0.replay-vbr/d7/f7f- 1");echo XXRETCODE:$?'
May 17 19:35:43 client-18 xinetd[3390]: EXIT: shell status=0 pid=4324 duration=0(sec)
May 17 19:35:43 client-18 xinetd[3390]: START: shell pid=4359 from=192.168.4.5
May 17 19:35:43 client-18 rshd[4360]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "grep -c /mnt/lustre2' ' /proc/mounts");echo XXRETCODE:$?'
May 17 19:35:43 client-18 xinetd[3390]: EXIT: shell status=0 pid=4359 duration=0(sec)
May 17 19:35:43 client-18 xinetd[3390]: START: shell pid=4394 from=192.168.4.5
May 17 19:35:43 client-18 rshd[4395]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lsof -t /mnt/lustre2");echo XXRETCODE:$?'
May 17 19:35:43 client-18 xinetd[3390]: EXIT: shell status=0 pid=4394 duration=0(sec)
May 17 19:35:43 client-18 xinetd[3390]: START: shell pid=4430 from=192.168.4.5
May 17 19:35:43 client-18 rshd[4431]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "umount  /mnt/lustre2 2>&1");echo XXRETCODE:$?'
May 17 19:35:43 client-18 xinetd[3390]: EXIT: shell status=0 pid=4430 duration=0(sec)
May 17 19:36:02 client-18 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The obd_ping operation failed with -107
May 17 19:36:02 client-18 kernel: LustreError: Skipped 7 previous similar messages
May 17 19:36:02 client-18 kernel: Lustre: 8885:0:(import.c:885:ptlrpc_connect_interpret()) MGS@MGC192.168.4.128@o2ib_0 changed server handle from 0xfff98862bf659397 to 0xfff98862bf659901
May 17 19:38:08 client-18 xinetd[3390]: START: shell pid=4478 from=192.168.4.5
May 17 19:38:08 client-18 rshd[4479]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mkdir -p /mnt/lustre2");echo XXRETCODE:$?'
May 17 19:38:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=4478 duration=0(sec)
May 17 19:38:08 client-18 xinetd[3390]: START: shell pid=4513 from=192.168.4.5
May 17 19:38:08 client-18 rshd[4514]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre2");echo XXRETCODE:$?'
May 17 19:38:09 client-18 xinetd[3390]: EXIT: shell status=0 pid=4513 duration=1(sec)
May 17 19:38:09 client-18 xinetd[3390]: START: shell pid=4551 from=192.168.4.5
May 17 19:38:09 client-18 rshd[4552]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param debug=\"0x33f0406\";         lctl set_param subsystem_debug=\"0xffb7e3ff\";         lctl set_param debug_mb=32");echo XXRETCODE:$?'
May 17 19:38:09 client-18 xinetd[3390]: EXIT: shell status=0 pid=4551 duration=0(sec)
May 17 19:38:09 client-18 xinetd[3390]: START: shell pid=4588 from=192.168.4.5
May 17 19:38:09 client-18 rshd[4589]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 19:38:09 client-18 xinetd[3390]: EXIT: shell status=0 pid=4588 duration=0(sec)
May 17 19:38:09 client-18 xinetd[3390]: START: shell pid=4627 from=192.168.4.5
May 17 19:38:09 client-18 rshd[4628]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "createmany -o /mnt/lustre2/d0.replay-vbr/d7/f7f- 1");echo XXRETCODE:$?'
May 17 19:38:10 client-18 xinetd[3390]: EXIT: shell status=0 pid=4627 duration=1(sec)
May 17 19:38:10 client-18 xinetd[3390]: START: shell pid=4662 from=192.168.4.5
May 17 19:38:10 client-18 rshd[4663]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "grep -c /mnt/lustre2' ' /proc/mounts");echo XXRETCODE:$?'
May 17 19:38:10 client-18 xinetd[3390]: EXIT: shell status=0 pid=4662 duration=0(sec)
May 17 19:38:10 client-18 xinetd[3390]: START: shell pid=4697 from=192.168.4.5
May 17 19:38:10 client-18 rshd[4698]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lsof -t /mnt/lustre2");echo XXRETCODE:$?'
May 17 19:38:10 client-18 xinetd[3390]: EXIT: shell status=0 pid=4697 duration=0(sec)
May 17 19:38:10 client-18 xinetd[3390]: START: shell pid=4733 from=192.168.4.5
May 17 19:38:10 client-18 rshd[4734]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "umount  /mnt/lustre2 2>&1");echo XXRETCODE:$?'
May 17 19:38:10 client-18 kernel: LustreError: 4768:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway
May 17 19:38:10 client-18 kernel: LustreError: 4768:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Skipped 14 previous similar messages
May 17 19:38:10 client-18 kernel: LustreError: 4768:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108
May 17 19:38:10 client-18 kernel: LustreError: 4768:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) Skipped 14 previous similar messages
May 17 19:38:10 client-18 kernel: Lustre: client ffff8103028bc800 umount complete
May 17 19:38:10 client-18 kernel: Lustre: Skipped 4 previous similar messages
May 17 19:38:10 client-18 xinetd[3390]: EXIT: shell status=0 pid=4733 duration=0(sec)
May 17 19:38:28 client-18 kernel: Lustre: 8885:0:(import.c:885:ptlrpc_connect_interpret()) MGS@MGC192.168.4.128@o2ib_0 changed server handle from 0xfff98862bf659901 to 0xfff98862bf659e1e
May 17 19:38:29 client-18 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import
May 17 19:38:29 client-18 kernel: Lustre: Skipped 4 previous similar messages
May 17 19:40:00 client-18 kernel: Lustre: lustre-MDT0000-mdc-ffff8103028bc400: Connection restored to service lustre-MDT0000 using nid 192.168.4.128@o2ib.
May 17 19:40:00 client-18 kernel: Lustre: Skipped 8 previous similar messages
May 17 19:40:35 client-18 xinetd[3390]: START: shell pid=4781 from=192.168.4.5
May 17 19:40:35 client-18 rshd[4782]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mkdir -p /mnt/lustre2");echo XXRETCODE:$?'
May 17 19:40:35 client-18 xinetd[3390]: START: shell pid=4816 from=192.168.4.5
May 17 19:40:35 client-18 xinetd[3390]: EXIT: shell status=0 pid=4781 duration=0(sec)
May 17 19:40:35 client-18 rshd[4817]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre2");echo XXRETCODE:$?'
May 17 19:40:35 client-18 xinetd[3390]: EXIT: shell status=0 pid=4816 duration=0(sec)
May 17 19:40:35 client-18 xinetd[3390]: START: shell pid=4854 from=192.168.4.5
May 17 19:40:35 client-18 rshd[4855]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param debug=\"0x33f0406\";         lctl set_param subsystem_debug=\"0xffb7e3ff\";         lctl set_param debug_mb=32");echo XXRETCODE:$?'
May 17 19:40:35 client-18 kernel: Lustre: 4889:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release.
May 17 19:40:35 client-18 kernel: Lustre: 4889:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 9 previous similar messages
May 17 19:40:35 client-18 xinetd[3390]: EXIT: shell status=0 pid=4854 duration=0(sec)
May 17 19:40:36 client-18 xinetd[3390]: START: shell pid=4891 from=192.168.4.5
May 17 19:40:36 client-18 rshd[4892]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 19:40:36 client-18 xinetd[3390]: EXIT: shell status=0 pid=4891 duration=0(sec)
May 17 19:40:36 client-18 xinetd[3390]: START: shell pid=4930 from=192.168.4.5
May 17 19:40:36 client-18 rshd[4931]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mkdir /mnt/lustre2/d0.replay-vbr/d7/f7f");echo XXRETCODE:$?'
May 17 19:40:36 client-18 xinetd[3390]: EXIT: shell status=0 pid=4930 duration=0(sec)
May 17 19:40:36 client-18 xinetd[3390]: START: shell pid=4965 from=192.168.4.5
May 17 19:40:36 client-18 rshd[4966]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "grep -c /mnt/lustre2' ' /proc/mounts");echo XXRETCODE:$?'
May 17 19:40:36 client-18 xinetd[3390]: EXIT: shell status=0 pid=4965 duration=0(sec)
May 17 19:40:36 client-18 xinetd[3390]: START: shell pid=5000 from=192.168.4.5
May 17 19:40:36 client-18 rshd[5001]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lsof -t /mnt/lustre2");echo XXRETCODE:$?'
May 17 19:40:36 client-18 xinetd[3390]: EXIT: shell status=0 pid=5000 duration=0(sec)
May 17 19:40:36 client-18 xinetd[3390]: START: shell pid=5036 from=192.168.4.5
May 17 19:40:36 client-18 rshd[5037]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "umount  /mnt/lustre2 2>&1");echo XXRETCODE:$?'
May 17 19:40:37 client-18 xinetd[3390]: EXIT: shell status=0 pid=5036 duration=1(sec)
May 17 19:40:57 client-18 kernel: Lustre: 8886:0:(import.c:529:import_select_connection()) MGC192.168.4.128@o2ib: tried all connections, increasing latency to 6s
May 17 19:40:57 client-18 kernel: Lustre: 8886:0:(import.c:529:import_select_connection()) Skipped 1 previous similar message
May 17 19:40:57 client-18 kernel: Lustre: 8885:0:(import.c:885:ptlrpc_connect_interpret()) MGS@MGC192.168.4.128@o2ib_0 changed server handle from 0xfff98862bf659e1e to 0xfff98862bf65a396
May 17 19:43:02 client-18 xinetd[3390]: START: shell pid=5084 from=192.168.4.5
May 17 19:43:02 client-18 rshd[5085]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mkdir -p /mnt/lustre2");echo XXRETCODE:$?'
May 17 19:43:02 client-18 xinetd[3390]: EXIT: shell status=0 pid=5084 duration=0(sec)
May 17 19:43:02 client-18 xinetd[3390]: START: shell pid=5119 from=192.168.4.5
May 17 19:43:02 client-18 rshd[5120]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre2");echo XXRETCODE:$?'
May 17 19:43:02 client-18 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'.
May 17 19:43:02 client-18 kernel: LustreError: Skipped 4 previous similar messages
May 17 19:43:02 client-18 kernel: Lustre: 5154:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff8102faf15c00->192.168.4.128@o2ib netid 50000: select flavor null
May 17 19:43:02 client-18 kernel: Lustre: 5154:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 34 previous similar messages
May 17 19:43:03 client-18 kernel: Lustre: Client lustre-client has started
May 17 19:43:03 client-18 kernel: Lustre: Skipped 4 previous similar messages
May 17 19:43:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=5119 duration=1(sec)
May 17 19:43:03 client-18 xinetd[3390]: START: shell pid=5157 from=192.168.4.5
May 17 19:43:03 client-18 rshd[5158]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param debug=\"0x33f0406\";         lctl set_param subsystem_debug=\"0xffb7e3ff\";         lctl set_param debug_mb=32");echo XXRETCODE:$?'
May 17 19:43:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=5157 duration=0(sec)
May 17 19:43:03 client-18 xinetd[3390]: START: shell pid=5194 from=192.168.4.5
May 17 19:43:03 client-18 rshd[5195]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 19:43:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=5194 duration=0(sec)
May 17 19:43:03 client-18 xinetd[3390]: START: shell pid=5233 from=192.168.4.5
May 17 19:43:03 client-18 rshd[5234]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mv /mnt/lustre2/d0.replay-vbr/d7/f7f-1 /mnt/lustre2/d0.replay-vbr/d7/f7f-0");echo XXRETCODE:$?'
May 17 19:43:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=5233 duration=0(sec)
May 17 19:43:03 client-18 xinetd[3390]: START: shell pid=5268 from=192.168.4.5
May 17 19:43:04 client-18 rshd[5269]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "grep -c /mnt/lustre2' ' /proc/mounts");echo XXRETCODE:$?'
May 17 19:43:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=5268 duration=1(sec)
May 17 19:43:04 client-18 xinetd[3390]: START: shell pid=5303 from=192.168.4.5
May 17 19:43:04 client-18 rshd[5304]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lsof -t /mnt/lustre2");echo XXRETCODE:$?'
May 17 19:43:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=5303 duration=0(sec)
May 17 19:43:04 client-18 xinetd[3390]: START: shell pid=5339 from=192.168.4.5
May 17 19:43:04 client-18 rshd[5340]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "umount  /mnt/lustre2 2>&1");echo XXRETCODE:$?'
May 17 19:43:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=5339 duration=0(sec)
May 17 19:43:34 client-18 kernel: Lustre: 8885:0:(import.c:885:ptlrpc_connect_interpret()) MGS@MGC192.168.4.128@o2ib_0 changed server handle from 0xfff98862bf65a396 to 0xfff98862bf65a9d2
May 17 19:45:29 client-18 xinetd[3390]: START: shell pid=5387 from=192.168.4.5
May 17 19:45:29 client-18 rshd[5388]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mkdir -p /mnt/lustre2");echo XXRETCODE:$?'
May 17 19:45:29 client-18 xinetd[3390]: EXIT: shell status=0 pid=5387 duration=0(sec)
May 17 19:45:29 client-18 xinetd[3390]: START: shell pid=5422 from=192.168.4.5
May 17 19:45:29 client-18 rshd[5423]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre2");echo XXRETCODE:$?'
May 17 19:45:30 client-18 xinetd[3390]: EXIT: shell status=0 pid=5422 duration=1(sec)
May 17 19:45:30 client-18 xinetd[3390]: START: shell pid=5460 from=192.168.4.5
May 17 19:45:30 client-18 rshd[5461]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param debug=\"0x33f0406\";         lctl set_param subsystem_debug=\"0xffb7e3ff\";         lctl set_param debug_mb=32");echo XXRETCODE:$?'
May 17 19:45:30 client-18 xinetd[3390]: EXIT: shell status=0 pid=5460 duration=0(sec)
May 17 19:45:30 client-18 xinetd[3390]: START: shell pid=5497 from=192.168.4.5
May 17 19:45:30 client-18 rshd[5498]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 19:45:30 client-18 xinetd[3390]: EXIT: shell status=0 pid=5497 duration=0(sec)
May 17 19:45:30 client-18 xinetd[3390]: START: shell pid=5536 from=192.168.4.5
May 17 19:45:30 client-18 rshd[5537]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mkdir /mnt/lustre2/d0.replay-vbr/d7/f7f-0");echo XXRETCODE:$?'
May 17 19:45:31 client-18 xinetd[3390]: EXIT: shell status=0 pid=5536 duration=1(sec)
May 17 19:45:31 client-18 xinetd[3390]: START: shell pid=5571 from=192.168.4.5
May 17 19:45:31 client-18 rshd[5572]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "grep -c /mnt/lustre2' ' /proc/mounts");echo XXRETCODE:$?'
May 17 19:45:31 client-18 xinetd[3390]: EXIT: shell status=0 pid=5571 duration=0(sec)
May 17 19:45:31 client-18 xinetd[3390]: START: shell pid=5606 from=192.168.4.5
May 17 19:45:31 client-18 rshd[5607]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lsof -t /mnt/lustre2");echo XXRETCODE:$?'
May 17 19:45:31 client-18 xinetd[3390]: EXIT: shell status=0 pid=5606 duration=0(sec)
May 17 19:45:31 client-18 xinetd[3390]: START: shell pid=5642 from=192.168.4.5
May 17 19:45:31 client-18 rshd[5643]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "umount  /mnt/lustre2 2>&1");echo XXRETCODE:$?'
May 17 19:45:31 client-18 xinetd[3390]: EXIT: shell status=0 pid=5642 duration=0(sec)
May 17 19:45:50 client-18 kernel: Lustre: lustre-MDT0000-mdc-ffff8103028bc400: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete.
May 17 19:45:50 client-18 kernel: Lustre: Skipped 4 previous similar messages
May 17 19:45:50 client-18 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail.
May 17 19:45:50 client-18 kernel: LustreError: Skipped 4 previous similar messages
May 17 19:45:50 client-18 kernel: Lustre: 8885:0:(import.c:885:ptlrpc_connect_interpret()) MGS@MGC192.168.4.128@o2ib_0 changed server handle from 0xfff98862bf65a9d2 to 0xfff98862bf65aee1
May 17 19:46:13 client-18 kernel: Lustre: 8884:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369101282854553 sent from lustre-MDT0000-mdc-ffff8103028bc400 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305686739] [real_sent 1305686739] [current 1305686772] [deadline 33s] [delay 0s]  req@ffff8102edee1000 x1369101282854553/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 192/192 e 0 to 1 dl 1305686772 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
May 17 19:46:13 client-18 kernel: Lustre: 8884:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 13 previous similar messages
May 17 19:47:56 client-18 xinetd[3390]: START: shell pid=5690 from=192.168.4.5
May 17 19:47:56 client-18 rshd[5691]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 19:47:56 client-18 xinetd[3390]: EXIT: shell status=0 pid=5690 duration=0(sec)
May 17 19:47:56 client-18 xinetd[3390]: START: shell pid=5726 from=192.168.4.5
May 17 19:47:56 client-18 rshd[5727]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 19:47:56 client-18 xinetd[3390]: EXIT: shell status=0 pid=5726 duration=0(sec)
May 17 19:47:56 client-18 xinetd[3390]: START: shell pid=5763 from=192.168.4.5
May 17 19:47:56 client-18 rshd[5764]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-vbr test 7g: rename, {lost}, create ======================================================== 19:47:56 \(1305686876\)");echo XXRETCODE:$?'
May 17 19:47:56 client-18 kernel: Lustre: DEBUG MARKER: == replay-vbr test 7g: rename, {lost}, create ======================================================== 19:47:56 (1305686876)
May 17 19:47:56 client-18 xinetd[3390]: EXIT: shell status=0 pid=5763 duration=0(sec)
May 17 19:47:56 client-18 xinetd[3390]: START: shell pid=5798 from=192.168.4.5
May 17 19:47:56 client-18 rshd[5799]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mkdir -p /mnt/lustre2");echo XXRETCODE:$?'
May 17 19:47:57 client-18 xinetd[3390]: START: shell pid=5833 from=192.168.4.5
May 17 19:47:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=5798 duration=1(sec)
May 17 19:47:57 client-18 rshd[5834]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre2");echo XXRETCODE:$?'
May 17 19:47:57 client-18 xinetd[3390]: START: shell pid=5871 from=192.168.4.5
May 17 19:47:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=5833 duration=0(sec)
May 17 19:47:57 client-18 rshd[5872]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param debug=\"0x33f0406\";         lctl set_param subsystem_debug=\"0xffb7e3ff\";         lctl set_param debug_mb=32");echo XXRETCODE:$?'
May 17 19:47:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=5871 duration=0(sec)
May 17 19:47:57 client-18 xinetd[3390]: START: shell pid=5908 from=192.168.4.5
May 17 19:47:57 client-18 rshd[5909]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 19:47:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=5908 duration=0(sec)
May 17 19:47:58 client-18 xinetd[3390]: START: shell pid=5947 from=192.168.4.5
May 17 19:47:58 client-18 rshd[5948]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mkdir /mnt/lustre2/d0.replay-vbr/d7/f7g-0;rmdir /mnt/lustre2/d0.replay-vbr/d7/f7g-0");echo XXRETCODE:$?'
May 17 19:47:58 client-18 xinetd[3390]: EXIT: shell status=0 pid=5947 duration=0(sec)
May 17 19:47:58 client-18 xinetd[3390]: START: shell pid=5984 from=192.168.4.5
May 17 19:47:58 client-18 rshd[5985]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "grep -c /mnt/lustre2' ' /proc/mounts");echo XXRETCODE:$?'
May 17 19:47:58 client-18 xinetd[3390]: EXIT: shell status=0 pid=5984 duration=0(sec)
May 17 19:47:58 client-18 xinetd[3390]: START: shell pid=6019 from=192.168.4.5
May 17 19:47:58 client-18 rshd[6020]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lsof -t /mnt/lustre2");echo XXRETCODE:$?'
May 17 19:47:58 client-18 xinetd[3390]: EXIT: shell status=0 pid=6019 duration=0(sec)
May 17 19:47:58 client-18 xinetd[3390]: START: shell pid=6055 from=192.168.4.5
May 17 19:47:58 client-18 rshd[6056]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "umount  /mnt/lustre2 2>&1");echo XXRETCODE:$?'
May 17 19:47:58 client-18 xinetd[3390]: EXIT: shell status=0 pid=6055 duration=0(sec)
May 17 19:48:17 client-18 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The obd_ping operation failed with -107
May 17 19:48:17 client-18 kernel: LustreError: Skipped 7 previous similar messages
May 17 19:48:17 client-18 kernel: Lustre: 8885:0:(import.c:885:ptlrpc_connect_interpret()) MGS@MGC192.168.4.128@o2ib_0 changed server handle from 0xfff98862bf65aee1 to 0xfff98862bf65b467
May 17 19:49:50 client-18 xinetd[3390]: START: shell pid=6099 from=192.168.4.5
May 17 19:49:50 client-18 rshd[6100]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mkdir -p /mnt/lustre2");echo XXRETCODE:$?'
May 17 19:49:50 client-18 xinetd[3390]: START: shell pid=6135 from=192.168.4.5
May 17 19:49:50 client-18 xinetd[3390]: EXIT: shell status=0 pid=6099 duration=0(sec)
May 17 19:49:50 client-18 rshd[6136]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre2");echo XXRETCODE:$?'
May 17 19:49:50 client-18 xinetd[3390]: EXIT: shell status=0 pid=6135 duration=0(sec)
May 17 19:49:50 client-18 xinetd[3390]: START: shell pid=6173 from=192.168.4.5
May 17 19:49:50 client-18 rshd[6174]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param debug=\"0x33f0406\";         lctl set_param subsystem_debug=\"0xffb7e3ff\";         lctl set_param debug_mb=32");echo XXRETCODE:$?'
May 17 19:49:50 client-18 xinetd[3390]: EXIT: shell status=0 pid=6173 duration=0(sec)
May 17 19:49:51 client-18 xinetd[3390]: START: shell pid=6210 from=192.168.4.5
May 17 19:49:51 client-18 rshd[6211]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 19:49:51 client-18 xinetd[3390]: EXIT: shell status=0 pid=6210 duration=0(sec)
May 17 19:49:51 client-18 xinetd[3390]: START: shell pid=6249 from=192.168.4.5
May 17 19:49:51 client-18 rshd[6250]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "createmany -o /mnt/lustre2/d0.replay-vbr/d7/f7g- 1; rm /mnt/lustre2/d0.replay-vbr/d7/f7g-0");echo XXRETCODE:$?'
May 17 19:49:51 client-18 xinetd[3390]: EXIT: shell status=0 pid=6249 duration=0(sec)
May 17 19:49:51 client-18 xinetd[3390]: START: shell pid=6286 from=192.168.4.5
May 17 19:49:51 client-18 rshd[6287]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "grep -c /mnt/lustre2' ' /proc/mounts");echo XXRETCODE:$?'
May 17 19:49:51 client-18 xinetd[3390]: START: shell pid=6321 from=192.168.4.5
May 17 19:49:51 client-18 xinetd[3390]: EXIT: shell status=0 pid=6286 duration=0(sec)
May 17 19:49:51 client-18 rshd[6322]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lsof -t /mnt/lustre2");echo XXRETCODE:$?'
May 17 19:49:52 client-18 xinetd[3390]: EXIT: shell status=0 pid=6321 duration=1(sec)
May 17 19:49:52 client-18 xinetd[3390]: START: shell pid=6357 from=192.168.4.5
May 17 19:49:52 client-18 rshd[6358]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "umount  /mnt/lustre2 2>&1");echo XXRETCODE:$?'
May 17 19:49:52 client-18 kernel: LustreError: 6392:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway
May 17 19:49:52 client-18 kernel: LustreError: 6392:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Skipped 14 previous similar messages
May 17 19:49:52 client-18 kernel: LustreError: 6392:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108
May 17 19:49:52 client-18 kernel: LustreError: 6392:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) Skipped 14 previous similar messages
May 17 19:49:52 client-18 kernel: Lustre: client ffff8102fe352800 umount complete
May 17 19:49:52 client-18 kernel: Lustre: Skipped 4 previous similar messages
May 17 19:49:52 client-18 xinetd[3390]: EXIT: shell status=0 pid=6357 duration=0(sec)
May 17 19:50:10 client-18 kernel: Lustre: 8885:0:(import.c:885:ptlrpc_connect_interpret()) MGS@MGC192.168.4.128@o2ib_0 changed server handle from 0xfff98862bf65b467 to 0xfff98862bf65b9f4
May 17 19:50:10 client-18 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import
May 17 19:50:10 client-18 kernel: Lustre: Skipped 4 previous similar messages
May 17 19:50:10 client-18 kernel: Lustre: MGC192.168.4.128@o2ib: Connection restored to service MGS using nid 192.168.4.128@o2ib.
May 17 19:50:10 client-18 kernel: Lustre: Skipped 8 previous similar messages
May 17 19:51:44 client-18 xinetd[3390]: START: shell pid=6405 from=192.168.4.5
May 17 19:51:44 client-18 rshd[6406]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mkdir -p /mnt/lustre2");echo XXRETCODE:$?'
May 17 19:51:44 client-18 xinetd[3390]: START: shell pid=6440 from=192.168.4.5
May 17 19:51:44 client-18 xinetd[3390]: EXIT: shell status=0 pid=6405 duration=0(sec)
May 17 19:51:44 client-18 rshd[6441]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre2");echo XXRETCODE:$?'
May 17 19:51:44 client-18 xinetd[3390]: EXIT: shell status=0 pid=6440 duration=0(sec)
May 17 19:51:44 client-18 xinetd[3390]: START: shell pid=6478 from=192.168.4.5
May 17 19:51:44 client-18 rshd[6479]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param debug=\"0x33f0406\";         lctl set_param subsystem_debug=\"0xffb7e3ff\";         lctl set_param debug_mb=32");echo XXRETCODE:$?'
May 17 19:51:44 client-18 kernel: Lustre: 6513:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release.
May 17 19:51:44 client-18 kernel: Lustre: 6513:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 9 previous similar messages
May 17 19:51:44 client-18 xinetd[3390]: EXIT: shell status=0 pid=6478 duration=0(sec)
May 17 19:51:44 client-18 xinetd[3390]: START: shell pid=6515 from=192.168.4.5
May 17 19:51:44 client-18 rshd[6516]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 19:51:45 client-18 xinetd[3390]: EXIT: shell status=0 pid=6515 duration=1(sec)
May 17 19:51:45 client-18 xinetd[3390]: START: shell pid=6554 from=192.168.4.5
May 17 19:51:45 client-18 rshd[6555]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "createmany -o /mnt/lustre2/d0.replay-vbr/d7/f7g- 1");echo XXRETCODE:$?'
May 17 19:51:45 client-18 xinetd[3390]: EXIT: shell status=0 pid=6554 duration=0(sec)
May 17 19:51:45 client-18 xinetd[3390]: START: shell pid=6589 from=192.168.4.5
May 17 19:51:45 client-18 rshd[6590]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "grep -c /mnt/lustre2' ' /proc/mounts");echo XXRETCODE:$?'
May 17 19:51:45 client-18 xinetd[3390]: EXIT: shell status=0 pid=6589 duration=0(sec)
May 17 19:51:45 client-18 xinetd[3390]: START: shell pid=6624 from=192.168.4.5
May 17 19:51:45 client-18 rshd[6625]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lsof -t /mnt/lustre2");echo XXRETCODE:$?'
May 17 19:51:45 client-18 xinetd[3390]: EXIT: shell status=0 pid=6624 duration=0(sec)
May 17 19:51:45 client-18 xinetd[3390]: START: shell pid=6660 from=192.168.4.5
May 17 19:51:45 client-18 rshd[6661]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "umount  /mnt/lustre2 2>&1");echo XXRETCODE:$?'
May 17 19:51:46 client-18 xinetd[3390]: EXIT: shell status=0 pid=6660 duration=1(sec)
May 17 19:52:04 client-18 kernel: Lustre: 8885:0:(import.c:885:ptlrpc_connect_interpret()) MGS@MGC192.168.4.128@o2ib_0 changed server handle from 0xfff98862bf65b9f4 to 0xfff98862bf65bfe3
May 17 19:54:10 client-18 xinetd[3390]: START: shell pid=6708 from=192.168.4.5
May 17 19:54:11 client-18 rshd[6709]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 19:54:11 client-18 xinetd[3390]: EXIT: shell status=0 pid=6708 duration=1(sec)
May 17 19:54:11 client-18 xinetd[3390]: START: shell pid=6744 from=192.168.4.5
May 17 19:54:11 client-18 rshd[6745]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 19:54:11 client-18 xinetd[3390]: EXIT: shell status=0 pid=6744 duration=0(sec)
May 17 19:54:11 client-18 xinetd[3390]: START: shell pid=6781 from=192.168.4.5
May 17 19:54:11 client-18 rshd[6782]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-vbr test 7h: rename, {lost}, unlink ======================================================== 19:54:11 \(1305687251\)");echo XXRETCODE:$?'
May 17 19:54:11 client-18 kernel: Lustre: DEBUG MARKER: == replay-vbr test 7h: rename, {lost}, unlink ======================================================== 19:54:11 (1305687251)
May 17 19:54:11 client-18 xinetd[3390]: EXIT: shell status=0 pid=6781 duration=0(sec)
May 17 19:54:11 client-18 xinetd[3390]: START: shell pid=6816 from=192.168.4.5
May 17 19:54:11 client-18 rshd[6817]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mkdir -p /mnt/lustre2");echo XXRETCODE:$?'
May 17 19:54:11 client-18 xinetd[3390]: START: shell pid=6851 from=192.168.4.5
May 17 19:54:11 client-18 xinetd[3390]: EXIT: shell status=0 pid=6816 duration=0(sec)
May 17 19:54:11 client-18 rshd[6852]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre2");echo XXRETCODE:$?'
May 17 19:54:11 client-18 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'.
May 17 19:54:11 client-18 kernel: LustreError: Skipped 4 previous similar messages
May 17 19:54:11 client-18 kernel: Lustre: 6886:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff8102ed203000->192.168.4.128@o2ib netid 50000: select flavor null
May 17 19:54:11 client-18 kernel: Lustre: 6886:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 34 previous similar messages
May 17 19:54:11 client-18 kernel: Lustre: Client lustre-client has started
May 17 19:54:11 client-18 kernel: Lustre: Skipped 4 previous similar messages
May 17 19:54:11 client-18 xinetd[3390]: EXIT: shell status=0 pid=6851 duration=0(sec)
May 17 19:54:11 client-18 xinetd[3390]: START: shell pid=6889 from=192.168.4.5
May 17 19:54:11 client-18 rshd[6890]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param debug=\"0x33f0406\";         lctl set_param subsystem_debug=\"0xffb7e3ff\";         lctl set_param debug_mb=32");echo XXRETCODE:$?'
May 17 19:54:11 client-18 xinetd[3390]: EXIT: shell status=0 pid=6889 duration=0(sec)
May 17 19:54:12 client-18 xinetd[3390]: START: shell pid=6926 from=192.168.4.5
May 17 19:54:12 client-18 rshd[6927]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 19:54:12 client-18 xinetd[3390]: EXIT: shell status=0 pid=6926 duration=0(sec)
May 17 19:54:12 client-18 xinetd[3390]: START: shell pid=6965 from=192.168.4.5
May 17 19:54:12 client-18 rshd[6966]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "createmany -o /mnt/lustre2/d0.replay-vbr/d7/f7h- 1");echo XXRETCODE:$?'
May 17 19:54:12 client-18 xinetd[3390]: EXIT: shell status=0 pid=6965 duration=0(sec)
May 17 19:54:12 client-18 xinetd[3390]: START: shell pid=7000 from=192.168.4.5
May 17 19:54:12 client-18 rshd[7001]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "grep -c /mnt/lustre2' ' /proc/mounts");echo XXRETCODE:$?'
May 17 19:54:12 client-18 xinetd[3390]: START: shell pid=7035 from=192.168.4.5
May 17 19:54:12 client-18 xinetd[3390]: EXIT: shell status=0 pid=7000 duration=0(sec)
May 17 19:54:12 client-18 rshd[7036]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lsof -t /mnt/lustre2");echo XXRETCODE:$?'
May 17 19:54:12 client-18 xinetd[3390]: EXIT: shell status=0 pid=7035 duration=0(sec)
May 17 19:54:12 client-18 xinetd[3390]: START: shell pid=7071 from=192.168.4.5
May 17 19:54:12 client-18 rshd[7072]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "umount  /mnt/lustre2 2>&1");echo XXRETCODE:$?'
May 17 19:54:13 client-18 xinetd[3390]: EXIT: shell status=0 pid=7071 duration=1(sec)
May 17 19:54:33 client-18 kernel: Lustre: 8886:0:(import.c:529:import_select_connection()) MGC192.168.4.128@o2ib: tried all connections, increasing latency to 6s
May 17 19:54:33 client-18 kernel: Lustre: 8886:0:(import.c:529:import_select_connection()) Skipped 1 previous similar message
May 17 19:54:33 client-18 kernel: Lustre: 8885:0:(import.c:885:ptlrpc_connect_interpret()) MGS@MGC192.168.4.128@o2ib_0 changed server handle from 0xfff98862bf65bfe3 to 0xfff98862bf65c546
May 17 19:56:37 client-18 xinetd[3390]: START: shell pid=7119 from=192.168.4.5
May 17 19:56:37 client-18 rshd[7120]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mkdir -p /mnt/lustre2");echo XXRETCODE:$?'
May 17 19:56:37 client-18 xinetd[3390]: START: shell pid=7154 from=192.168.4.5
May 17 19:56:37 client-18 xinetd[3390]: EXIT: shell status=0 pid=7119 duration=0(sec)
May 17 19:56:37 client-18 rshd[7155]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre2");echo XXRETCODE:$?'
May 17 19:56:38 client-18 xinetd[3390]: EXIT: shell status=0 pid=7154 duration=1(sec)
May 17 19:56:38 client-18 xinetd[3390]: START: shell pid=7192 from=192.168.4.5
May 17 19:56:38 client-18 rshd[7193]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param debug=\"0x33f0406\";         lctl set_param subsystem_debug=\"0xffb7e3ff\";         lctl set_param debug_mb=32");echo XXRETCODE:$?'
May 17 19:56:38 client-18 xinetd[3390]: EXIT: shell status=0 pid=7192 duration=0(sec)
May 17 19:56:38 client-18 xinetd[3390]: START: shell pid=7229 from=192.168.4.5
May 17 19:56:38 client-18 rshd[7230]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 19:56:38 client-18 xinetd[3390]: EXIT: shell status=0 pid=7229 duration=0(sec)
May 17 19:56:38 client-18 xinetd[3390]: START: shell pid=7268 from=192.168.4.5
May 17 19:56:38 client-18 rshd[7269]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "rm /mnt/lustre2/d0.replay-vbr/d7/f7h-0; createmany -o /mnt/lustre2/d0.replay-vbr/d7/f7h- 1");echo XXRETCODE:$?'
May 17 19:56:39 client-18 xinetd[3390]: EXIT: shell status=0 pid=7268 duration=1(sec)
May 17 19:56:39 client-18 xinetd[3390]: START: shell pid=7305 from=192.168.4.5
May 17 19:56:39 client-18 rshd[7306]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "grep -c /mnt/lustre2' ' /proc/mounts");echo XXRETCODE:$?'
May 17 19:56:39 client-18 xinetd[3390]: EXIT: shell status=0 pid=7305 duration=0(sec)
May 17 19:56:39 client-18 xinetd[3390]: START: shell pid=7340 from=192.168.4.5
May 17 19:56:39 client-18 rshd[7341]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lsof -t /mnt/lustre2");echo XXRETCODE:$?'
May 17 19:56:39 client-18 xinetd[3390]: EXIT: shell status=0 pid=7340 duration=0(sec)
May 17 19:56:39 client-18 xinetd[3390]: START: shell pid=7376 from=192.168.4.5
May 17 19:56:39 client-18 rshd[7377]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "umount  /mnt/lustre2 2>&1");echo XXRETCODE:$?'
May 17 19:56:39 client-18 xinetd[3390]: EXIT: shell status=0 pid=7376 duration=0(sec)
May 17 19:56:55 client-18 kernel: Lustre: 8884:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369101282855728 sent from MGC192.168.4.128@o2ib to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305687402] [real_sent 1305687402] [current 1305687414] [deadline 12s] [delay 0s]  req@ffff8102f87be400 x1369101282855728/t0(0) o-1->MGS@MGC192.168.4.128@o2ib_0:26/25 lens 192/192 e 0 to 1 dl 1305687414 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
May 17 19:56:55 client-18 kernel: Lustre: 8884:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 11 previous similar messages
May 17 19:56:55 client-18 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail.
May 17 19:56:55 client-18 kernel: LustreError: Skipped 4 previous similar messages
May 17 19:57:00 client-18 kernel: Lustre: lustre-MDT0000-mdc-ffff8103028bc400: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete.
May 17 19:57:00 client-18 kernel: Lustre: Skipped 4 previous similar messages
May 17 19:57:10 client-18 kernel: Lustre: 8885:0:(import.c:885:ptlrpc_connect_interpret()) MGS@MGC192.168.4.128@o2ib_0 changed server handle from 0xfff98862bf65c546 to 0xfff98862bf65caef
May 17 19:59:04 client-18 xinetd[3390]: START: shell pid=7424 from=192.168.4.5
May 17 19:59:04 client-18 rshd[7425]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 19:59:05 client-18 xinetd[3390]: EXIT: shell status=0 pid=7424 duration=1(sec)
May 17 19:59:05 client-18 xinetd[3390]: START: shell pid=7460 from=192.168.4.5
May 17 19:59:05 client-18 rshd[7461]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 19:59:05 client-18 xinetd[3390]: EXIT: shell status=0 pid=7460 duration=0(sec)
May 17 19:59:05 client-18 xinetd[3390]: START: shell pid=7497 from=192.168.4.5
May 17 19:59:05 client-18 rshd[7498]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-vbr test 7i: rename, {lost}, rename ======================================================== 19:59:05 \(1305687545\)");echo XXRETCODE:$?'
May 17 19:59:05 client-18 kernel: Lustre: DEBUG MARKER: == replay-vbr test 7i: rename, {lost}, rename ======================================================== 19:59:05 (1305687545)
May 17 19:59:05 client-18 xinetd[3390]: EXIT: shell status=0 pid=7497 duration=0(sec)
May 17 19:59:05 client-18 xinetd[3390]: START: shell pid=7532 from=192.168.4.5
May 17 19:59:05 client-18 rshd[7533]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mkdir -p /mnt/lustre2");echo XXRETCODE:$?'
May 17 19:59:05 client-18 xinetd[3390]: START: shell pid=7567 from=192.168.4.5
May 17 19:59:05 client-18 xinetd[3390]: EXIT: shell status=0 pid=7532 duration=0(sec)
May 17 19:59:05 client-18 rshd[7568]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre2");echo XXRETCODE:$?'
May 17 19:59:05 client-18 xinetd[3390]: EXIT: shell status=0 pid=7567 duration=0(sec)
May 17 19:59:05 client-18 xinetd[3390]: START: shell pid=7605 from=192.168.4.5
May 17 19:59:05 client-18 rshd[7606]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param debug=\"0x33f0406\";         lctl set_param subsystem_debug=\"0xffb7e3ff\";         lctl set_param debug_mb=32");echo XXRETCODE:$?'
May 17 19:59:05 client-18 xinetd[3390]: EXIT: shell status=0 pid=7605 duration=0(sec)
May 17 19:59:06 client-18 xinetd[3390]: START: shell pid=7642 from=192.168.4.5
May 17 19:59:06 client-18 rshd[7643]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 19:59:06 client-18 xinetd[3390]: EXIT: shell status=0 pid=7642 duration=0(sec)
May 17 19:59:06 client-18 xinetd[3390]: START: shell pid=7681 from=192.168.4.5
May 17 19:59:06 client-18 rshd[7682]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "createmany -o /mnt/lustre2/d0.replay-vbr/d7/f7i- 1");echo XXRETCODE:$?'
May 17 19:59:06 client-18 xinetd[3390]: EXIT: shell status=0 pid=7681 duration=0(sec)
May 17 19:59:06 client-18 xinetd[3390]: START: shell pid=7716 from=192.168.4.5
May 17 19:59:06 client-18 rshd[7717]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "grep -c /mnt/lustre2' ' /proc/mounts");echo XXRETCODE:$?'
May 17 19:59:06 client-18 xinetd[3390]: START: shell pid=7751 from=192.168.4.5
May 17 19:59:06 client-18 xinetd[3390]: EXIT: shell status=0 pid=7716 duration=0(sec)
May 17 19:59:06 client-18 rshd[7752]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lsof -t /mnt/lustre2");echo XXRETCODE:$?'
May 17 19:59:06 client-18 xinetd[3390]: START: shell pid=7787 from=192.168.4.5
May 17 19:59:06 client-18 xinetd[3390]: EXIT: shell status=0 pid=7751 duration=0(sec)
May 17 19:59:06 client-18 rshd[7788]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "umount  /mnt/lustre2 2>&1");echo XXRETCODE:$?'
May 17 19:59:07 client-18 xinetd[3390]: EXIT: shell status=0 pid=7787 duration=1(sec)
May 17 19:59:25 client-18 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The obd_ping operation failed with -107
May 17 19:59:25 client-18 kernel: LustreError: Skipped 7 previous similar messages
May 17 19:59:25 client-18 kernel: Lustre: 8885:0:(import.c:885:ptlrpc_connect_interpret()) MGS@MGC192.168.4.128@o2ib_0 changed server handle from 0xfff98862bf65caef to 0xfff98862bf65d060
May 17 20:00:56 client-18 kernel: Lustre: lustre-MDT0000-mdc-ffff8103028bc400: Connection restored to service lustre-MDT0000 using nid 192.168.4.128@o2ib.
May 17 20:00:56 client-18 kernel: Lustre: Skipped 8 previous similar messages
May 17 20:01:32 client-18 xinetd[3390]: START: shell pid=7841 from=192.168.4.5
May 17 20:01:32 client-18 rshd[7842]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mkdir -p /mnt/lustre2");echo XXRETCODE:$?'
May 17 20:01:32 client-18 xinetd[3390]: START: shell pid=7877 from=192.168.4.5
May 17 20:01:32 client-18 xinetd[3390]: EXIT: shell status=0 pid=7841 duration=0(sec)
May 17 20:01:32 client-18 rshd[7878]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre2");echo XXRETCODE:$?'
May 17 20:01:32 client-18 xinetd[3390]: EXIT: shell status=0 pid=7877 duration=0(sec)
May 17 20:01:32 client-18 xinetd[3390]: START: shell pid=7915 from=192.168.4.5
May 17 20:01:32 client-18 rshd[7916]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param debug=\"0x33f0406\";         lctl set_param subsystem_debug=\"0xffb7e3ff\";         lctl set_param debug_mb=32");echo XXRETCODE:$?'
May 17 20:01:32 client-18 xinetd[3390]: EXIT: shell status=0 pid=7915 duration=0(sec)
May 17 20:01:32 client-18 xinetd[3390]: START: shell pid=7952 from=192.168.4.5
May 17 20:01:32 client-18 rshd[7953]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 20:01:32 client-18 xinetd[3390]: EXIT: shell status=0 pid=7952 duration=0(sec)
May 17 20:01:33 client-18 xinetd[3390]: START: shell pid=7991 from=192.168.4.5
May 17 20:01:33 client-18 rshd[7992]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mkdir /mnt/lustre2/d0.replay-vbr/d7/f7i-0");echo XXRETCODE:$?'
May 17 20:01:33 client-18 xinetd[3390]: EXIT: shell status=0 pid=7991 duration=0(sec)
May 17 20:01:33 client-18 xinetd[3390]: START: shell pid=8026 from=192.168.4.5
May 17 20:01:33 client-18 rshd[8027]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "grep -c /mnt/lustre2' ' /proc/mounts");echo XXRETCODE:$?'
May 17 20:01:33 client-18 xinetd[3390]: START: shell pid=8061 from=192.168.4.5
May 17 20:01:33 client-18 xinetd[3390]: EXIT: shell status=0 pid=8026 duration=0(sec)
May 17 20:01:33 client-18 rshd[8062]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lsof -t /mnt/lustre2");echo XXRETCODE:$?'
May 17 20:01:33 client-18 xinetd[3390]: EXIT: shell status=0 pid=8061 duration=0(sec)
May 17 20:01:33 client-18 xinetd[3390]: START: shell pid=8097 from=192.168.4.5
May 17 20:01:33 client-18 rshd[8098]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "umount  /mnt/lustre2 2>&1");echo XXRETCODE:$?'
May 17 20:01:33 client-18 kernel: LustreError: 8132:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway
May 17 20:01:33 client-18 kernel: LustreError: 8132:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Skipped 14 previous similar messages
May 17 20:01:33 client-18 kernel: LustreError: 8132:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108
May 17 20:01:33 client-18 kernel: LustreError: 8132:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) Skipped 14 previous similar messages
May 17 20:01:33 client-18 kernel: Lustre: client ffff8102fb5b4800 umount complete
May 17 20:01:33 client-18 kernel: Lustre: Skipped 4 previous similar messages
May 17 20:01:33 client-18 xinetd[3390]: EXIT: shell status=0 pid=8097 duration=0(sec)
May 17 20:01:52 client-18 kernel: Lustre: 8885:0:(import.c:885:ptlrpc_connect_interpret()) MGS@MGC192.168.4.128@o2ib_0 changed server handle from 0xfff98862bf65d060 to 0xfff98862bf65d5ed
May 17 20:01:52 client-18 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import
May 17 20:01:52 client-18 kernel: Lustre: Skipped 4 previous similar messages
May 17 20:03:58 client-18 xinetd[3390]: START: shell pid=8145 from=192.168.4.5
May 17 20:03:58 client-18 rshd[8146]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mkdir -p /mnt/lustre2");echo XXRETCODE:$?'
May 17 20:03:58 client-18 xinetd[3390]: START: shell pid=8180 from=192.168.4.5
May 17 20:03:58 client-18 xinetd[3390]: EXIT: shell status=0 pid=8145 duration=0(sec)
May 17 20:03:58 client-18 rshd[8181]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre2");echo XXRETCODE:$?'
May 17 20:03:58 client-18 xinetd[3390]: EXIT: shell status=0 pid=8180 duration=0(sec)
May 17 20:03:58 client-18 xinetd[3390]: START: shell pid=8218 from=192.168.4.5
May 17 20:03:58 client-18 rshd[8219]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param debug=\"0x33f0406\";         lctl set_param subsystem_debug=\"0xffb7e3ff\";         lctl set_param debug_mb=32");echo XXRETCODE:$?'
May 17 20:03:59 client-18 kernel: Lustre: 8253:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release.
May 17 20:03:59 client-18 kernel: Lustre: 8253:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 9 previous similar messages
May 17 20:03:59 client-18 xinetd[3390]: EXIT: shell status=0 pid=8218 duration=1(sec)
May 17 20:03:59 client-18 xinetd[3390]: START: shell pid=8255 from=192.168.4.5
May 17 20:03:59 client-18 rshd[8256]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 20:03:59 client-18 xinetd[3390]: EXIT: shell status=0 pid=8255 duration=0(sec)
May 17 20:03:59 client-18 xinetd[3390]: START: shell pid=8294 from=192.168.4.5
May 17 20:03:59 client-18 rshd[8295]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mv /mnt/lustre2/d0.replay-vbr/d7/f7i-2 /mnt/lustre2/d0.replay-vbr/d7/f7i-0");echo XXRETCODE:$?'
May 17 20:03:59 client-18 xinetd[3390]: EXIT: shell status=0 pid=8294 duration=0(sec)
May 17 20:03:59 client-18 xinetd[3390]: START: shell pid=8329 from=192.168.4.5
May 17 20:03:59 client-18 rshd[8330]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "grep -c /mnt/lustre2' ' /proc/mounts");echo XXRETCODE:$?'
May 17 20:04:00 client-18 xinetd[3390]: EXIT: shell status=0 pid=8329 duration=1(sec)
May 17 20:04:00 client-18 xinetd[3390]: START: shell pid=8364 from=192.168.4.5
May 17 20:04:00 client-18 rshd[8365]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lsof -t /mnt/lustre2");echo XXRETCODE:$?'
May 17 20:04:00 client-18 xinetd[3390]: EXIT: shell status=0 pid=8364 duration=0(sec)
May 17 20:04:00 client-18 xinetd[3390]: START: shell pid=8400 from=192.168.4.5
May 17 20:04:00 client-18 rshd[8401]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "umount  /mnt/lustre2 2>&1");echo XXRETCODE:$?'
May 17 20:04:00 client-18 xinetd[3390]: EXIT: shell status=0 pid=8400 duration=0(sec)
May 17 20:04:18 client-18 kernel: Lustre: 8885:0:(import.c:885:ptlrpc_connect_interpret()) MGS@MGC192.168.4.128@o2ib_0 changed server handle from 0xfff98862bf65d5ed to 0xfff98862bf65dbff
May 17 20:06:25 client-18 xinetd[3390]: START: shell pid=8448 from=192.168.4.5
May 17 20:06:25 client-18 rshd[8449]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mkdir -p /mnt/lustre2");echo XXRETCODE:$?'
May 17 20:06:25 client-18 xinetd[3390]: START: shell pid=8483 from=192.168.4.5
May 17 20:06:25 client-18 xinetd[3390]: EXIT: shell status=0 pid=8448 duration=0(sec)
May 17 20:06:25 client-18 rshd[8484]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre2");echo XXRETCODE:$?'
May 17 20:06:25 client-18 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'.
May 17 20:06:25 client-18 kernel: LustreError: Skipped 4 previous similar messages
May 17 20:06:25 client-18 kernel: Lustre: 8518:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff8102f5a1a400->192.168.4.128@o2ib netid 50000: select flavor null
May 17 20:06:25 client-18 kernel: Lustre: 8518:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 34 previous similar messages
May 17 20:06:25 client-18 kernel: Lustre: Client lustre-client has started
May 17 20:06:25 client-18 kernel: Lustre: Skipped 4 previous similar messages
May 17 20:06:25 client-18 xinetd[3390]: EXIT: shell status=0 pid=8483 duration=0(sec)
May 17 20:06:25 client-18 xinetd[3390]: START: shell pid=8521 from=192.168.4.5
May 17 20:06:25 client-18 rshd[8522]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param debug=\"0x33f0406\";         lctl set_param subsystem_debug=\"0xffb7e3ff\";         lctl set_param debug_mb=32");echo XXRETCODE:$?'
May 17 20:06:25 client-18 xinetd[3390]: EXIT: shell status=0 pid=8521 duration=0(sec)
May 17 20:06:26 client-18 xinetd[3390]: START: shell pid=8558 from=192.168.4.5
May 17 20:06:26 client-18 rshd[8559]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 20:06:26 client-18 xinetd[3390]: EXIT: shell status=0 pid=8558 duration=0(sec)
May 17 20:06:26 client-18 xinetd[3390]: START: shell pid=8597 from=192.168.4.5
May 17 20:06:26 client-18 rshd[8598]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "rm /mnt/lustre2/d0.replay-vbr/d7/f7i-1");echo XXRETCODE:$?'
May 17 20:06:26 client-18 xinetd[3390]: EXIT: shell status=0 pid=8597 duration=0(sec)
May 17 20:06:26 client-18 xinetd[3390]: START: shell pid=8632 from=192.168.4.5
May 17 20:06:26 client-18 rshd[8633]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "grep -c /mnt/lustre2' ' /proc/mounts");echo XXRETCODE:$?'
May 17 20:06:26 client-18 xinetd[3390]: START: shell pid=8667 from=192.168.4.5
May 17 20:06:26 client-18 xinetd[3390]: EXIT: shell status=0 pid=8632 duration=0(sec)
May 17 20:06:26 client-18 rshd[8668]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lsof -t /mnt/lustre2");echo XXRETCODE:$?'
May 17 20:06:26 client-18 xinetd[3390]: EXIT: shell status=0 pid=8667 duration=0(sec)
May 17 20:06:26 client-18 xinetd[3390]: START: shell pid=8704 from=192.168.4.5
May 17 20:06:26 client-18 rshd[8705]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "umount  /mnt/lustre2 2>&1");echo XXRETCODE:$?'
May 17 20:06:27 client-18 xinetd[3390]: EXIT: shell status=0 pid=8704 duration=1(sec)
May 17 20:06:47 client-18 kernel: Lustre: 8886:0:(import.c:529:import_select_connection()) MGC192.168.4.128@o2ib: tried all connections, increasing latency to 6s
May 17 20:06:47 client-18 kernel: Lustre: 8886:0:(import.c:529:import_select_connection()) Skipped 1 previous similar message
May 17 20:06:47 client-18 kernel: Lustre: 8885:0:(import.c:885:ptlrpc_connect_interpret()) MGS@MGC192.168.4.128@o2ib_0 changed server handle from 0xfff98862bf65dbff to 0xfff98862bf65e265
May 17 20:07:08 client-18 kernel: Lustre: 8884:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369101282856777 sent from lustre-MDT0000-mdc-ffff8103028bc400 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305687995] [real_sent 1305687995] [current 1305688028] [deadline 33s] [delay 0s]  req@ffff8103120c3000 x1369101282856777/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 192/192 e 0 to 1 dl 1305688028 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
May 17 20:07:08 client-18 kernel: Lustre: 8884:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 11 previous similar messages
May 17 20:08:52 client-18 xinetd[3390]: START: shell pid=8752 from=192.168.4.5
May 17 20:08:52 client-18 rshd[8753]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 20:08:52 client-18 xinetd[3390]: EXIT: shell status=0 pid=8752 duration=0(sec)
May 17 20:08:52 client-18 xinetd[3390]: START: shell pid=8788 from=192.168.4.5
May 17 20:08:52 client-18 rshd[8789]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 20:08:52 client-18 xinetd[3390]: EXIT: shell status=0 pid=8788 duration=0(sec)
May 17 20:08:52 client-18 xinetd[3390]: START: shell pid=8830 from=192.168.4.5
May 17 20:08:52 client-18 rshd[8831]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-vbr test 8a: create \| unlink, create shouldn\'t fail ======================================== 20:08:52 \(1305688132\)");echo XXRETCODE:$?'
May 17 20:08:52 client-18 kernel: Lustre: DEBUG MARKER: == replay-vbr test 8a: create | unlink, create shouldn't fail ======================================== 20:08:52 (1305688132)
May 17 20:08:52 client-18 xinetd[3390]: EXIT: shell status=0 pid=8830 duration=0(sec)
May 17 20:08:52 client-18 xinetd[3390]: START: shell pid=8891 from=192.168.4.5
May 17 20:08:52 client-18 rshd[8892]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mkdir -p /mnt/lustre2");echo XXRETCODE:$?'
May 17 20:08:53 client-18 xinetd[3390]: START: shell pid=8926 from=192.168.4.5
May 17 20:08:53 client-18 xinetd[3390]: EXIT: shell status=0 pid=8891 duration=1(sec)
May 17 20:08:53 client-18 rshd[8927]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre2");echo XXRETCODE:$?'
May 17 20:08:53 client-18 xinetd[3390]: EXIT: shell status=0 pid=8926 duration=0(sec)
May 17 20:08:53 client-18 xinetd[3390]: START: shell pid=8965 from=192.168.4.5
May 17 20:08:53 client-18 rshd[8966]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param debug=\"0x33f0406\";         lctl set_param subsystem_debug=\"0xffb7e3ff\";         lctl set_param debug_mb=32");echo XXRETCODE:$?'
May 17 20:08:53 client-18 xinetd[3390]: EXIT: shell status=0 pid=8965 duration=0(sec)
May 17 20:08:53 client-18 xinetd[3390]: START: shell pid=9002 from=192.168.4.5
May 17 20:08:53 client-18 rshd[9003]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 20:08:53 client-18 xinetd[3390]: EXIT: shell status=0 pid=9002 duration=0(sec)
May 17 20:08:53 client-18 xinetd[3390]: START: shell pid=9041 from=192.168.4.5
May 17 20:08:54 client-18 rshd[9042]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "touch /mnt/lustre2/f8a-2/f8a");echo XXRETCODE:$?'
May 17 20:08:54 client-18 xinetd[3390]: EXIT: shell status=0 pid=9041 duration=1(sec)
May 17 20:08:54 client-18 xinetd[3390]: START: shell pid=9076 from=192.168.4.5
May 17 20:08:54 client-18 rshd[9077]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "grep -c /mnt/lustre2' ' /proc/mounts");echo XXRETCODE:$?'
May 17 20:08:54 client-18 xinetd[3390]: EXIT: shell status=0 pid=9076 duration=0(sec)
May 17 20:08:54 client-18 xinetd[3390]: START: shell pid=9111 from=192.168.4.5
May 17 20:08:54 client-18 rshd[9112]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lsof -t /mnt/lustre2");echo XXRETCODE:$?'
May 17 20:08:54 client-18 xinetd[3390]: EXIT: shell status=0 pid=9111 duration=0(sec)
May 17 20:08:54 client-18 xinetd[3390]: START: shell pid=9147 from=192.168.4.5
May 17 20:08:54 client-18 rshd[9148]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "umount  /mnt/lustre2 2>&1");echo XXRETCODE:$?'
May 17 20:08:54 client-18 xinetd[3390]: EXIT: shell status=0 pid=9147 duration=0(sec)
May 17 20:09:10 client-18 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail.
May 17 20:09:10 client-18 kernel: LustreError: Skipped 4 previous similar messages
May 17 20:09:15 client-18 kernel: Lustre: lustre-MDT0000-mdc-ffff8103028bc400: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete.
May 17 20:09:15 client-18 kernel: Lustre: Skipped 4 previous similar messages
May 17 20:09:25 client-18 kernel: Lustre: 8885:0:(import.c:885:ptlrpc_connect_interpret()) MGS@MGC192.168.4.128@o2ib_0 changed server handle from 0xfff98862bf65e265 to 0xfff98862bf65e766
May 17 20:10:44 client-18 xinetd[3390]: START: shell pid=9191 from=192.168.4.5
May 17 20:10:44 client-18 rshd[9192]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 20:10:44 client-18 xinetd[3390]: EXIT: shell status=0 pid=9191 duration=0(sec)
May 17 20:10:44 client-18 xinetd[3390]: START: shell pid=9227 from=192.168.4.5
May 17 20:10:44 client-18 rshd[9228]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 20:10:44 client-18 xinetd[3390]: EXIT: shell status=0 pid=9227 duration=0(sec)
May 17 20:10:44 client-18 xinetd[3390]: START: shell pid=9264 from=192.168.4.5
May 17 20:10:44 client-18 rshd[9265]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-vbr test 8b: create \| unlink, create shouldn\'t fail ======================================== 20:10:44 \(1305688244\)");echo XXRETCODE:$?'
May 17 20:10:44 client-18 kernel: Lustre: DEBUG MARKER: == replay-vbr test 8b: create | unlink, create shouldn't fail ======================================== 20:10:44 (1305688244)
May 17 20:10:44 client-18 xinetd[3390]: EXIT: shell status=0 pid=9264 duration=0(sec)
May 17 20:10:44 client-18 xinetd[3390]: START: shell pid=9299 from=192.168.4.5
May 17 20:10:44 client-18 rshd[9300]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mkdir -p /mnt/lustre2");echo XXRETCODE:$?'
May 17 20:10:45 client-18 xinetd[3390]: EXIT: shell status=0 pid=9299 duration=1(sec)
May 17 20:10:45 client-18 xinetd[3390]: START: shell pid=9334 from=192.168.4.5
May 17 20:10:45 client-18 rshd[9335]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre2");echo XXRETCODE:$?'
May 17 20:10:45 client-18 xinetd[3390]: START: shell pid=9373 from=192.168.4.5
May 17 20:10:45 client-18 xinetd[3390]: EXIT: shell status=0 pid=9334 duration=0(sec)
May 17 20:10:45 client-18 rshd[9374]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param debug=\"0x33f0406\";         lctl set_param subsystem_debug=\"0xffb7e3ff\";         lctl set_param debug_mb=32");echo XXRETCODE:$?'
May 17 20:10:45 client-18 xinetd[3390]: EXIT: shell status=0 pid=9373 duration=0(sec)
May 17 20:10:45 client-18 xinetd[3390]: START: shell pid=9410 from=192.168.4.5
May 17 20:10:45 client-18 rshd[9411]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 20:10:45 client-18 xinetd[3390]: EXIT: shell status=0 pid=9410 duration=0(sec)
May 17 20:10:46 client-18 xinetd[3390]: START: shell pid=9449 from=192.168.4.5
May 17 20:10:46 client-18 rshd[9450]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "touch /mnt/lustre2/f8b-2/f8b");echo XXRETCODE:$?'
May 17 20:10:46 client-18 xinetd[3390]: EXIT: shell status=0 pid=9449 duration=0(sec)
May 17 20:10:46 client-18 xinetd[3390]: START: shell pid=9484 from=192.168.4.5
May 17 20:10:46 client-18 rshd[9485]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "grep -c /mnt/lustre2' ' /proc/mounts");echo XXRETCODE:$?'
May 17 20:10:46 client-18 xinetd[3390]: EXIT: shell status=0 pid=9484 duration=0(sec)
May 17 20:10:46 client-18 xinetd[3390]: START: shell pid=9519 from=192.168.4.5
May 17 20:10:46 client-18 rshd[9520]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lsof -t /mnt/lustre2");echo XXRETCODE:$?'
May 17 20:10:46 client-18 xinetd[3390]: EXIT: shell status=0 pid=9519 duration=0(sec)
May 17 20:10:46 client-18 xinetd[3390]: START: shell pid=9555 from=192.168.4.5
May 17 20:10:46 client-18 rshd[9556]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "umount  /mnt/lustre2 2>&1");echo XXRETCODE:$?'
May 17 20:10:46 client-18 xinetd[3390]: EXIT: shell status=0 pid=9555 duration=0(sec)
May 17 20:11:05 client-18 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The obd_ping operation failed with -107
May 17 20:11:05 client-18 kernel: LustreError: Skipped 7 previous similar messages
May 17 20:11:05 client-18 kernel: Lustre: 8885:0:(import.c:885:ptlrpc_connect_interpret()) MGS@MGC192.168.4.128@o2ib_0 changed server handle from 0xfff98862bf65e766 to 0xfff98862bf65eb09
May 17 20:11:05 client-18 kernel: Lustre: MGC192.168.4.128@o2ib: Connection restored to service MGS using nid 192.168.4.128@o2ib.
May 17 20:11:05 client-18 kernel: Lustre: Skipped 8 previous similar messages
May 17 20:12:36 client-18 xinetd[3390]: START: shell pid=9606 from=192.168.4.5
May 17 20:12:36 client-18 rshd[9607]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 20:12:36 client-18 xinetd[3390]: EXIT: shell status=0 pid=9606 duration=0(sec)
May 17 20:12:36 client-18 xinetd[3390]: START: shell pid=9642 from=192.168.4.5
May 17 20:12:36 client-18 rshd[9643]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 20:12:36 client-18 xinetd[3390]: EXIT: shell status=0 pid=9642 duration=0(sec)
May 17 20:12:36 client-18 xinetd[3390]: START: shell pid=9679 from=192.168.4.5
May 17 20:12:36 client-18 rshd[9680]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-vbr test 8c: create \| unlink, create shouldn\'t fail ======================================== 20:12:36 \(1305688356\)");echo XXRETCODE:$?'
May 17 20:12:36 client-18 kernel: Lustre: DEBUG MARKER: == replay-vbr test 8c: create | unlink, create shouldn't fail ======================================== 20:12:36 (1305688356)
May 17 20:12:36 client-18 xinetd[3390]: EXIT: shell status=0 pid=9679 duration=0(sec)
May 17 20:12:36 client-18 xinetd[3390]: START: shell pid=9714 from=192.168.4.5
May 17 20:12:36 client-18 rshd[9715]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mkdir -p /mnt/lustre2");echo XXRETCODE:$?'
May 17 20:12:36 client-18 xinetd[3390]: EXIT: shell status=0 pid=9714 duration=0(sec)
May 17 20:12:36 client-18 xinetd[3390]: START: shell pid=9749 from=192.168.4.5
May 17 20:12:36 client-18 rshd[9750]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre2");echo XXRETCODE:$?'
May 17 20:12:36 client-18 xinetd[3390]: EXIT: shell status=0 pid=9749 duration=0(sec)
May 17 20:12:36 client-18 xinetd[3390]: START: shell pid=9787 from=192.168.4.5
May 17 20:12:36 client-18 rshd[9788]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param debug=\"0x33f0406\";         lctl set_param subsystem_debug=\"0xffb7e3ff\";         lctl set_param debug_mb=32");echo XXRETCODE:$?'
May 17 20:12:37 client-18 xinetd[3390]: EXIT: shell status=0 pid=9787 duration=1(sec)
May 17 20:12:37 client-18 xinetd[3390]: START: shell pid=9824 from=192.168.4.5
May 17 20:12:37 client-18 rshd[9825]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 20:12:37 client-18 xinetd[3390]: EXIT: shell status=0 pid=9824 duration=0(sec)
May 17 20:12:37 client-18 xinetd[3390]: START: shell pid=9863 from=192.168.4.5
May 17 20:12:37 client-18 rshd[9864]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "touch /mnt/lustre2/f8c-2/f8c");echo XXRETCODE:$?'
May 17 20:12:37 client-18 xinetd[3390]: EXIT: shell status=0 pid=9863 duration=0(sec)
May 17 20:12:37 client-18 xinetd[3390]: START: shell pid=9898 from=192.168.4.5
May 17 20:12:37 client-18 rshd[9899]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "grep -c /mnt/lustre2' ' /proc/mounts");echo XXRETCODE:$?'
May 17 20:12:37 client-18 xinetd[3390]: EXIT: shell status=0 pid=9898 duration=0(sec)
May 17 20:12:37 client-18 xinetd[3390]: START: shell pid=9933 from=192.168.4.5
May 17 20:12:38 client-18 rshd[9934]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lsof -t /mnt/lustre2");echo XXRETCODE:$?'
May 17 20:12:38 client-18 xinetd[3390]: EXIT: shell status=0 pid=9933 duration=1(sec)
May 17 20:12:38 client-18 xinetd[3390]: START: shell pid=9969 from=192.168.4.5
May 17 20:12:38 client-18 rshd[9970]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "umount  /mnt/lustre2 2>&1");echo XXRETCODE:$?'
May 17 20:12:38 client-18 kernel: LustreError: 10004:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway
May 17 20:12:38 client-18 kernel: LustreError: 10004:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Skipped 12 previous similar messages
May 17 20:12:38 client-18 kernel: LustreError: 10004:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108
May 17 20:12:38 client-18 kernel: LustreError: 10004:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) Skipped 12 previous similar messages
May 17 20:12:38 client-18 kernel: Lustre: client ffff81032619d400 umount complete
May 17 20:12:38 client-18 kernel: Lustre: Skipped 4 previous similar messages
May 17 20:12:38 client-18 xinetd[3390]: EXIT: shell status=0 pid=9969 duration=0(sec)
May 17 20:12:56 client-18 kernel: Lustre: 8885:0:(import.c:885:ptlrpc_connect_interpret()) MGS@MGC192.168.4.128@o2ib_0 changed server handle from 0xfff98862bf65eb09 to 0xfff98862bf65eedd
May 17 20:12:56 client-18 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import
May 17 20:12:56 client-18 kernel: Lustre: Skipped 4 previous similar messages
May 17 20:14:28 client-18 xinetd[3390]: START: shell pid=10013 from=192.168.4.5
May 17 20:14:28 client-18 rshd[10014]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 20:14:28 client-18 xinetd[3390]: EXIT: shell status=0 pid=10013 duration=0(sec)
May 17 20:14:28 client-18 xinetd[3390]: START: shell pid=10049 from=192.168.4.5
May 17 20:14:28 client-18 rshd[10050]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 20:14:28 client-18 xinetd[3390]: EXIT: shell status=0 pid=10049 duration=0(sec)
May 17 20:14:28 client-18 xinetd[3390]: START: shell pid=10086 from=192.168.4.5
May 17 20:14:28 client-18 rshd[10087]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-vbr test 10b: 3 clients: some, none, and all reqs replayed ================================= 20:14:28 \(1305688468\)");echo XXRETCODE:$?'
May 17 20:14:28 client-18 kernel: Lustre: DEBUG MARKER: == replay-vbr test 10b: 3 clients: some, none, and all reqs replayed ================================= 20:14:28 (1305688468)
May 17 20:14:28 client-18 xinetd[3390]: EXIT: shell status=0 pid=10086 duration=0(sec)
May 17 20:14:28 client-18 xinetd[3390]: START: shell pid=10121 from=192.168.4.5
May 17 20:14:28 client-18 rshd[10122]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mkdir -p /mnt/lustre");echo XXRETCODE:$?'
May 17 20:14:28 client-18 xinetd[3390]: START: shell pid=10156 from=192.168.4.5
May 17 20:14:28 client-18 xinetd[3390]: EXIT: shell status=0 pid=10121 duration=0(sec)
May 17 20:14:28 client-18 rshd[10157]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre");echo XXRETCODE:$?'
May 17 20:14:28 client-18 xinetd[3390]: START: shell pid=10192 from=192.168.4.5
May 17 20:14:28 client-18 xinetd[3390]: EXIT: shell status=0 pid=10156 duration=0(sec)
May 17 20:14:28 client-18 rshd[10193]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mkdir -p /mnt/lustre2");echo XXRETCODE:$?'
May 17 20:14:29 client-18 xinetd[3390]: START: shell pid=10227 from=192.168.4.5
May 17 20:14:29 client-18 xinetd[3390]: EXIT: shell status=0 pid=10192 duration=1(sec)
May 17 20:14:29 client-18 rshd[10228]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre2");echo XXRETCODE:$?'
May 17 20:14:29 client-18 xinetd[3390]: EXIT: shell status=0 pid=10227 duration=0(sec)
May 17 20:14:29 client-18 xinetd[3390]: START: shell pid=10265 from=192.168.4.5
May 17 20:14:29 client-18 rshd[10266]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param debug=\"0x33f0406\";         lctl set_param subsystem_debug=\"0xffb7e3ff\";         lctl set_param debug_mb=32");echo XXRETCODE:$?'
May 17 20:14:29 client-18 kernel: Lustre: 10300:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release.
May 17 20:14:29 client-18 kernel: Lustre: 10300:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 9 previous similar messages
May 17 20:14:29 client-18 xinetd[3390]: EXIT: shell status=0 pid=10265 duration=0(sec)
May 17 20:14:29 client-18 xinetd[3390]: START: shell pid=10302 from=192.168.4.5
May 17 20:14:29 client-18 rshd[10303]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 20:14:29 client-18 xinetd[3390]: EXIT: shell status=0 pid=10302 duration=0(sec)
May 17 20:14:30 client-18 xinetd[3390]: START: shell pid=10341 from=192.168.4.5
May 17 20:14:30 client-18 rshd[10342]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "chmod 666 /mnt/lustre/f10b-b");echo XXRETCODE:$?'
May 17 20:14:30 client-18 xinetd[3390]: EXIT: shell status=0 pid=10341 duration=0(sec)
May 17 20:14:30 client-18 xinetd[3390]: START: shell pid=10376 from=192.168.4.5
May 17 20:14:30 client-18 rshd[10377]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "chgrp 500 /mnt/lustre2/f10b-a");echo XXRETCODE:$?'
May 17 20:14:30 client-18 xinetd[3390]: EXIT: shell status=0 pid=10376 duration=0(sec)
May 17 20:14:30 client-18 xinetd[3390]: START: shell pid=10411 from=192.168.4.5
May 17 20:14:30 client-18 rshd[10412]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/lib64/lustre/tests/truncate /mnt/lustre2/f10b-b 1");echo XXRETCODE:$?'
May 17 20:14:30 client-18 xinetd[3390]: EXIT: shell status=0 pid=10411 duration=0(sec)
May 17 20:14:30 client-18 xinetd[3390]: START: shell pid=10446 from=192.168.4.5
May 17 20:14:30 client-18 rshd[10447]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "chgrp 500 /mnt/lustre/f10b-b");echo XXRETCODE:$?'
May 17 20:14:30 client-18 xinetd[3390]: EXIT: shell status=0 pid=10446 duration=0(sec)
May 17 20:14:30 client-18 xinetd[3390]: START: shell pid=10481 from=192.168.4.5
May 17 20:14:30 client-18 rshd[10482]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "grep -c /mnt/lustre2' ' /proc/mounts");echo XXRETCODE:$?'
May 17 20:14:30 client-18 xinetd[3390]: EXIT: shell status=0 pid=10481 duration=0(sec)
May 17 20:14:30 client-18 xinetd[3390]: START: shell pid=10516 from=192.168.4.5
May 17 20:14:30 client-18 rshd[10517]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lsof -t /mnt/lustre2");echo XXRETCODE:$?'
May 17 20:14:30 client-18 xinetd[3390]: EXIT: shell status=0 pid=10516 duration=0(sec)
May 17 20:14:30 client-18 xinetd[3390]: START: shell pid=10552 from=192.168.4.5
May 17 20:14:30 client-18 rshd[10553]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "umount  /mnt/lustre2 2>&1");echo XXRETCODE:$?'
May 17 20:14:31 client-18 xinetd[3390]: EXIT: shell status=0 pid=10552 duration=1(sec)
May 17 20:14:49 client-18 kernel: Lustre: 8885:0:(import.c:885:ptlrpc_connect_interpret()) MGS@MGC192.168.4.128@o2ib_0 changed server handle from 0xfff98862bf65eedd to 0xfff98862bf65f37c
May 17 20:16:54 client-18 xinetd[3390]: START: shell pid=10600 from=192.168.4.5
May 17 20:16:54 client-18 rshd[10601]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 20:16:54 client-18 xinetd[3390]: EXIT: shell status=0 pid=10600 duration=0(sec)
May 17 20:16:54 client-18 xinetd[3390]: START: shell pid=10634 from=192.168.4.5
May 17 20:16:54 client-18 rshd[10635]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "chmod 666 /mnt/lustre/f10b");echo XXRETCODE:$?'
May 17 20:16:54 client-18 xinetd[3390]: START: shell pid=10669 from=192.168.4.5
May 17 20:16:54 client-18 xinetd[3390]: EXIT: shell status=0 pid=10634 duration=0(sec)
May 17 20:16:54 client-18 rshd[10670]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/bin/lfs path2fid /mnt/lustre/f10b");echo XXRETCODE:$?'
May 17 20:16:54 client-18 xinetd[3390]: EXIT: shell status=0 pid=10669 duration=0(sec)
May 17 20:16:54 client-18 xinetd[3390]: START: shell pid=10704 from=192.168.4.5
May 17 20:16:54 client-18 rshd[10705]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "stat /mnt/lustre/f10b-a");echo XXRETCODE:$?'
May 17 20:16:54 client-18 xinetd[3390]: EXIT: shell status=0 pid=10704 duration=0(sec)
May 17 20:16:54 client-18 xinetd[3390]: START: shell pid=10739 from=192.168.4.5
May 17 20:16:54 client-18 rshd[10740]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "stat /mnt/lustre/f10b-b");echo XXRETCODE:$?'
May 17 20:16:55 client-18 xinetd[3390]: START: shell pid=10774 from=192.168.4.5
May 17 20:16:55 client-18 xinetd[3390]: EXIT: shell status=0 pid=10739 duration=1(sec)
May 17 20:16:55 client-18 rshd[10775]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "checkstat -v -p 0666 -u \#0 -g \#0 /mnt/lustre/f10b-a");echo XXRETCODE:$?'
May 17 20:16:55 client-18 xinetd[3390]: START: shell pid=10809 from=192.168.4.5
May 17 20:16:55 client-18 xinetd[3390]: EXIT: shell status=0 pid=10774 duration=0(sec)
May 17 20:16:55 client-18 rshd[10810]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "checkstat -v -p 0666 -u \#500 -g \#500 /mnt/lustre/f10b-b");echo XXRETCODE:$?'
May 17 20:16:55 client-18 xinetd[3390]: EXIT: shell status=0 pid=10809 duration=0(sec)
May 17 20:16:55 client-18 xinetd[3390]: START: shell pid=10844 from=192.168.4.5
May 17 20:16:55 client-18 rshd[10845]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "grep -c /mnt/lustre' ' /proc/mounts");echo XXRETCODE:$?'
May 17 20:16:55 client-18 xinetd[3390]: EXIT: shell status=0 pid=10844 duration=0(sec)
May 17 20:16:55 client-18 xinetd[3390]: START: shell pid=10879 from=192.168.4.5
May 17 20:16:55 client-18 rshd[10880]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lsof -t /mnt/lustre");echo XXRETCODE:$?'
May 17 20:16:55 client-18 xinetd[3390]: EXIT: shell status=0 pid=10879 duration=0(sec)
May 17 20:16:55 client-18 xinetd[3390]: START: shell pid=10915 from=192.168.4.5
May 17 20:16:55 client-18 rshd[10916]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "umount  /mnt/lustre 2>&1");echo XXRETCODE:$?'
May 17 20:16:55 client-18 xinetd[3390]: EXIT: shell status=0 pid=10915 duration=0(sec)
May 17 20:16:55 client-18 xinetd[3390]: START: shell pid=10951 from=192.168.4.5
May 17 20:16:55 client-18 rshd[10952]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 20:16:55 client-18 xinetd[3390]: EXIT: shell status=0 pid=10951 duration=0(sec)
May 17 20:16:55 client-18 xinetd[3390]: START: shell pid=10987 from=192.168.4.5
May 17 20:16:56 client-18 rshd[10988]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 20:16:56 client-18 xinetd[3390]: EXIT: shell status=0 pid=10987 duration=1(sec)
May 17 20:16:56 client-18 xinetd[3390]: START: shell pid=11024 from=192.168.4.5
May 17 20:16:56 client-18 rshd[11025]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-vbr test 11a: concurrent creates don\'t affect each other =================================== 20:16:56 \(1305688616\)");echo XXRETCODE:$?'
May 17 20:16:56 client-18 kernel: Lustre: DEBUG MARKER: == replay-vbr test 11a: concurrent creates don't affect each other =================================== 20:16:56 (1305688616)
May 17 20:16:56 client-18 xinetd[3390]: EXIT: shell status=0 pid=11024 duration=0(sec)
May 17 20:16:56 client-18 xinetd[3390]: START: shell pid=11059 from=192.168.4.5
May 17 20:16:56 client-18 rshd[11060]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mkdir -p /mnt/lustre2");echo XXRETCODE:$?'
May 17 20:16:56 client-18 xinetd[3390]: EXIT: shell status=0 pid=11059 duration=0(sec)
May 17 20:16:56 client-18 xinetd[3390]: START: shell pid=11094 from=192.168.4.5
May 17 20:16:56 client-18 rshd[11095]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre2");echo XXRETCODE:$?'
May 17 20:16:56 client-18 kernel: Lustre: 11129:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null
May 17 20:16:56 client-18 kernel: Lustre: 11129:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 34 previous similar messages
May 17 20:16:56 client-18 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'.
May 17 20:16:56 client-18 kernel: LustreError: Skipped 4 previous similar messages
May 17 20:16:56 client-18 kernel: Lustre: Client lustre-client has started
May 17 20:16:56 client-18 kernel: Lustre: Skipped 4 previous similar messages
May 17 20:16:56 client-18 xinetd[3390]: EXIT: shell status=0 pid=11094 duration=0(sec)
May 17 20:16:56 client-18 xinetd[3390]: START: shell pid=11142 from=192.168.4.5
May 17 20:16:56 client-18 rshd[11143]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param debug=\"0x33f0406\";         lctl set_param subsystem_debug=\"0xffb7e3ff\";         lctl set_param debug_mb=32");echo XXRETCODE:$?'
May 17 20:16:56 client-18 xinetd[3390]: EXIT: shell status=0 pid=11142 duration=0(sec)
May 17 20:16:56 client-18 xinetd[3390]: START: shell pid=11179 from=192.168.4.5
May 17 20:16:56 client-18 rshd[11180]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 20:16:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=11179 duration=1(sec)
May 17 20:16:57 client-18 xinetd[3390]: START: shell pid=11218 from=192.168.4.5
May 17 20:16:57 client-18 rshd[11219]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "createmany -o /mnt/lustre2/f11a-2- 100");echo XXRETCODE:$?'
May 17 20:16:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=11218 duration=0(sec)
May 17 20:16:57 client-18 xinetd[3390]: START: shell pid=11253 from=192.168.4.5
May 17 20:16:57 client-18 rshd[11254]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "grep -c /mnt/lustre2' ' /proc/mounts");echo XXRETCODE:$?'
May 17 20:16:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=11253 duration=0(sec)
May 17 20:16:57 client-18 xinetd[3390]: START: shell pid=11288 from=192.168.4.5
May 17 20:16:57 client-18 rshd[11289]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lsof -t /mnt/lustre2");echo XXRETCODE:$?'
May 17 20:16:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=11288 duration=0(sec)
May 17 20:16:57 client-18 xinetd[3390]: START: shell pid=11324 from=192.168.4.5
May 17 20:16:57 client-18 rshd[11325]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "umount  /mnt/lustre2 2>&1");echo XXRETCODE:$?'
May 17 20:16:58 client-18 xinetd[3390]: EXIT: shell status=0 pid=11324 duration=1(sec)
May 17 20:18:47 client-18 xinetd[3390]: START: shell pid=11368 from=192.168.4.5
May 17 20:18:47 client-18 rshd[11369]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 20:18:47 client-18 xinetd[3390]: EXIT: shell status=0 pid=11368 duration=0(sec)
May 17 20:18:47 client-18 xinetd[3390]: START: shell pid=11404 from=192.168.4.5
May 17 20:18:48 client-18 rshd[11405]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 20:18:48 client-18 xinetd[3390]: EXIT: shell status=0 pid=11404 duration=1(sec)
May 17 20:18:48 client-18 xinetd[3390]: START: shell pid=11441 from=192.168.4.5
May 17 20:18:48 client-18 rshd[11442]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-vbr test 11b: concurrent creates and unlinks don\'t affect each other ======================= 20:18:48 \(1305688728\)");echo XXRETCODE:$?'
May 17 20:18:48 client-18 kernel: Lustre: DEBUG MARKER: == replay-vbr test 11b: concurrent creates and unlinks don't affect each other ======================= 20:18:48 (1305688728)
May 17 20:18:48 client-18 xinetd[3390]: EXIT: shell status=0 pid=11441 duration=0(sec)
May 17 20:18:48 client-18 xinetd[3390]: START: shell pid=11476 from=192.168.4.5
May 17 20:18:48 client-18 rshd[11477]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mkdir -p /mnt/lustre2");echo XXRETCODE:$?'
May 17 20:18:48 client-18 xinetd[3390]: EXIT: shell status=0 pid=11476 duration=0(sec)
May 17 20:18:48 client-18 xinetd[3390]: START: shell pid=11511 from=192.168.4.5
May 17 20:18:48 client-18 rshd[11512]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre2");echo XXRETCODE:$?'
May 17 20:18:48 client-18 xinetd[3390]: EXIT: shell status=0 pid=11511 duration=0(sec)
May 17 20:18:48 client-18 xinetd[3390]: START: shell pid=11559 from=192.168.4.5
May 17 20:18:48 client-18 rshd[11560]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param debug=\"0x33f0406\";         lctl set_param subsystem_debug=\"0xffb7e3ff\";         lctl set_param debug_mb=32");echo XXRETCODE:$?'
May 17 20:18:48 client-18 xinetd[3390]: EXIT: shell status=0 pid=11559 duration=0(sec)
May 17 20:18:48 client-18 xinetd[3390]: START: shell pid=11596 from=192.168.4.5
May 17 20:18:48 client-18 rshd[11597]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "createmany -o /mnt/lustre2/f11b-2- 100");echo XXRETCODE:$?'
May 17 20:18:48 client-18 xinetd[3390]: EXIT: shell status=0 pid=11596 duration=0(sec)
May 17 20:18:49 client-18 xinetd[3390]: START: shell pid=11631 from=192.168.4.5
May 17 20:18:49 client-18 rshd[11632]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?'
May 17 20:18:49 client-18 xinetd[3390]: EXIT: shell status=0 pid=11631 duration=0(sec)
May 17 20:18:49 client-18 xinetd[3390]: START: shell pid=11670 from=192.168.4.5
May 17 20:18:49 client-18 rshd[11671]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "unlinkmany -o /mnt/lustre2/f11b-2- 100");echo XXRETCODE:$?'
May 17 20:18:49 client-18 xinetd[3390]: EXIT: shell status=0 pid=11670 duration=0(sec)
May 17 20:18:49 client-18 xinetd[3390]: START: shell pid=11705 from=192.168.4.5
May 17 20:18:49 client-18 rshd[11706]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "grep -c /mnt/lustre2' ' /proc/mounts");echo XXRETCODE:$?'
May 17 20:18:49 client-18 xinetd[3390]: START: shell pid=11740 from=192.168.4.5
May 17 20:18:49 client-18 xinetd[3390]: EXIT: shell status=0 pid=11705 duration=0(sec)
May 17 20:18:49 client-18 rshd[11741]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lsof -t /mnt/lustre2");echo XXRETCODE:$?'
May 17 20:18:49 client-18 xinetd[3390]: EXIT: shell status=0 pid=11740 duration=0(sec)
May 17 20:18:49 client-18 xinetd[3390]: START: shell pid=11776 from=192.168.4.5
May 17 20:18:49 client-18 rshd[11777]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "umount  /mnt/lustre2 2>&1");echo XXRETCODE:$?'
May 17 20:18:50 client-18 xinetd[3390]: EXIT: shell status=0 pid=11776 duration=1(sec)
May 17 20:20:39 client-18 xinetd[3390]: START: shell pid=11816 from=192.168.4.5
May 17 20:20:39 client-18 rshd[11817]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 20:20:39 client-18 xinetd[3390]: EXIT: shell status=0 pid=11816 duration=0(sec)
May 17 20:20:39 client-18 xinetd[3390]: START: shell pid=11852 from=192.168.4.5
May 17 20:20:39 client-18 rshd[11853]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 20:20:40 client-18 xinetd[3390]: EXIT: shell status=0 pid=11852 duration=1(sec)
May 17 20:20:40 client-18 xinetd[3390]: START: shell pid=11889 from=192.168.4.5
May 17 20:20:40 client-18 rshd[11890]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark  SKIP: replay-vbr test_12a skipping ALWAYS excluded test 12a");echo XXRETCODE:$?'
May 17 20:20:40 client-18 kernel: Lustre: DEBUG MARKER: SKIP: replay-vbr test_12a skipping ALWAYS excluded test 12a
May 17 20:20:40 client-18 xinetd[3390]: EXIT: shell status=0 pid=11889 duration=0(sec)
May 17 20:20:40 client-18 xinetd[3390]: START: shell pid=11924 from=192.168.4.5
May 17 20:20:40 client-18 rshd[11925]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c " running=\$(mount | grep -c /mnt/lustre' '); rc=0; if [ \$running -eq 0 ] ; then     mkdir -p /mnt/lustre;     mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre;     rc=\$?; fi; exit \$rc");echo XXRETCODE:$?'
May 17 20:20:40 client-18 xinetd[3390]: EXIT: shell status=0 pid=11924 duration=0(sec)
May 17 20:20:40 client-18 xinetd[3390]: START: shell pid=11977 from=192.168.4.5
May 17 20:20:40 client-18 rshd[11978]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "mount | grep -w /mnt/lustre");echo XXRETCODE:$?'
May 17 20:20:40 client-18 xinetd[3390]: EXIT: shell status=0 pid=11977 duration=0(sec)
May 17 20:20:40 client-18 xinetd[3390]: START: shell pid=12014 from=192.168.4.5
May 17 20:20:40 client-18 rshd[12015]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param debug=\"0x33f0406\";         lctl set_param subsystem_debug=\"0xffb7e3ff\";         lctl set_param debug_mb=32;");echo XXRETCODE:$?'
May 17 20:20:40 client-18 xinetd[3390]: EXIT: shell status=0 pid=12014 duration=0(sec)
May 17 20:20:40 client-18 xinetd[3390]: START: shell pid=12051 from=192.168.4.5
May 17 20:20:40 client-18 rshd[12052]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == replay-vbr replay-vbr.sh test complete, duration 7137 sec ========================================= 20:20:40 \(1305688840\)");echo XXRETCODE:$?'
May 17 20:20:40 client-18 kernel: Lustre: DEBUG MARKER: == replay-vbr replay-vbr.sh test complete, duration 7137 sec ========================================= 20:20:40 (1305688840)
May 17 20:20:40 client-18 xinetd[3390]: EXIT: shell status=0 pid=12051 duration=0(sec)
May 17 20:20:52 client-18 kernel: Lustre: 8884:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369101282858375 sent from MGC192.168.4.128@o2ib to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305688845] [real_sent 1305688845] [current 1305688852] [deadline 7s] [delay 0s]  req@ffff810303331000 x1369101282858375/t0(0) o-1->MGS@MGC192.168.4.128@o2ib_0:26/25 lens 192/192 e 0 to 1 dl 1305688852 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
May 17 20:20:52 client-18 kernel: Lustre: 8884:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 11 previous similar messages
May 17 20:20:52 client-18 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail.
May 17 20:20:52 client-18 kernel: LustreError: Skipped 3 previous similar messages
May 17 20:20:59 client-18 xinetd[3390]: START: shell pid=12086 from=192.168.4.5
May 17 20:20:59 client-18 rshd[12087]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 20:20:59 client-18 kernel: Lustre: lustre-MDT0000-mdc-ffff81031cc35400: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete.
May 17 20:20:59 client-18 kernel: Lustre: Skipped 3 previous similar messages
May 17 20:21:02 client-18 kernel: Lustre: 8886:0:(import.c:529:import_select_connection()) MGC192.168.4.128@o2ib: tried all connections, increasing latency to 6s
May 17 20:21:02 client-18 kernel: Lustre: 8886:0:(import.c:529:import_select_connection()) Skipped 1 previous similar message
May 17 20:21:02 client-18 kernel: Lustre: 8885:0:(import.c:885:ptlrpc_connect_interpret()) MGS@MGC192.168.4.128@o2ib_0 changed server handle from 0xfff98862bf6661e6 to 0xfff98862bf667e5d
May 17 20:21:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=12086 duration=4(sec)
May 17 20:21:03 client-18 xinetd[3390]: START: shell pid=12122 from=192.168.4.5
May 17 20:21:03 client-18 rshd[12123]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:-f; lsof -t /mnt/lustre || need_kill=no; if [ x-f != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount -f /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre is still busy, wait one second && sleep 1;     umount -f /mnt/lustre; fi fi");echo XXRETCODE:$?'
May 17 20:21:03 client-18 kernel: Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
May 17 20:21:03 client-18 kernel: Lustre: Skipped 6 previous similar messages
May 17 20:21:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=12122 duration=0(sec)
May 17 20:21:03 client-18 xinetd[3390]: START: shell pid=12166 from=192.168.4.5
May 17 20:21:03 client-18 rshd[12167]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:-f; lsof -t /mnt/lustre2 || need_kill=no; if [ x-f != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre2 | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount -f /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre2 is still busy, wait one second && sleep 1;     umount -f /mnt/lustre2; fi fi");echo XXRETCODE:$?'
May 17 20:21:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=12166 duration=0(sec)
May 17 20:21:36 client-18 xinetd[3390]: START: shell pid=12207 from=192.168.4.5
May 17 20:21:36 client-18 rshd[12208]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark -----============= acceptance-small: insanity ============----- Tue May 17 20:21:35 PDT 2011");echo XXRETCODE:$?'
May 17 20:21:36 client-18 kernel: Lustre: DEBUG MARKER: -----============= acceptance-small: insanity ============----- Tue May 17 20:21:35 PDT 2011
May 17 20:21:36 client-18 xinetd[3390]: EXIT: shell status=0 pid=12207 duration=0(sec)
May 17 20:21:36 client-18 xinetd[3390]: START: shell pid=12242 from=192.168.4.5
May 17 20:21:36 client-18 rshd[12243]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark excepting tests: 10");echo XXRETCODE:$?'
May 17 20:21:36 client-18 kernel: Lustre: DEBUG MARKER: excepting tests: 10
May 17 20:21:36 client-18 xinetd[3390]: EXIT: shell status=0 pid=12242 duration=0(sec)
May 17 20:21:36 client-18 xinetd[3390]: START: shell pid=12277 from=192.168.4.5
May 17 20:21:36 client-18 rshd[12278]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:; lsof -t /mnt/lustre || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount  /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre is still busy, wait one second && sleep 1;     umount  /mnt/lustre; fi fi");echo XXRETCODE:$?'
May 17 20:21:36 client-18 xinetd[3390]: EXIT: shell status=0 pid=12277 duration=0(sec)
May 17 20:21:36 client-18 xinetd[3390]: START: shell pid=12314 from=192.168.4.5
May 17 20:21:36 client-18 rshd[12315]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:; lsof -t /mnt/lustre2 || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre2 | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount  /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre2 is still busy, wait one second && sleep 1;     umount  /mnt/lustre2; fi fi");echo XXRETCODE:$?'
May 17 20:21:36 client-18 xinetd[3390]: EXIT: shell status=0 pid=12314 duration=0(sec)
May 17 20:22:36 client-18 xinetd[3390]: START: shell pid=12355 from=192.168.4.5
May 17 20:22:36 client-18 rshd[12356]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); mpts=\$(mount | grep -w -c /mnt/lustre); if [ \$running -ne \$mpts ]; then     echo \$(hostname) env are INSANE!;     exit 1; fi");echo XXRETCODE:$?'
May 17 20:22:36 client-18 xinetd[3390]: EXIT: shell status=0 pid=12355 duration=0(sec)
May 17 20:22:36 client-18 xinetd[3390]: START: shell pid=12395 from=192.168.4.5
May 17 20:22:36 client-18 rshd[12396]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); mpts=\$(mount | grep -w -c /mnt/lustre2); if [ \$running -ne \$mpts ]; then     echo \$(hostname) env are INSANE!;     exit 1; fi");echo XXRETCODE:$?'
May 17 20:22:36 client-18 xinetd[3390]: EXIT: shell status=0 pid=12395 duration=0(sec)
May 17 20:22:45 client-18 xinetd[3390]: START: shell pid=12435 from=192.168.4.5
May 17 20:22:45 client-18 rshd[12436]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c " running=\$(mount | grep -c /mnt/lustre' '); rc=0; if [ \$running -eq 0 ] ; then     mkdir -p /mnt/lustre;     mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre;     rc=\$?; fi; exit \$rc");echo XXRETCODE:$?'
May 17 20:22:45 client-18 xinetd[3390]: EXIT: shell status=0 pid=12435 duration=0(sec)
May 17 20:22:45 client-18 xinetd[3390]: START: shell pid=12488 from=192.168.4.5
May 17 20:22:45 client-18 rshd[12489]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "mount | grep -w /mnt/lustre");echo XXRETCODE:$?'
May 17 20:22:46 client-18 xinetd[3390]: EXIT: shell status=0 pid=12488 duration=1(sec)
May 17 20:22:46 client-18 xinetd[3390]: START: shell pid=12525 from=192.168.4.5
May 17 20:22:46 client-18 rshd[12526]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param debug=\"0x33f0406\";         lctl set_param subsystem_debug=\"0xffb7e3ff\";         lctl set_param debug_mb=32;");echo XXRETCODE:$?'
May 17 20:22:46 client-18 xinetd[3390]: EXIT: shell status=0 pid=12525 duration=0(sec)
May 17 20:22:46 client-18 xinetd[3390]: START: shell pid=12562 from=192.168.4.5
May 17 20:22:46 client-18 rshd[12563]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark Using TIMEOUT=20");echo XXRETCODE:$?'
May 17 20:22:46 client-18 kernel: Lustre: DEBUG MARKER: Using TIMEOUT=20
May 17 20:22:46 client-18 xinetd[3390]: EXIT: shell status=0 pid=12562 duration=0(sec)
May 17 20:22:51 client-18 xinetd[3390]: START: shell pid=12597 from=192.168.4.5
May 17 20:22:51 client-18 rshd[12598]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "cat /proc/mounts");echo XXRETCODE:$?'
May 17 20:22:51 client-18 xinetd[3390]: EXIT: shell status=0 pid=12597 duration=0(sec)
May 17 20:22:52 client-18 xinetd[3390]: START: shell pid=12632 from=192.168.4.5
May 17 20:22:52 client-18 rshd[12633]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl get_param -n osc.*OST*-osc-[^M][^D][^T]*.srpc_info");echo XXRETCODE:$?'
May 17 20:22:52 client-18 xinetd[3390]: EXIT: shell status=0 pid=12632 duration=0(sec)
May 17 20:22:52 client-18 xinetd[3390]: START: shell pid=12667 from=192.168.4.5
May 17 20:22:52 client-18 rshd[12668]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl get_param -n mdc.*-MDT*-mdc-*.srpc_info");echo XXRETCODE:$?'
May 17 20:22:52 client-18 xinetd[3390]: EXIT: shell status=0 pid=12667 duration=0(sec)
May 17 20:22:52 client-18 xinetd[3390]: START: shell pid=12702 from=192.168.4.5
May 17 20:22:52 client-18 rshd[12703]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == insanity test 0: Fail all nodes, independently ==================================================== 20:22:52 \(1305688972\)");echo XXRETCODE:$?'
May 17 20:22:53 client-18 kernel: Lustre: DEBUG MARKER: == insanity test 0: Fail all nodes, independently ==================================================== 20:22:52 (1305688972)
May 17 20:22:53 client-18 xinetd[3390]: EXIT: shell status=0 pid=12702 duration=1(sec)
May 17 20:23:11 client-18 xinetd[3390]: START: shell pid=12737 from=192.168.4.5
May 17 20:23:11 client-18 rshd[12738]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 20:23:12 client-18 kernel: Lustre: 8885:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf668078 to 0xfff98862bf6681e4
May 17 20:23:12 client-18 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import
May 17 20:23:12 client-18 kernel: Lustre: Skipped 6 previous similar messages
May 17 20:23:12 client-18 kernel: Lustre: MGC192.168.4.128@o2ib: Connection restored to service MGS using nid 192.168.4.128@o2ib.
May 17 20:23:12 client-18 kernel: Lustre: Skipped 7 previous similar messages
May 17 20:23:17 client-18 xinetd[3390]: EXIT: shell status=0 pid=12737 duration=6(sec)
May 17 20:23:22 client-18 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.129@o2ib. The obd_ping operation failed with -107
May 17 20:23:22 client-18 kernel: LustreError: Skipped 6 previous similar messages
May 17 20:23:31 client-18 xinetd[3390]: START: shell pid=12775 from=192.168.4.5
May 17 20:23:31 client-18 rshd[12776]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 20:23:37 client-18 xinetd[3390]: EXIT: shell status=0 pid=12775 duration=6(sec)
May 17 20:23:51 client-18 xinetd[3390]: START: shell pid=12813 from=192.168.4.5
May 17 20:23:51 client-18 rshd[12814]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 20:23:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=12813 duration=6(sec)
May 17 20:24:11 client-18 xinetd[3390]: START: shell pid=12847 from=192.168.4.5
May 17 20:24:11 client-18 rshd[12848]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 20:24:17 client-18 xinetd[3390]: EXIT: shell status=0 pid=12847 duration=6(sec)
May 17 20:24:34 client-18 xinetd[3390]: START: shell pid=12881 from=192.168.4.5
May 17 20:24:34 client-18 rshd[12882]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 20:24:37 client-18 xinetd[3390]: EXIT: shell status=0 pid=12881 duration=3(sec)
May 17 20:24:51 client-18 xinetd[3390]: START: shell pid=12915 from=192.168.4.5
May 17 20:24:51 client-18 rshd[12916]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 20:24:55 client-18 xinetd[3390]: EXIT: shell status=0 pid=12915 duration=4(sec)
May 17 20:25:09 client-18 xinetd[3390]: START: shell pid=12953 from=192.168.4.5
May 17 20:25:09 client-18 rshd[12954]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 20:25:15 client-18 xinetd[3390]: EXIT: shell status=0 pid=12953 duration=6(sec)
May 17 20:25:15 client-18 xinetd[3390]: START: shell pid=12987 from=192.168.4.5
May 17 20:25:15 client-18 rshd[12988]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 20:25:15 client-18 xinetd[3390]: EXIT: shell status=0 pid=12987 duration=0(sec)
May 17 20:25:15 client-18 xinetd[3390]: START: shell pid=13023 from=192.168.4.5
May 17 20:25:15 client-18 rshd[13024]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 20:25:15 client-18 xinetd[3390]: EXIT: shell status=0 pid=13023 duration=0(sec)
May 17 20:25:15 client-18 xinetd[3390]: START: shell pid=13060 from=192.168.4.5
May 17 20:25:15 client-18 rshd[13061]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == insanity test 1: MDS\/MDS failure ================================================================== 20:25:15 \(1305689115\)");echo XXRETCODE:$?'
May 17 20:25:15 client-18 kernel: Lustre: DEBUG MARKER: == insanity test 1: MDS/MDS failure ================================================================== 20:25:15 (1305689115)
May 17 20:25:15 client-18 xinetd[3390]: EXIT: shell status=0 pid=13060 duration=0(sec)
May 17 20:25:15 client-18 xinetd[3390]: START: shell pid=13095 from=192.168.4.5
May 17 20:25:15 client-18 rshd[13096]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 20:25:15 client-18 xinetd[3390]: EXIT: shell status=0 pid=13095 duration=0(sec)
May 17 20:25:15 client-18 xinetd[3390]: START: shell pid=13131 from=192.168.4.5
May 17 20:25:15 client-18 rshd[13132]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 20:25:15 client-18 xinetd[3390]: EXIT: shell status=0 pid=13131 duration=0(sec)
May 17 20:25:16 client-18 xinetd[3390]: START: shell pid=13168 from=192.168.4.5
May 17 20:25:16 client-18 rshd[13169]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == insanity test 2: Second Failure Mode: MDS\/OST Tue May 17 20:25:15 PDT 2011 ======================== 20:25:15 \(1305689115\)");echo XXRETCODE:$?'
May 17 20:25:16 client-18 kernel: Lustre: DEBUG MARKER: == insanity test 2: Second Failure Mode: MDS/OST Tue May 17 20:25:15 PDT 2011 ======================== 20:25:15 (1305689115)
May 17 20:25:16 client-18 xinetd[3390]: EXIT: shell status=0 pid=13168 duration=0(sec)
May 17 20:25:17 client-18 xinetd[3390]: START: shell pid=13203 from=192.168.4.5
May 17 20:25:17 client-18 rshd[13204]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 20:25:17 client-18 xinetd[3390]: EXIT: shell status=0 pid=13203 duration=0(sec)
May 17 20:25:45 client-18 xinetd[3390]: START: shell pid=13237 from=192.168.4.5
May 17 20:25:45 client-18 rshd[13238]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 20:26:53 client-18 xinetd[3390]: EXIT: shell status=0 pid=13237 duration=68(sec)
May 17 20:26:54 client-18 xinetd[3390]: START: shell pid=13275 from=192.168.4.5
May 17 20:26:54 client-18 rshd[13276]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 20:26:55 client-18 xinetd[3390]: EXIT: shell status=0 pid=13275 duration=1(sec)
May 17 20:26:55 client-18 xinetd[3390]: START: shell pid=13309 from=192.168.4.5
May 17 20:26:55 client-18 rshd[13310]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 20:26:55 client-18 xinetd[3390]: EXIT: shell status=0 pid=13309 duration=0(sec)
May 17 20:26:55 client-18 xinetd[3390]: START: shell pid=13345 from=192.168.4.5
May 17 20:26:55 client-18 rshd[13346]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 20:26:55 client-18 xinetd[3390]: EXIT: shell status=0 pid=13345 duration=0(sec)
May 17 20:26:55 client-18 xinetd[3390]: START: shell pid=13382 from=192.168.4.5
May 17 20:26:55 client-18 rshd[13383]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == insanity test 3: Thirdb Failure Mode: MDS\/CLIENT Tue May 17 20:26:55 PDT 2011 ===================== 20:26:55 \(1305689215\)");echo XXRETCODE:$?'
May 17 20:26:55 client-18 kernel: Lustre: DEBUG MARKER: == insanity test 3: Thirdb Failure Mode: MDS/CLIENT Tue May 17 20:26:55 PDT 2011 ===================== 20:26:55 (1305689215)
May 17 20:26:55 client-18 xinetd[3390]: EXIT: shell status=0 pid=13382 duration=0(sec)
May 17 20:27:13 client-18 xinetd[3390]: START: shell pid=13417 from=192.168.4.5
May 17 20:27:13 client-18 rshd[13418]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 20:27:13 client-18 xinetd[3390]: EXIT: shell status=0 pid=13417 duration=0(sec)
May 17 20:27:14 client-18 xinetd[3390]: START: shell pid=13451 from=192.168.4.5
May 17 20:27:15 client-18 rshd[13452]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 20:27:15 client-18 xinetd[3390]: EXIT: shell status=0 pid=13451 duration=1(sec)
May 17 20:27:15 client-18 xinetd[3390]: START: shell pid=13485 from=192.168.4.5
May 17 20:27:15 client-18 rshd[13486]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark Request clients to fail: 2. Num of clients to fail: 1, already failed: 0");echo XXRETCODE:$?'
May 17 20:27:15 client-18 kernel: Lustre: DEBUG MARKER: Request clients to fail: 2. Num of clients to fail: 1, already failed: 0
May 17 20:27:15 client-18 xinetd[3390]: EXIT: shell status=0 pid=13485 duration=0(sec)
May 17 20:27:15 client-18 xinetd[3390]: START: shell pid=13520 from=192.168.4.5
May 17 20:27:15 client-18 rshd[13521]: root@client-5-ib.lab.whamcloud.com as root: cmd='mkdir /mnt/lustre/d0.insanity/client-18-ib && ls -l /mnt/lustre/d0.insanity/client-18-ib;echo XXRETCODE:$?'
May 17 20:27:15 client-18 xinetd[3390]: EXIT: shell status=0 pid=13520 duration=0(sec)
May 17 20:27:15 client-18 xinetd[3390]: START: shell pid=13555 from=192.168.4.5
May 17 20:27:15 client-18 rshd[13556]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:-f; lsof -t /mnt/lustre || need_kill=no; if [ x-f != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount -f /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre is still busy, wait one second && sleep 1;     umount -f /mnt/lustre; fi fi");echo XXRETCODE:$?'
May 17 20:27:15 client-18 kernel: Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
May 17 20:27:15 client-18 kernel: Lustre: Skipped 6 previous similar messages
May 17 20:27:24 client-18 kernel: Lustre: client ffff8102ee809400 umount complete
May 17 20:27:24 client-18 kernel: Lustre: Skipped 5 previous similar messages
May 17 20:27:24 client-18 xinetd[3390]: EXIT: shell status=0 pid=13555 duration=9(sec)
May 17 20:27:37 client-18 xinetd[3390]: START: shell pid=13603 from=192.168.4.5
May 17 20:27:37 client-18 rshd[13604]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 20:27:37 client-18 xinetd[3390]: EXIT: shell status=0 pid=13603 duration=0(sec)
May 17 20:27:37 client-18 xinetd[3390]: START: shell pid=13637 from=192.168.4.5
May 17 20:27:37 client-18 rshd[13638]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "hostname");echo XXRETCODE:$?'
May 17 20:27:37 client-18 xinetd[3390]: START: shell pid=13672 from=192.168.4.5
May 17 20:27:37 client-18 xinetd[3390]: EXIT: shell status=0 pid=13637 duration=0(sec)
May 17 20:27:37 client-18 rshd[13673]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mkdir -p /mnt/lustre");echo XXRETCODE:$?'
May 17 20:27:37 client-18 xinetd[3390]: EXIT: shell status=0 pid=13672 duration=0(sec)
May 17 20:27:37 client-18 xinetd[3390]: START: shell pid=13707 from=192.168.4.5
May 17 20:27:37 client-18 rshd[13708]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre");echo XXRETCODE:$?'
May 17 20:27:37 client-18 kernel: Lustre: 13742:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null
May 17 20:27:37 client-18 kernel: Lustre: 13742:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 31 previous similar messages
May 17 20:27:37 client-18 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'.
May 17 20:27:37 client-18 kernel: LustreError: Skipped 3 previous similar messages
May 17 20:27:37 client-18 kernel: Lustre: Client lustre-client has started
May 17 20:27:37 client-18 kernel: Lustre: Skipped 3 previous similar messages
May 17 20:27:37 client-18 xinetd[3390]: EXIT: shell status=0 pid=13707 duration=0(sec)
May 17 20:27:37 client-18 xinetd[3390]: START: shell pid=13755 from=192.168.4.5
May 17 20:27:37 client-18 rshd[13756]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param debug=\"0x33f0406\";         lctl set_param subsystem_debug=\"0xffb7e3ff\";         lctl set_param debug_mb=32");echo XXRETCODE:$?'
May 17 20:27:37 client-18 kernel: Lustre: 13790:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release.
May 17 20:27:37 client-18 kernel: Lustre: 13790:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 9 previous similar messages
May 17 20:27:37 client-18 xinetd[3390]: EXIT: shell status=0 pid=13755 duration=0(sec)
May 17 20:27:38 client-18 xinetd[3390]: START: shell pid=13792 from=192.168.4.5
May 17 20:27:38 client-18 rshd[13793]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 20:27:38 client-18 xinetd[3390]: EXIT: shell status=0 pid=13792 duration=0(sec)
May 17 20:27:40 client-18 xinetd[3390]: START: shell pid=13826 from=192.168.4.5
May 17 20:27:40 client-18 rshd[13827]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 20:27:40 client-18 xinetd[3390]: EXIT: shell status=0 pid=13826 duration=0(sec)
May 17 20:27:41 client-18 xinetd[3390]: START: shell pid=13862 from=192.168.4.5
May 17 20:27:41 client-18 rshd[13863]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 20:27:41 client-18 xinetd[3390]: EXIT: shell status=0 pid=13862 duration=0(sec)
May 17 20:27:41 client-18 xinetd[3390]: START: shell pid=13899 from=192.168.4.5
May 17 20:27:41 client-18 rshd[13900]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == insanity test 4: Fourth Failure Mode: OST\/MDS Tue May 17 20:27:41 PDT 2011 ======================== 20:27:41 \(1305689261\)");echo XXRETCODE:$?'
May 17 20:27:41 client-18 kernel: Lustre: DEBUG MARKER: == insanity test 4: Fourth Failure Mode: OST/MDS Tue May 17 20:27:41 PDT 2011 ======================== 20:27:41 (1305689261)
May 17 20:27:41 client-18 xinetd[3390]: EXIT: shell status=0 pid=13899 duration=0(sec)
May 17 20:27:43 client-18 xinetd[3390]: START: shell pid=13934 from=192.168.4.5
May 17 20:27:43 client-18 rshd[13935]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 20:28:15 client-18 xinetd[3390]: START: shell pid=13968 from=192.168.4.5
May 17 20:28:15 client-18 rshd[13969]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 20:29:38 client-18 kernel: Lustre: 8885:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf668605 to 0xfff98862bf668732
May 17 20:30:48 client-18 xinetd[3390]: EXIT: shell status=0 pid=13934 duration=185(sec)
May 17 20:30:48 client-18 xinetd[3390]: EXIT: shell status=0 pid=13968 duration=153(sec)
May 17 20:30:49 client-18 xinetd[3390]: START: shell pid=14014 from=192.168.4.5
May 17 20:30:49 client-18 rshd[14015]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 20:30:49 client-18 xinetd[3390]: EXIT: shell status=0 pid=14014 duration=0(sec)
May 17 20:30:49 client-18 xinetd[3390]: START: shell pid=14048 from=192.168.4.5
May 17 20:30:49 client-18 rshd[14049]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 20:30:49 client-18 xinetd[3390]: EXIT: shell status=0 pid=14048 duration=0(sec)
May 17 20:30:49 client-18 xinetd[3390]: START: shell pid=14084 from=192.168.4.5
May 17 20:30:49 client-18 rshd[14085]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 20:30:49 client-18 xinetd[3390]: EXIT: shell status=0 pid=14084 duration=0(sec)
May 17 20:30:50 client-18 xinetd[3390]: START: shell pid=14121 from=192.168.4.5
May 17 20:30:50 client-18 rshd[14122]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == insanity test 5: Fifth Failure Mode: OST\/OST Tue May 17 20:30:50 PDT 2011 ========================= 20:30:50 \(1305689450\)");echo XXRETCODE:$?'
May 17 20:30:50 client-18 kernel: Lustre: DEBUG MARKER: == insanity test 5: Fifth Failure Mode: OST/OST Tue May 17 20:30:50 PDT 2011 ========================= 20:30:50 (1305689450)
May 17 20:30:50 client-18 xinetd[3390]: EXIT: shell status=0 pid=14121 duration=0(sec)
May 17 20:30:51 client-18 xinetd[3390]: START: shell pid=14156 from=192.168.4.5
May 17 20:30:51 client-18 rshd[14157]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 20:30:51 client-18 xinetd[3390]: EXIT: shell status=0 pid=14156 duration=0(sec)
May 17 20:31:03 client-18 kernel: Lustre: 8886:0:(import.c:529:import_select_connection()) lustre-OST0000-osc-ffff81033cfc4400: tried all connections, increasing latency to 21s
May 17 20:31:03 client-18 kernel: Lustre: 8886:0:(import.c:529:import_select_connection()) Skipped 54 previous similar messages
May 17 20:31:03 client-18 xinetd[3390]: START: shell pid=14190 from=192.168.4.5
May 17 20:31:03 client-18 rshd[14191]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 20:31:13 client-18 kernel: Lustre: lustre-OST0001-osc-ffff81033cfc4400: Connection to service lustre-OST0001 via nid 192.168.4.129@o2ib was lost; in progress operations using this service will wait for recovery to complete.
May 17 20:31:13 client-18 kernel: Lustre: Skipped 13 previous similar messages
May 17 20:31:20 client-18 xinetd[3390]: START: shell pid=14228 from=192.168.4.5
May 17 20:31:20 client-18 rshd[14229]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 20:31:29 client-18 xinetd[3390]: EXIT: shell status=0 pid=14190 duration=26(sec)
May 17 20:31:29 client-18 xinetd[3390]: EXIT: shell status=0 pid=14228 duration=9(sec)
May 17 20:31:47 client-18 xinetd[3390]: START: shell pid=14262 from=192.168.4.5
May 17 20:31:47 client-18 rshd[14263]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 20:31:47 client-18 xinetd[3390]: EXIT: shell status=0 pid=14262 duration=0(sec)
May 17 20:31:47 client-18 xinetd[3390]: START: shell pid=14296 from=192.168.4.5
May 17 20:31:47 client-18 rshd[14297]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 20:31:47 client-18 xinetd[3390]: EXIT: shell status=0 pid=14296 duration=0(sec)
May 17 20:31:47 client-18 xinetd[3390]: START: shell pid=14332 from=192.168.4.5
May 17 20:31:47 client-18 rshd[14333]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 20:31:47 client-18 xinetd[3390]: EXIT: shell status=0 pid=14332 duration=0(sec)
May 17 20:31:47 client-18 xinetd[3390]: START: shell pid=14369 from=192.168.4.5
May 17 20:31:47 client-18 rshd[14370]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == insanity test 6: Sixth Failure Mode: OST\/CLIENT Tue May 17 20:31:47 PDT 2011 ====================== 20:31:47 \(1305689507\)");echo XXRETCODE:$?'
May 17 20:31:47 client-18 kernel: Lustre: DEBUG MARKER: == insanity test 6: Sixth Failure Mode: OST/CLIENT Tue May 17 20:31:47 PDT 2011 ====================== 20:31:47 (1305689507)
May 17 20:31:47 client-18 xinetd[3390]: EXIT: shell status=0 pid=14369 duration=0(sec)
May 17 20:31:48 client-18 xinetd[3390]: START: shell pid=14404 from=192.168.4.5
May 17 20:31:48 client-18 rshd[14405]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 20:31:48 client-18 xinetd[3390]: EXIT: shell status=0 pid=14404 duration=0(sec)
May 17 20:31:49 client-18 xinetd[3390]: START: shell pid=14438 from=192.168.4.5
May 17 20:31:49 client-18 rshd[14439]: root@client-5-ib.lab.whamcloud.com as root: cmd='touch /mnt/lustre/d0.insanity/client-18-ib_testfile;echo XXRETCODE:$?'
May 17 20:31:49 client-18 xinetd[3390]: EXIT: shell status=0 pid=14438 duration=0(sec)
May 17 20:32:04 client-18 xinetd[3390]: START: shell pid=14472 from=192.168.4.5
May 17 20:32:04 client-18 rshd[14473]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 20:32:08 client-18 xinetd[3390]: START: shell pid=14506 from=192.168.4.5
May 17 20:32:08 client-18 rshd[14507]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark Request clients to fail: . Num of clients to fail: 1, already failed: 0");echo XXRETCODE:$?'
May 17 20:32:08 client-18 kernel: Lustre: DEBUG MARKER: Request clients to fail: . Num of clients to fail: 1, already failed: 0
May 17 20:32:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=14506 duration=0(sec)
May 17 20:32:08 client-18 xinetd[3390]: START: shell pid=14541 from=192.168.4.5
May 17 20:32:08 client-18 rshd[14542]: root@client-5-ib.lab.whamcloud.com as root: cmd='mkdir /mnt/lustre/d0.insanity/client-18-ib && ls -l /mnt/lustre/d0.insanity/client-18-ib;echo XXRETCODE:$?'
May 17 20:32:08 client-18 xinetd[3390]: EXIT: shell status=0 pid=14541 duration=0(sec)
May 17 20:32:08 client-18 xinetd[3390]: START: shell pid=14576 from=192.168.4.5
May 17 20:32:08 client-18 rshd[14577]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:-f; lsof -t /mnt/lustre || need_kill=no; if [ x-f != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount -f /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre is still busy, wait one second && sleep 1;     umount -f /mnt/lustre; fi fi");echo XXRETCODE:$?'
May 17 20:32:09 client-18 kernel: Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
May 17 20:32:09 client-18 kernel: Lustre: Skipped 6 previous similar messages
May 17 20:32:09 client-18 xinetd[3390]: EXIT: shell status=0 pid=14472 duration=5(sec)
May 17 20:32:09 client-18 kernel: LustreError: 14618:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway
May 17 20:32:09 client-18 kernel: LustreError: 14618:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Skipped 10 previous similar messages
May 17 20:32:09 client-18 kernel: LustreError: 14618:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108
May 17 20:32:09 client-18 kernel: LustreError: 14618:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) Skipped 10 previous similar messages
May 17 20:32:09 client-18 xinetd[3390]: EXIT: shell status=0 pid=14576 duration=1(sec)
May 17 20:32:21 client-18 xinetd[3390]: START: shell pid=14620 from=192.168.4.5
May 17 20:32:21 client-18 rshd[14621]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 20:32:21 client-18 xinetd[3390]: EXIT: shell status=0 pid=14620 duration=0(sec)
May 17 20:32:26 client-18 xinetd[3390]: START: shell pid=14654 from=192.168.4.5
May 17 20:32:26 client-18 rshd[14655]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "hostname");echo XXRETCODE:$?'
May 17 20:32:26 client-18 xinetd[3390]: EXIT: shell status=0 pid=14654 duration=0(sec)
May 17 20:32:26 client-18 xinetd[3390]: START: shell pid=14689 from=192.168.4.5
May 17 20:32:26 client-18 rshd[14690]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mkdir -p /mnt/lustre");echo XXRETCODE:$?'
May 17 20:32:26 client-18 xinetd[3390]: START: shell pid=14724 from=192.168.4.5
May 17 20:32:26 client-18 xinetd[3390]: EXIT: shell status=0 pid=14689 duration=0(sec)
May 17 20:32:26 client-18 rshd[14725]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre");echo XXRETCODE:$?'
May 17 20:32:26 client-18 xinetd[3390]: EXIT: shell status=0 pid=14724 duration=0(sec)
May 17 20:32:26 client-18 xinetd[3390]: START: shell pid=14772 from=192.168.4.5
May 17 20:32:26 client-18 rshd[14773]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param debug=\"0x33f0406\";         lctl set_param subsystem_debug=\"0xffb7e3ff\";         lctl set_param debug_mb=32");echo XXRETCODE:$?'
May 17 20:32:26 client-18 xinetd[3390]: EXIT: shell status=0 pid=14772 duration=0(sec)
May 17 20:33:26 client-18 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.129@o2ib. The ost_connect operation failed with -16
May 17 20:33:26 client-18 kernel: LustreError: Skipped 69 previous similar messages
May 17 20:35:02 client-18 xinetd[3390]: START: shell pid=14821 from=192.168.4.5
May 17 20:35:02 client-18 rshd[14822]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 20:35:02 client-18 xinetd[3390]: EXIT: shell status=0 pid=14821 duration=0(sec)
May 17 20:35:02 client-18 xinetd[3390]: START: shell pid=14855 from=192.168.4.5
May 17 20:35:02 client-18 rshd[14856]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 20:35:02 client-18 xinetd[3390]: EXIT: shell status=0 pid=14855 duration=0(sec)
May 17 20:35:03 client-18 xinetd[3390]: START: shell pid=14891 from=192.168.4.5
May 17 20:35:03 client-18 rshd[14892]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 20:35:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=14891 duration=0(sec)
May 17 20:35:03 client-18 xinetd[3390]: START: shell pid=14928 from=192.168.4.5
May 17 20:35:03 client-18 rshd[14929]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == insanity test 7: Seventh Failure Mode: CLIENT\/MDS Tue May 17 20:35:03 PDT 2011 ==================== 20:35:03 \(1305689703\)");echo XXRETCODE:$?'
May 17 20:35:03 client-18 kernel: Lustre: DEBUG MARKER: == insanity test 7: Seventh Failure Mode: CLIENT/MDS Tue May 17 20:35:03 PDT 2011 ==================== 20:35:03 (1305689703)
May 17 20:35:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=14928 duration=0(sec)
May 17 20:35:04 client-18 xinetd[3390]: START: shell pid=14963 from=192.168.4.5
May 17 20:35:04 client-18 rshd[14964]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 20:35:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=14963 duration=0(sec)
May 17 20:35:04 client-18 xinetd[3390]: START: shell pid=14997 from=192.168.4.5
May 17 20:35:04 client-18 rshd[14998]: root@client-5-ib.lab.whamcloud.com as root: cmd='touch /mnt/lustre/d0.insanity/client-18-ib_testfile;echo XXRETCODE:$?'
May 17 20:35:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=14997 duration=0(sec)
May 17 20:35:04 client-18 xinetd[3390]: START: shell pid=15031 from=192.168.4.5
May 17 20:35:04 client-18 rshd[15032]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark Request clients to fail: 2. Num of clients to fail: 1, already failed: 0");echo XXRETCODE:$?'
May 17 20:35:04 client-18 kernel: Lustre: DEBUG MARKER: Request clients to fail: 2. Num of clients to fail: 1, already failed: 0
May 17 20:35:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=15031 duration=0(sec)
May 17 20:35:04 client-18 xinetd[3390]: START: shell pid=15066 from=192.168.4.5
May 17 20:35:04 client-18 rshd[15067]: root@client-5-ib.lab.whamcloud.com as root: cmd='mkdir /mnt/lustre/d0.insanity/client-18-ib && ls -l /mnt/lustre/d0.insanity/client-18-ib;echo XXRETCODE:$?'
May 17 20:35:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=15066 duration=0(sec)
May 17 20:35:04 client-18 xinetd[3390]: START: shell pid=15101 from=192.168.4.5
May 17 20:35:05 client-18 rshd[15102]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:-f; lsof -t /mnt/lustre || need_kill=no; if [ x-f != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount -f /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre is still busy, wait one second && sleep 1;     umount -f /mnt/lustre; fi fi");echo XXRETCODE:$?'
May 17 20:35:05 client-18 kernel: Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
May 17 20:35:05 client-18 kernel: Lustre: Skipped 6 previous similar messages
May 17 20:35:05 client-18 xinetd[3390]: EXIT: shell status=0 pid=15101 duration=1(sec)
May 17 20:35:17 client-18 xinetd[3390]: START: shell pid=15145 from=192.168.4.5
May 17 20:35:17 client-18 rshd[15146]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 20:35:17 client-18 xinetd[3390]: EXIT: shell status=0 pid=15145 duration=0(sec)
May 17 20:36:18 client-18 xinetd[3390]: START: shell pid=15183 from=192.168.4.5
May 17 20:36:18 client-18 rshd[15184]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 20:36:19 client-18 xinetd[3390]: EXIT: shell status=0 pid=15183 duration=1(sec)
May 17 20:36:19 client-18 xinetd[3390]: START: shell pid=15217 from=192.168.4.5
May 17 20:36:19 client-18 rshd[15218]: root@client-5-ib.lab.whamcloud.com as root: cmd='rm /mnt/lustre/d0.insanity/client-18-ib_testfile;echo XXRETCODE:$?'
May 17 20:36:19 client-18 xinetd[3390]: EXIT: shell status=0 pid=15217 duration=0(sec)
May 17 20:36:37 client-18 xinetd[3390]: START: shell pid=15251 from=192.168.4.5
May 17 20:36:37 client-18 rshd[15252]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 20:36:37 client-18 xinetd[3390]: EXIT: shell status=0 pid=15251 duration=0(sec)
May 17 20:36:38 client-18 xinetd[3390]: START: shell pid=15285 from=192.168.4.5
May 17 20:36:38 client-18 rshd[15286]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "hostname");echo XXRETCODE:$?'
May 17 20:36:38 client-18 xinetd[3390]: START: shell pid=15320 from=192.168.4.5
May 17 20:36:38 client-18 xinetd[3390]: EXIT: shell status=0 pid=15285 duration=0(sec)
May 17 20:36:38 client-18 rshd[15321]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mkdir -p /mnt/lustre");echo XXRETCODE:$?'
May 17 20:36:38 client-18 xinetd[3390]: START: shell pid=15355 from=192.168.4.5
May 17 20:36:38 client-18 xinetd[3390]: EXIT: shell status=0 pid=15320 duration=0(sec)
May 17 20:36:38 client-18 rshd[15356]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre");echo XXRETCODE:$?'
May 17 20:36:38 client-18 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import
May 17 20:36:38 client-18 kernel: Lustre: Skipped 3 previous similar messages
May 17 20:36:38 client-18 xinetd[3390]: EXIT: shell status=0 pid=15355 duration=0(sec)
May 17 20:36:38 client-18 xinetd[3390]: START: shell pid=15403 from=192.168.4.5
May 17 20:36:38 client-18 rshd[15404]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param debug=\"0x33f0406\";         lctl set_param subsystem_debug=\"0xffb7e3ff\";         lctl set_param debug_mb=32");echo XXRETCODE:$?'
May 17 20:36:38 client-18 xinetd[3390]: EXIT: shell status=0 pid=15403 duration=0(sec)
May 17 20:36:39 client-18 xinetd[3390]: START: shell pid=15440 from=192.168.4.5
May 17 20:36:39 client-18 rshd[15441]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 20:36:39 client-18 xinetd[3390]: EXIT: shell status=0 pid=15440 duration=0(sec)
May 17 20:37:39 client-18 xinetd[3390]: START: shell pid=15478 from=192.168.4.5
May 17 20:37:39 client-18 rshd[15479]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 20:37:39 client-18 xinetd[3390]: EXIT: shell status=0 pid=15478 duration=0(sec)
May 17 20:37:39 client-18 xinetd[3390]: START: shell pid=15514 from=192.168.4.5
May 17 20:37:39 client-18 rshd[15515]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 20:37:40 client-18 xinetd[3390]: EXIT: shell status=0 pid=15514 duration=1(sec)
May 17 20:37:40 client-18 xinetd[3390]: START: shell pid=15551 from=192.168.4.5
May 17 20:37:40 client-18 rshd[15552]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == insanity test 8: Eighth Failure Mode: CLIENT\/OST Tue May 17 20:37:40 PDT 2011 ===================== 20:37:40 \(1305689860\)");echo XXRETCODE:$?'
May 17 20:37:40 client-18 kernel: Lustre: DEBUG MARKER: == insanity test 8: Eighth Failure Mode: CLIENT/OST Tue May 17 20:37:40 PDT 2011 ===================== 20:37:40 (1305689860)
May 17 20:37:40 client-18 xinetd[3390]: EXIT: shell status=0 pid=15551 duration=0(sec)
May 17 20:37:41 client-18 xinetd[3390]: START: shell pid=15586 from=192.168.4.5
May 17 20:37:41 client-18 rshd[15587]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 20:37:41 client-18 xinetd[3390]: EXIT: shell status=0 pid=15586 duration=0(sec)
May 17 20:37:41 client-18 xinetd[3390]: START: shell pid=15620 from=192.168.4.5
May 17 20:37:41 client-18 rshd[15621]: root@client-5-ib.lab.whamcloud.com as root: cmd='touch /mnt/lustre/d0.insanity/client-18-ib_testfile;echo XXRETCODE:$?'
May 17 20:37:41 client-18 xinetd[3390]: EXIT: shell status=0 pid=15620 duration=0(sec)
May 17 20:37:41 client-18 xinetd[3390]: START: shell pid=15654 from=192.168.4.5
May 17 20:37:41 client-18 rshd[15655]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark Request clients to fail: 2. Num of clients to fail: 1, already failed: 0");echo XXRETCODE:$?'
May 17 20:37:41 client-18 kernel: Lustre: DEBUG MARKER: Request clients to fail: 2. Num of clients to fail: 1, already failed: 0
May 17 20:37:41 client-18 xinetd[3390]: EXIT: shell status=0 pid=15654 duration=0(sec)
May 17 20:37:41 client-18 xinetd[3390]: START: shell pid=15689 from=192.168.4.5
May 17 20:37:41 client-18 rshd[15690]: root@client-5-ib.lab.whamcloud.com as root: cmd='mkdir /mnt/lustre/d0.insanity/client-18-ib && ls -l /mnt/lustre/d0.insanity/client-18-ib;echo XXRETCODE:$?'
May 17 20:37:41 client-18 xinetd[3390]: EXIT: shell status=0 pid=15689 duration=0(sec)
May 17 20:37:41 client-18 xinetd[3390]: START: shell pid=15724 from=192.168.4.5
May 17 20:37:41 client-18 rshd[15725]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:-f; lsof -t /mnt/lustre || need_kill=no; if [ x-f != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount -f /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre is still busy, wait one second && sleep 1;     umount -f /mnt/lustre; fi fi");echo XXRETCODE:$?'
May 17 20:37:42 client-18 kernel: Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
May 17 20:37:42 client-18 kernel: Lustre: Skipped 6 previous similar messages
May 17 20:37:42 client-18 kernel: Lustre: client ffff8102f78ee800 umount complete
May 17 20:37:42 client-18 kernel: Lustre: Skipped 2 previous similar messages
May 17 20:37:42 client-18 xinetd[3390]: EXIT: shell status=0 pid=15724 duration=1(sec)
May 17 20:37:54 client-18 xinetd[3390]: START: shell pid=15768 from=192.168.4.5
May 17 20:37:54 client-18 rshd[15769]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 20:37:54 client-18 xinetd[3390]: EXIT: shell status=0 pid=15768 duration=0(sec)
May 17 20:38:55 client-18 xinetd[3390]: START: shell pid=15806 from=192.168.4.5
May 17 20:38:55 client-18 rshd[15807]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 20:38:55 client-18 xinetd[3390]: EXIT: shell status=0 pid=15806 duration=0(sec)
May 17 20:39:09 client-18 xinetd[3390]: START: shell pid=15840 from=192.168.4.5
May 17 20:39:09 client-18 rshd[15841]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 20:39:09 client-18 xinetd[3390]: EXIT: shell status=0 pid=15840 duration=0(sec)
May 17 20:39:13 client-18 xinetd[3390]: START: shell pid=15874 from=192.168.4.5
May 17 20:39:13 client-18 rshd[15875]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "hostname");echo XXRETCODE:$?'
May 17 20:39:13 client-18 xinetd[3390]: START: shell pid=15909 from=192.168.4.5
May 17 20:39:13 client-18 xinetd[3390]: EXIT: shell status=0 pid=15874 duration=0(sec)
May 17 20:39:13 client-18 rshd[15910]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mkdir -p /mnt/lustre");echo XXRETCODE:$?'
May 17 20:39:13 client-18 xinetd[3390]: EXIT: shell status=0 pid=15909 duration=0(sec)
May 17 20:39:13 client-18 xinetd[3390]: START: shell pid=15944 from=192.168.4.5
May 17 20:39:13 client-18 rshd[15945]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre");echo XXRETCODE:$?'
May 17 20:39:13 client-18 kernel: Lustre: 15979:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null
May 17 20:39:13 client-18 kernel: Lustre: 15979:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 23 previous similar messages
May 17 20:39:13 client-18 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'.
May 17 20:39:13 client-18 kernel: LustreError: Skipped 2 previous similar messages
May 17 20:39:13 client-18 kernel: Lustre: Client lustre-client has started
May 17 20:39:13 client-18 kernel: Lustre: Skipped 2 previous similar messages
May 17 20:39:13 client-18 xinetd[3390]: START: shell pid=15992 from=192.168.4.5
May 17 20:39:13 client-18 xinetd[3390]: EXIT: shell status=0 pid=15944 duration=0(sec)
May 17 20:39:13 client-18 rshd[15993]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param debug=\"0x33f0406\";         lctl set_param subsystem_debug=\"0xffb7e3ff\";         lctl set_param debug_mb=32");echo XXRETCODE:$?'
May 17 20:39:13 client-18 kernel: Lustre: 16027:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release.
May 17 20:39:13 client-18 kernel: Lustre: 16027:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 5 previous similar messages
May 17 20:39:13 client-18 xinetd[3390]: EXIT: shell status=0 pid=15992 duration=0(sec)
May 17 20:39:22 client-18 xinetd[3390]: START: shell pid=16029 from=192.168.4.5
May 17 20:39:22 client-18 rshd[16030]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 20:39:23 client-18 xinetd[3390]: EXIT: shell status=0 pid=16029 duration=1(sec)
May 17 20:39:23 client-18 xinetd[3390]: START: shell pid=16063 from=192.168.4.5
May 17 20:39:23 client-18 rshd[16064]: root@client-5-ib.lab.whamcloud.com as root: cmd='touch /mnt/lustre/d0.insanity/client-18-ib_testfile2;echo XXRETCODE:$?'
May 17 20:39:23 client-18 xinetd[3390]: EXIT: shell status=0 pid=16063 duration=0(sec)
May 17 20:40:23 client-18 xinetd[3390]: START: shell pid=16101 from=192.168.4.5
May 17 20:40:23 client-18 rshd[16102]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 20:40:24 client-18 xinetd[3390]: EXIT: shell status=0 pid=16101 duration=1(sec)
May 17 20:40:24 client-18 xinetd[3390]: START: shell pid=16137 from=192.168.4.5
May 17 20:40:24 client-18 rshd[16138]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 20:40:24 client-18 xinetd[3390]: EXIT: shell status=0 pid=16137 duration=0(sec)
May 17 20:40:24 client-18 xinetd[3390]: START: shell pid=16174 from=192.168.4.5
May 17 20:40:24 client-18 rshd[16175]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == insanity test 9: Ninth Failure Mode: CLIENT\/CLIENT Tue May 17 20:40:24 PDT 2011 =================== 20:40:24 \(1305690024\)");echo XXRETCODE:$?'
May 17 20:40:24 client-18 kernel: Lustre: DEBUG MARKER: == insanity test 9: Ninth Failure Mode: CLIENT/CLIENT Tue May 17 20:40:24 PDT 2011 =================== 20:40:24 (1305690024)
May 17 20:40:24 client-18 xinetd[3390]: EXIT: shell status=0 pid=16174 duration=0(sec)
May 17 20:40:25 client-18 xinetd[3390]: START: shell pid=16209 from=192.168.4.5
May 17 20:40:25 client-18 rshd[16210]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 20:40:25 client-18 xinetd[3390]: EXIT: shell status=0 pid=16209 duration=0(sec)
May 17 20:40:25 client-18 xinetd[3390]: START: shell pid=16243 from=192.168.4.5
May 17 20:40:25 client-18 rshd[16244]: root@client-5-ib.lab.whamcloud.com as root: cmd='touch /mnt/lustre/d0.insanity/client-18-ib_testfile;echo XXRETCODE:$?'
May 17 20:40:25 client-18 xinetd[3390]: EXIT: shell status=0 pid=16243 duration=0(sec)
May 17 20:40:25 client-18 xinetd[3390]: START: shell pid=16277 from=192.168.4.5
May 17 20:40:25 client-18 rshd[16278]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark Request clients to fail: 2. Num of clients to fail: 1, already failed: 0");echo XXRETCODE:$?'
May 17 20:40:25 client-18 kernel: Lustre: DEBUG MARKER: Request clients to fail: 2. Num of clients to fail: 1, already failed: 0
May 17 20:40:25 client-18 xinetd[3390]: EXIT: shell status=0 pid=16277 duration=0(sec)
May 17 20:40:25 client-18 xinetd[3390]: START: shell pid=16312 from=192.168.4.5
May 17 20:40:25 client-18 rshd[16313]: root@client-5-ib.lab.whamcloud.com as root: cmd='mkdir /mnt/lustre/d0.insanity/client-18-ib && ls -l /mnt/lustre/d0.insanity/client-18-ib;echo XXRETCODE:$?'
May 17 20:40:26 client-18 xinetd[3390]: EXIT: shell status=0 pid=16312 duration=1(sec)
May 17 20:40:26 client-18 xinetd[3390]: START: shell pid=16347 from=192.168.4.5
May 17 20:40:26 client-18 rshd[16348]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:-f; lsof -t /mnt/lustre || need_kill=no; if [ x-f != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount -f /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre is still busy, wait one second && sleep 1;     umount -f /mnt/lustre; fi fi");echo XXRETCODE:$?'
May 17 20:40:26 client-18 kernel: Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
May 17 20:40:26 client-18 kernel: Lustre: Skipped 6 previous similar messages
May 17 20:40:26 client-18 xinetd[3390]: EXIT: shell status=0 pid=16347 duration=0(sec)
May 17 20:40:38 client-18 xinetd[3390]: START: shell pid=16391 from=192.168.4.5
May 17 20:40:38 client-18 rshd[16392]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 20:40:38 client-18 xinetd[3390]: EXIT: shell status=0 pid=16391 duration=0(sec)
May 17 20:41:40 client-18 xinetd[3390]: START: shell pid=16429 from=192.168.4.5
May 17 20:41:40 client-18 rshd[16430]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark Request clients to fail: 2. Num of clients to fail: 1, already failed: 1");echo XXRETCODE:$?'
May 17 20:41:40 client-18 kernel: Lustre: DEBUG MARKER: Request clients to fail: 2. Num of clients to fail: 1, already failed: 1
May 17 20:41:40 client-18 xinetd[3390]: EXIT: shell status=0 pid=16429 duration=0(sec)
May 17 20:41:40 client-18 xinetd[3390]: START: shell pid=16464 from=192.168.4.5
May 17 20:41:40 client-18 rshd[16465]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark No clients failed!");echo XXRETCODE:$?'
May 17 20:41:40 client-18 kernel: Lustre: DEBUG MARKER: No clients failed!
May 17 20:41:40 client-18 xinetd[3390]: EXIT: shell status=0 pid=16464 duration=0(sec)
May 17 20:41:41 client-18 xinetd[3390]: START: shell pid=16499 from=192.168.4.5
May 17 20:41:41 client-18 rshd[16500]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 20:41:41 client-18 xinetd[3390]: EXIT: shell status=0 pid=16499 duration=0(sec)
May 17 20:41:41 client-18 xinetd[3390]: START: shell pid=16533 from=192.168.4.5
May 17 20:41:41 client-18 rshd[16534]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "hostname");echo XXRETCODE:$?'
May 17 20:41:41 client-18 xinetd[3390]: START: shell pid=16569 from=192.168.4.5
May 17 20:41:41 client-18 xinetd[3390]: EXIT: shell status=0 pid=16533 duration=0(sec)
May 17 20:41:41 client-18 rshd[16570]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mkdir -p /mnt/lustre");echo XXRETCODE:$?'
May 17 20:41:41 client-18 xinetd[3390]: START: shell pid=16604 from=192.168.4.5
May 17 20:41:41 client-18 xinetd[3390]: EXIT: shell status=0 pid=16569 duration=0(sec)
May 17 20:41:41 client-18 rshd[16605]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre");echo XXRETCODE:$?'
May 17 20:41:42 client-18 xinetd[3390]: EXIT: shell status=0 pid=16604 duration=1(sec)
May 17 20:41:42 client-18 xinetd[3390]: START: shell pid=16652 from=192.168.4.5
May 17 20:41:42 client-18 rshd[16653]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param debug=\"0x33f0406\";         lctl set_param subsystem_debug=\"0xffb7e3ff\";         lctl set_param debug_mb=32");echo XXRETCODE:$?'
May 17 20:41:42 client-18 xinetd[3390]: EXIT: shell status=0 pid=16652 duration=0(sec)
May 17 20:41:43 client-18 xinetd[3390]: START: shell pid=16689 from=192.168.4.5
May 17 20:41:43 client-18 rshd[16690]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 20:41:43 client-18 xinetd[3390]: EXIT: shell status=0 pid=16689 duration=0(sec)
May 17 20:42:43 client-18 xinetd[3390]: START: shell pid=16727 from=192.168.4.5
May 17 20:42:43 client-18 rshd[16728]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 20:42:43 client-18 xinetd[3390]: EXIT: shell status=0 pid=16727 duration=0(sec)
May 17 20:42:43 client-18 xinetd[3390]: START: shell pid=16763 from=192.168.4.5
May 17 20:42:43 client-18 rshd[16764]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 20:42:43 client-18 xinetd[3390]: EXIT: shell status=0 pid=16763 duration=0(sec)
May 17 20:42:43 client-18 xinetd[3390]: START: shell pid=16800 from=192.168.4.5
May 17 20:42:43 client-18 rshd[16801]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark  SKIP: insanity test_10 skipping ALWAYS excluded test 10");echo XXRETCODE:$?'
May 17 20:42:43 client-18 kernel: Lustre: DEBUG MARKER: SKIP: insanity test_10 skipping ALWAYS excluded test 10
May 17 20:42:43 client-18 xinetd[3390]: EXIT: shell status=0 pid=16800 duration=0(sec)
May 17 20:42:43 client-18 xinetd[3390]: START: shell pid=16835 from=192.168.4.5
May 17 20:42:43 client-18 rshd[16836]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == insanity insanity.sh test complete, duration 1267 sec ============================================= 20:42:43 \(1305690163\)");echo XXRETCODE:$?'
May 17 20:42:43 client-18 kernel: Lustre: DEBUG MARKER: == insanity insanity.sh test complete, duration 1267 sec ============================================= 20:42:43 (1305690163)
May 17 20:42:43 client-18 xinetd[3390]: EXIT: shell status=0 pid=16835 duration=0(sec)
May 17 20:42:54 client-18 kernel: Lustre: 8884:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369101282860031 sent from MGC192.168.4.128@o2ib to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305690167] [real_sent 1305690167] [current 1305690174] [deadline 7s] [delay 0s]  req@ffff81033feb0400 x1369101282860031/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 192/192 e 0 to 1 dl 1305690174 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
May 17 20:42:54 client-18 kernel: Lustre: 8884:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 37 previous similar messages
May 17 20:42:54 client-18 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail.
May 17 20:42:54 client-18 kernel: LustreError: Skipped 3 previous similar messages
May 17 20:43:02 client-18 xinetd[3390]: START: shell pid=16870 from=192.168.4.5
May 17 20:43:02 client-18 rshd[16871]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?'
May 17 20:43:02 client-18 kernel: Lustre: lustre-MDT0000-mdc-ffff81032819b400: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete.
May 17 20:43:02 client-18 kernel: Lustre: Skipped 1 previous similar message
May 17 20:43:02 client-18 kernel: Lustre: lustre-MDT0000-mdc-ffff81032819b400: Connection restored to service lustre-MDT0000 using nid 192.168.4.128@o2ib.
May 17 20:43:02 client-18 kernel: Lustre: Skipped 15 previous similar messages
May 17 20:43:02 client-18 xinetd[3390]: EXIT: shell status=0 pid=16870 duration=0(sec)
May 17 20:43:02 client-18 xinetd[3390]: START: shell pid=16904 from=192.168.4.5
May 17 20:43:02 client-18 rshd[16905]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:-f; lsof -t /mnt/lustre || need_kill=no; if [ x-f != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount -f /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre is still busy, wait one second && sleep 1;     umount -f /mnt/lustre; fi fi");echo XXRETCODE:$?'
May 17 20:43:02 client-18 kernel: Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
May 17 20:43:02 client-18 kernel: Lustre: Skipped 6 previous similar messages
May 17 20:43:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=16904 duration=1(sec)
May 17 20:43:10 client-18 xinetd[3390]: START: shell pid=16948 from=192.168.4.5
May 17 20:43:10 client-18 rshd[16949]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:-f; lsof -t /mnt/lustre2 || need_kill=no; if [ x-f != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre2 | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount -f /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre2 is still busy, wait one second && sleep 1;     umount -f /mnt/lustre2; fi fi");echo XXRETCODE:$?'
May 17 20:43:10 client-18 xinetd[3390]: EXIT: shell status=0 pid=16948 duration=0(sec)
May 17 20:43:41 client-18 xinetd[3390]: START: shell pid=16985 from=192.168.4.5
May 17 20:43:41 client-18 rshd[16986]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark -----============= acceptance-small: sanity-quota ============----- Tue May 17 20:43:40 PDT 2011");echo XXRETCODE:$?'
May 17 20:43:41 client-18 kernel: Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-quota ============----- Tue May 17 20:43:40 PDT 2011
May 17 20:43:41 client-18 xinetd[3390]: EXIT: shell status=0 pid=16985 duration=0(sec)
May 17 20:43:41 client-18 xinetd[3390]: START: shell pid=17020 from=192.168.4.5
May 17 20:43:41 client-18 rshd[17021]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:; lsof -t /mnt/lustre || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount  /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre is still busy, wait one second && sleep 1;     umount  /mnt/lustre; fi fi");echo XXRETCODE:$?'
May 17 20:43:41 client-18 xinetd[3390]: EXIT: shell status=0 pid=17020 duration=0(sec)
May 17 20:43:41 client-18 xinetd[3390]: START: shell pid=17057 from=192.168.4.5
May 17 20:43:41 client-18 rshd[17058]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:; lsof -t /mnt/lustre2 || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then     pids=\$(lsof -t /mnt/lustre2 | sort -u);     if [ -n \"\$pids\" ]; then              kill -9 \$pids;     fi fi; busy=\$(umount  /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then     echo /mnt/lustre2 is still busy, wait one second && sleep 1;     umount  /mnt/lustre2; fi fi");echo XXRETCODE:$?'
May 17 20:43:41 client-18 xinetd[3390]: EXIT: shell status=0 pid=17057 duration=0(sec)
May 17 20:44:41 client-18 xinetd[3390]: START: shell pid=17098 from=192.168.4.5
May 17 20:44:41 client-18 rshd[17099]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); mpts=\$(mount | grep -w -c /mnt/lustre); if [ \$running -ne \$mpts ]; then     echo \$(hostname) env are INSANE!;     exit 1; fi");echo XXRETCODE:$?'
May 17 20:44:41 client-18 xinetd[3390]: EXIT: shell status=0 pid=17098 duration=0(sec)
May 17 20:44:41 client-18 xinetd[3390]: START: shell pid=17138 from=192.168.4.5
May 17 20:44:41 client-18 rshd[17139]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); mpts=\$(mount | grep -w -c /mnt/lustre2); if [ \$running -ne \$mpts ]; then     echo \$(hostname) env are INSANE!;     exit 1; fi");echo XXRETCODE:$?'
May 17 20:44:41 client-18 xinetd[3390]: EXIT: shell status=0 pid=17138 duration=0(sec)
May 17 20:44:50 client-18 xinetd[3390]: START: shell pid=17178 from=192.168.4.5
May 17 20:44:50 client-18 rshd[17179]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c " running=\$(mount | grep -c /mnt/lustre' '); rc=0; if [ \$running -eq 0 ] ; then     mkdir -p /mnt/lustre;     mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre;     rc=\$?; fi; exit \$rc");echo XXRETCODE:$?'
May 17 20:44:51 client-18 xinetd[3390]: EXIT: shell status=0 pid=17178 duration=1(sec)
May 17 20:44:51 client-18 xinetd[3390]: START: shell pid=17231 from=192.168.4.5
May 17 20:44:51 client-18 rshd[17232]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "mount | grep -w /mnt/lustre");echo XXRETCODE:$?'
May 17 20:44:51 client-18 xinetd[3390]: EXIT: shell status=0 pid=17231 duration=0(sec)
May 17 20:44:51 client-18 xinetd[3390]: START: shell pid=17268 from=192.168.4.5
May 17 20:44:51 client-18 rshd[17269]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param debug=\"0x33f0406\";         lctl set_param subsystem_debug=\"0xffb7e3ff\";         lctl set_param debug_mb=32;");echo XXRETCODE:$?'
May 17 20:44:51 client-18 xinetd[3390]: EXIT: shell status=0 pid=17268 duration=0(sec)
May 17 20:44:51 client-18 xinetd[3390]: START: shell pid=17305 from=192.168.4.5
May 17 20:44:51 client-18 rshd[17306]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark Using TIMEOUT=20");echo XXRETCODE:$?'
May 17 20:44:51 client-18 kernel: Lustre: DEBUG MARKER: Using TIMEOUT=20
May 17 20:44:51 client-18 xinetd[3390]: EXIT: shell status=0 pid=17305 duration=0(sec)
May 17 20:44:56 client-18 xinetd[3390]: START: shell pid=17340 from=192.168.4.5
May 17 20:44:56 client-18 rshd[17341]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "cat /proc/mounts");echo XXRETCODE:$?'
May 17 20:44:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=17340 duration=1(sec)
May 17 20:44:57 client-18 xinetd[3390]: START: shell pid=17379 from=192.168.4.5
May 17 20:44:57 client-18 rshd[17380]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl get_param -n osc.*OST*-osc-[^M][^D][^T]*.srpc_info");echo XXRETCODE:$?'
May 17 20:44:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=17379 duration=0(sec)
May 17 20:44:57 client-18 xinetd[3390]: START: shell pid=17414 from=192.168.4.5
May 17 20:44:57 client-18 rshd[17415]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl get_param -n mdc.*-MDT*-mdc-*.srpc_info");echo XXRETCODE:$?'
May 17 20:44:57 client-18 xinetd[3390]: EXIT: shell status=0 pid=17414 duration=0(sec)
May 17 20:44:57 client-18 xinetd[3390]: START: shell pid=17449 from=192.168.4.5
May 17 20:44:58 client-18 rshd[17450]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark excepting tests: 10 12");echo XXRETCODE:$?'
May 17 20:44:58 client-18 kernel: Lustre: DEBUG MARKER: excepting tests: 10 12
May 17 20:44:58 client-18 xinetd[3390]: EXIT: shell status=0 pid=17449 duration=1(sec)
May 17 20:44:58 client-18 xinetd[3390]: START: shell pid=17484 from=192.168.4.5
May 17 20:44:58 client-18 rshd[17485]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param debug=+quota");echo XXRETCODE:$?'
May 17 20:44:58 client-18 xinetd[3390]: EXIT: shell status=0 pid=17484 duration=0(sec)
May 17 20:44:58 client-18 xinetd[3390]: START: shell pid=17519 from=192.168.4.5
May 17 20:44:58 client-18 rshd[17520]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark do the quotacheck ...");echo XXRETCODE:$?'
May 17 20:44:58 client-18 kernel: Lustre: DEBUG MARKER: do the quotacheck ...
May 17 20:44:58 client-18 xinetd[3390]: EXIT: shell status=0 pid=17519 duration=0(sec)
May 17 20:45:01 client-18 xinetd[3390]: START: shell pid=17554 from=192.168.4.5
May 17 20:45:01 client-18 rshd[17555]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity-quota test 0: Test basic quota performance ================================================= 20:45:01 \(1305690301\)");echo XXRETCODE:$?'
May 17 20:45:01 client-18 kernel: Lustre: DEBUG MARKER: == sanity-quota test 0: Test basic quota performance ================================================= 20:45:01 (1305690301)
May 17 20:45:01 client-18 xinetd[3390]: EXIT: shell status=0 pid=17554 duration=0(sec)
May 17 20:45:03 client-18 xinetd[3390]: START: shell pid=17589 from=192.168.4.5
May 17 20:45:03 client-18 rshd[17590]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?'
May 17 20:45:03 client-18 xinetd[3390]: EXIT: shell status=0 pid=17589 duration=0(sec)
May 17 20:45:04 client-18 xinetd[3390]: START: shell pid=17625 from=192.168.4.5
May 17 20:45:04 client-18 rshd[17626]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "rc=\$([ -f /proc/sys/lnet/catastrophe ] && echo \$(< /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc;");echo XXRETCODE:$?'
May 17 20:45:04 client-18 xinetd[3390]: EXIT: shell status=0 pid=17625 duration=0(sec)
May 17 20:45:05 client-18 xinetd[3390]: START: shell pid=17662 from=192.168.4.5
May 17 20:45:05 client-18 rshd[17663]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark == sanity-quota test 1: Block hard limit \(normal use and out of quota\) =============================== 20:45:05 \(1305690305\)");echo XXRETCODE:$?'
May 17 20:45:05 client-18 kernel: Lustre: DEBUG MARKER: == sanity-quota test 1: Block hard limit (normal use and out of quota) =============================== 20:45:05 (1305690305)
May 17 20:45:05 client-18 xinetd[3390]: EXIT: shell status=0 pid=17662 duration=0(sec)
May 17 20:45:07 client-18 xinetd[3390]: START: shell pid=17697 from=192.168.4.5
May 17 20:45:07 client-18 rshd[17698]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark   User quota \(limit: 115253 kbytes\)");echo XXRETCODE:$?'
May 17 20:45:07 client-18 kernel: Lustre: DEBUG MARKER: User quota (limit: 115253 kbytes)
May 17 20:45:07 client-18 xinetd[3390]: EXIT: shell status=0 pid=17697 duration=0(sec)
May 17 20:45:10 client-18 xinetd[3390]: START: shell pid=17732 from=192.168.4.5
May 17 20:45:10 client-18 rshd[17733]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark     Write ...");echo XXRETCODE:$?'
May 17 20:45:10 client-18 kernel: Lustre: DEBUG MARKER: Write ...
May 17 20:45:10 client-18 xinetd[3390]: EXIT: shell status=0 pid=17732 duration=0(sec)
May 17 20:45:16 client-18 xinetd[3390]: START: shell pid=17767 from=192.168.4.5
May 17 20:45:16 client-18 rshd[17768]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark     Done");echo XXRETCODE:$?'
May 17 20:45:16 client-18 kernel: Lustre: DEBUG MARKER: Done
May 17 20:45:16 client-18 xinetd[3390]: EXIT: shell status=0 pid=17767 duration=0(sec)
May 17 20:45:16 client-18 xinetd[3390]: START: shell pid=17802 from=192.168.4.5
May 17 20:45:16 client-18 rshd[17803]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark     Write out of block quota ...");echo XXRETCODE:$?'
May 17 20:45:16 client-18 kernel: Lustre: DEBUG MARKER: Write out of block quota ...
May 17 20:45:16 client-18 xinetd[3390]: EXIT: shell status=0 pid=17802 duration=0(sec)
May 17 20:45:23 client-18 xinetd[3390]: START: shell pid=17837 from=192.168.4.5
May 17 20:45:23 client-18 rshd[17838]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark --------------------------------------");echo XXRETCODE:$?'
May 17 20:45:23 client-18 kernel: Lustre: DEBUG MARKER: --------------------------------------
May 17 20:45:23 client-18 xinetd[3390]: EXIT: shell status=0 pid=17837 duration=0(sec)
May 17 20:45:23 client-18 xinetd[3390]: START: shell pid=17872 from=192.168.4.5
May 17 20:45:23 client-18 rshd[17873]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark   Group quota \(limit: 115253 kbytes\)");echo XXRETCODE:$?'
May 17 20:45:23 client-18 kernel: Lustre: DEBUG MARKER: Group quota (limit: 115253 kbytes)
May 17 20:45:23 client-18 xinetd[3390]: EXIT: shell status=0 pid=17872 duration=0(sec)
May 17 20:45:26 client-18 xinetd[3390]: START: shell pid=17907 from=192.168.4.5
May 17 20:45:27 client-18 rshd[17908]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  sh -c "/usr/sbin/lctl mark     Write ...");echo XXRETCODE:$?'
May 17 20:45:27 client-18 kernel: Lustre: DEBUG MARKER: Write ...
May 17 20:45:27 client-18 xinetd[3390]: EXIT: shell status=0 pid=17907 duration=1(sec)
May 17 21:23:24 client-18 xinetd[3390]: START: login pid=18071 from=192.168.4.5
May 17 21:23:58 client-18 xinetd[3390]: EXIT: login status=0 pid=18071 duration=34(sec)
May 17 21:36:17 client-18 xinetd[3390]: START: login pid=18163 from=10.10.4.5