May 16 23:03:20 client-5 syslogd 1.4.1: restart. May 16 23:03:20 client-5 kernel: klogd 1.4.1, log source = /proc/kmsg started. May 16 23:03:20 client-5 kernel: Linux version 2.6.18-194.17.1.el5 (mockbuild@builder10.centos.org) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-48)) #1 SMP Wed Sep 29 12:50:31 EDT 2010 May 16 23:03:20 client-5 kernel: Command line: ro root=LABEL=/ console=ttyS0,115200 May 16 23:03:20 client-5 kernel: BIOS-provided physical RAM map: May 16 23:03:20 client-5 kernel: BIOS-e820: 0000000000010000 - 000000000009cc00 (usable) May 16 23:03:20 client-5 kernel: BIOS-e820: 000000000009cc00 - 00000000000a0000 (reserved) May 16 23:03:20 client-5 kernel: BIOS-e820: 00000000000e6000 - 0000000000100000 (reserved) May 16 23:03:20 client-5 kernel: BIOS-e820: 0000000000100000 - 00000000bf760000 (usable) May 16 23:03:20 client-5 kernel: BIOS-e820: 00000000bf76e000 - 00000000bf770000 type 9 May 16 23:03:20 client-5 kernel: BIOS-e820: 00000000bf770000 - 00000000bf77e000 (ACPI data) May 16 23:03:20 client-5 kernel: BIOS-e820: 00000000bf77e000 - 00000000bf7d0000 (ACPI NVS) May 16 23:03:20 client-5 kernel: BIOS-e820: 00000000bf7d0000 - 00000000bf7e0000 (reserved) May 16 23:03:20 client-5 kernel: BIOS-e820: 00000000bf7ec000 - 00000000c0000000 (reserved) May 16 23:03:20 client-5 kernel: BIOS-e820: 00000000e0000000 - 00000000f0000000 (reserved) May 16 23:03:20 client-5 kernel: BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved) May 16 23:03:20 client-5 kernel: BIOS-e820: 00000000ffc00000 - 0000000100000000 (reserved) May 16 23:03:20 client-5 kernel: BIOS-e820: 0000000100000000 - 0000000340000000 (usable) May 16 23:03:20 client-5 kernel: DMI present. May 16 23:03:20 client-5 kernel: No NUMA configuration found May 16 23:03:20 client-5 kernel: Faking a node at 0000000000000000-0000000340000000 May 16 23:03:20 client-5 kernel: Bootmem setup node 0 0000000000000000-0000000340000000 May 16 23:03:20 client-5 kernel: Memory for crash kernel (0x0 to 0x0) notwithin permissible range May 16 23:03:20 client-5 kernel: disabling kdump May 16 23:03:20 client-5 kernel: ACPI: PM-Timer IO Port: 0x808 May 16 23:03:20 client-5 kernel: ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled) May 16 23:03:20 client-5 kernel: Processor #0 7:10 APIC version 21 May 16 23:03:20 client-5 kernel: ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled) May 16 23:03:20 client-5 kernel: Processor #2 7:10 APIC version 21 May 16 23:03:20 client-5 kernel: ACPI: LAPIC (acpi_id[0x03] lapic_id[0x04] enabled) May 16 23:03:20 client-5 kernel: Processor #4 7:10 APIC version 21 May 16 23:03:20 client-5 kernel: ACPI: LAPIC (acpi_id[0x04] lapic_id[0x06] enabled) May 16 23:03:20 client-5 kernel: Processor #6 7:10 APIC version 21 May 16 23:03:20 client-5 kernel: ACPI: LAPIC (acpi_id[0x05] lapic_id[0x84] disabled) May 16 23:03:20 client-5 kernel: ACPI: LAPIC (acpi_id[0x06] lapic_id[0x85] disabled) May 16 23:03:20 client-5 kernel: ACPI: LAPIC (acpi_id[0x07] lapic_id[0x86] disabled) May 16 23:03:20 client-5 kernel: ACPI: LAPIC (acpi_id[0x08] lapic_id[0x87] disabled) May 16 23:03:20 client-5 kernel: ACPI: LAPIC (acpi_id[0x09] lapic_id[0x88] disabled) May 16 23:03:20 client-5 kernel: ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x89] disabled) May 16 23:03:20 client-5 kernel: ACPI: LAPIC (acpi_id[0x0b] lapic_id[0x8a] disabled) May 16 23:03:20 client-5 kernel: ACPI: LAPIC (acpi_id[0x0c] lapic_id[0x8b] disabled) May 16 23:03:20 client-5 kernel: ACPI: LAPIC (acpi_id[0x0d] lapic_id[0x8c] disabled) May 16 23:03:20 client-5 kernel: ACPI: LAPIC (acpi_id[0x0e] lapic_id[0x8d] disabled) May 16 23:03:20 client-5 kernel: ACPI: LAPIC (acpi_id[0x0f] lapic_id[0x8e] disabled) May 16 23:03:20 client-5 kernel: ACPI: LAPIC (acpi_id[0x10] lapic_id[0x8f] disabled) May 16 23:03:20 client-5 kernel: ACPI: LAPIC (acpi_id[0x11] lapic_id[0x90] disabled) May 16 23:03:20 client-5 kernel: ACPI: LAPIC (acpi_id[0x12] lapic_id[0x91] disabled) May 16 23:03:20 client-5 kernel: ACPI: LAPIC (acpi_id[0x13] lapic_id[0x92] disabled) May 16 23:03:20 client-5 kernel: ACPI: LAPIC (acpi_id[0x14] lapic_id[0x93] disabled) May 16 23:03:20 client-5 kernel: ACPI: LAPIC (acpi_id[0x15] lapic_id[0x94] disabled) May 16 23:03:20 client-5 kernel: ACPI: LAPIC (acpi_id[0x16] lapic_id[0x95] disabled) May 16 23:03:20 client-5 kernel: ACPI: LAPIC (acpi_id[0x17] lapic_id[0x96] disabled) May 16 23:03:20 client-5 kernel: ACPI: LAPIC (acpi_id[0x18] lapic_id[0x97] disabled) May 16 23:03:20 client-5 kernel: ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) May 16 23:03:20 client-5 kernel: ACPI: IOAPIC (id[0x01] address[0xfec00000] gsi_base[0]) May 16 23:03:20 client-5 kernel: IOAPIC[0]: apic_id 1, version 32, address 0xfec00000, GSI 0-23 May 16 23:03:20 client-5 kernel: ACPI: IOAPIC (id[0x03] address[0xfec8a000] gsi_base[24]) May 16 23:03:20 client-5 kernel: IOAPIC[1]: apic_id 3, version 32, address 0xfec8a000, GSI 24-47 May 16 23:03:20 client-5 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) May 16 23:03:20 client-5 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) May 16 23:03:20 client-5 kernel: Setting APIC routing to physical flat May 16 23:03:20 client-5 kernel: ACPI: HPET id: 0x8086a301 base: 0xfed00000 May 16 23:03:20 client-5 kernel: Using ACPI (MADT) for SMP configuration information May 16 23:03:20 client-5 kernel: Nosave address range: 000000000009c000 - 000000000009d000 May 16 23:03:20 client-5 kernel: Nosave address range: 000000000009d000 - 00000000000a0000 May 16 23:03:20 client-5 kernel: Nosave address range: 00000000000a0000 - 00000000000e6000 May 16 23:03:20 client-5 kernel: Nosave address range: 00000000000e6000 - 0000000000100000 May 16 23:03:20 client-5 kernel: Nosave address range: 00000000bf760000 - 00000000bf76e000 May 16 23:03:20 client-5 kernel: Nosave address range: 00000000bf76e000 - 00000000bf770000 May 16 23:03:20 client-5 kernel: Nosave address range: 00000000bf770000 - 00000000bf77e000 May 16 23:03:20 client-5 kernel: Nosave address range: 00000000bf77e000 - 00000000bf7d0000 May 16 23:03:20 client-5 kernel: Nosave address range: 00000000bf7d0000 - 00000000bf7e0000 May 16 23:03:20 client-5 kernel: Nosave address range: 00000000bf7e0000 - 00000000bf7ec000 May 16 23:03:20 client-5 kernel: Nosave address range: 00000000bf7ec000 - 00000000c0000000 May 16 23:03:20 client-5 kernel: Nosave address range: 00000000c0000000 - 00000000e0000000 May 16 23:03:20 client-5 kernel: Nosave address range: 00000000e0000000 - 00000000f0000000 May 16 23:03:20 client-5 kernel: Nosave address range: 00000000f0000000 - 00000000fee00000 May 16 23:03:20 client-5 kernel: Nosave address range: 00000000fee00000 - 00000000fee01000 May 16 23:03:20 client-5 kernel: Nosave address range: 00000000fee01000 - 00000000ffc00000 May 16 23:03:20 client-5 kernel: Nosave address range: 00000000ffc00000 - 0000000100000000 May 16 23:03:20 client-5 kernel: Allocating PCI resources starting at c2000000 (gap: c0000000:20000000) May 16 23:03:20 client-5 kernel: SMP: Allowing 24 CPUs, 20 hotplug CPUs May 16 23:03:20 client-5 kernel: Built 1 zonelists. Total pages: 3095502 May 16 23:03:20 client-5 kernel: Kernel command line: ro root=LABEL=/ console=ttyS0,115200 May 16 23:03:20 client-5 kernel: Initializing CPU#0 May 16 23:03:20 client-5 kernel: PID hash table entries: 4096 (order: 12, 32768 bytes) May 16 23:03:20 client-5 kernel: Console: colour VGA+ 80x25 May 16 23:03:20 client-5 kernel: Dentry cache hash table entries: 2097152 (order: 12, 16777216 bytes) May 16 23:03:20 client-5 kernel: Inode-cache hash table entries: 1048576 (order: 11, 8388608 bytes) May 16 23:03:20 client-5 kernel: Checking aperture... May 16 23:03:20 client-5 kernel: PCI-DMA: Using software bounce buffering for IO (SWIOTLB) May 16 23:03:20 client-5 kernel: Placing software IO TLB between 0x28e4000 - 0x68e4000 May 16 23:03:20 client-5 kernel: Memory: 12287648k/13631488k available (2574k kernel code, 285968k reserved, 1304k data, 212k init) May 16 23:03:20 client-5 kernel: Calibrating delay loop (skipped), value calculated using timer frequency.. 4533.62 BogoMIPS (lpj=2266814) May 16 23:03:20 client-5 kernel: Security Framework v1.0.0 initialized May 16 23:03:20 client-5 kernel: SELinux: Initializing. May 16 23:03:20 client-5 kernel: selinux_register_security: Registering secondary module capability May 16 23:03:20 client-5 kernel: Capability LSM initialized as secondary May 16 23:03:20 client-5 kernel: Mount-cache hash table entries: 256 May 16 23:03:20 client-5 kernel: CPU: L1 I cache: 32K, L1 D cache: 32K May 16 23:03:20 client-5 kernel: CPU: L2 cache: 256K May 16 23:03:20 client-5 kernel: CPU: L3 cache: 4096K May 16 23:03:20 client-5 kernel: using mwait in idle threads. May 16 23:03:20 client-5 kernel: CPU: Physical Processor ID: 0 May 16 23:03:20 client-5 kernel: CPU: Processor Core ID: 0 May 16 23:03:20 client-5 kernel: CPU0: Thermal monitoring enabled (TM1) May 16 23:03:20 client-5 cpuspeed: Enabling ondemand cpu frequency scaling governor May 16 23:03:20 client-5 kernel: SMP alternatives: switching to UP code May 16 23:03:20 client-5 kernel: ACPI: Core revision 20060707 May 16 23:03:20 client-5 kernel: Using local APIC timer interrupts. May 16 23:03:20 client-5 kernel: Detected 8.333 MHz APIC timer. May 16 23:03:20 client-5 kernel: SMP alternatives: switching to SMP code May 16 23:03:20 client-5 kernel: Booting processor 1/4 APIC 0x2 May 16 23:03:20 client-5 kernel: Initializing CPU#1 May 16 23:03:20 client-5 kernel: Calibrating delay using timer specific routine.. 4533.40 BogoMIPS (lpj=2266701) May 16 23:03:20 client-5 kernel: CPU: L1 I cache: 32K, L1 D cache: 32K May 16 23:03:20 client-5 kernel: CPU: L2 cache: 256K May 16 23:03:20 client-5 kernel: CPU: L3 cache: 4096K May 16 23:03:20 client-5 kernel: CPU: Physical Processor ID: 0 May 16 23:03:20 client-5 kernel: CPU: Processor Core ID: 1 May 16 23:03:20 client-5 kernel: CPU1: Thermal monitoring enabled (TM1) May 16 23:03:20 client-5 kernel: Intel(R) Xeon(R) CPU E5507 @ 2.27GHz stepping 05 May 16 23:03:20 client-5 kernel: SMP alternatives: switching to SMP code May 16 23:03:20 client-5 kernel: Booting processor 2/4 APIC 0x4 May 16 23:03:20 client-5 kernel: Initializing CPU#2 May 16 23:03:20 client-5 kernel: Calibrating delay using timer specific routine.. 4533.66 BogoMIPS (lpj=2266834) May 16 23:03:20 client-5 kernel: CPU: L1 I cache: 32K, L1 D cache: 32K May 16 23:03:20 client-5 kernel: CPU: L2 cache: 256K May 16 23:03:20 client-5 kernel: CPU: L3 cache: 4096K May 16 23:03:20 client-5 kernel: CPU: Physical Processor ID: 0 May 16 23:03:20 client-5 kernel: CPU: Processor Core ID: 2 May 16 23:03:20 client-5 kernel: CPU2: Thermal monitoring enabled (TM1) May 16 23:03:20 client-5 kernel: Intel(R) Xeon(R) CPU E5507 @ 2.27GHz stepping 05 May 16 23:03:20 client-5 kernel: SMP alternatives: switching to SMP code May 16 23:03:20 client-5 kernel: Booting processor 3/4 APIC 0x6 May 16 23:03:20 client-5 kernel: Initializing CPU#3 May 16 23:03:20 client-5 kernel: Calibrating delay using timer specific routine.. 4533.40 BogoMIPS (lpj=2266702) May 16 23:03:20 client-5 kernel: CPU: L1 I cache: 32K, L1 D cache: 32K May 16 23:03:20 client-5 kernel: CPU: L2 cache: 256K May 16 23:03:20 client-5 kernel: CPU: L3 cache: 4096K May 16 23:03:20 client-5 kernel: CPU: Physical Processor ID: 0 May 16 23:03:20 client-5 kernel: CPU: Processor Core ID: 3 May 16 23:03:20 client-5 kernel: CPU3: Thermal monitoring enabled (TM1) May 16 23:03:20 client-5 kernel: Intel(R) Xeon(R) CPU E5507 @ 2.27GHz stepping 05 May 16 23:03:20 client-5 kernel: Brought up 4 CPUs May 16 23:03:20 client-5 kernel: testing NMI watchdog ... OK. May 16 23:03:20 client-5 kernel: time.c: Using 14.318180 MHz WALL HPET GTOD HPET/TSC timer. May 16 23:03:20 client-5 kernel: time.c: Detected 2266.814 MHz processor. May 16 23:03:20 client-5 kernel: migration_cost=21 May 16 23:03:20 client-5 kernel: checking if image is initramfs... it is May 16 23:03:20 client-5 kernel: Freeing initrd memory: 2611k freed May 16 23:03:20 client-5 kernel: NET: Registered protocol family 16 May 16 23:03:20 client-5 kernel: ACPI: bus type pci registered May 16 23:03:20 client-5 kernel: PCI: Using MMCONFIG at e0000000 May 16 23:03:20 client-5 kernel: ACPI: Interpreter enabled May 16 23:03:20 client-5 kernel: ACPI: Using IOAPIC for interrupt routing May 16 23:03:20 client-5 kernel: ACPI: No dock devices found. May 16 23:03:20 client-5 kernel: ACPI: PCI Root Bridge [PCI0] (0000:00) May 16 23:03:20 client-5 kernel: PCI: Transparent bridge - 0000:00:1e.0 May 16 23:03:20 client-5 kernel: ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 7 10 11 12 14 *15) May 16 23:03:20 client-5 kernel: ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 *5 6 7 10 11 12 14 15) May 16 23:03:20 client-5 kernel: ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 5 6 7 *10 11 12 14 15) May 16 23:03:20 client-5 kernel: ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 6 7 10 *11 12 14 15) May 16 23:03:20 client-5 kernel: ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 5 6 7 10 11 12 14 15) *0, disabled. May 16 23:03:20 client-5 kernel: ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 5 6 7 10 11 12 *14 15) May 16 23:03:20 client-5 kernel: ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 5 6 7 10 11 12 14 15) *0, disabled. May 16 23:03:20 client-5 kernel: ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 5 6 *7 10 11 12 14 15) May 16 23:03:20 client-5 kernel: Linux Plug and Play Support v0.97 (c) Adam Belay May 16 23:03:20 client-5 kernel: pnp: PnP ACPI init May 16 23:03:20 client-5 kernel: pnp: PnP ACPI: found 15 devices May 16 23:03:20 client-5 kernel: usbcore: registered new driver usbfs May 16 23:03:20 client-5 kernel: usbcore: registered new driver hub May 16 23:03:20 client-5 kernel: PCI: Using ACPI for IRQ routing May 16 23:03:20 client-5 kernel: PCI: If a device doesn't work, try "pci=routeirq". If it helps, post a report May 16 23:03:20 client-5 kernel: NetLabel: Initializing May 16 23:03:20 client-5 kernel: NetLabel: domain hash size = 128 May 16 23:03:20 client-5 kernel: NetLabel: protocols = UNLABELED CIPSOv4 May 16 23:03:20 client-5 kernel: NetLabel: unlabeled traffic allowed by default May 16 23:03:20 client-5 kernel: hpet0: at MMIO 0xfed00000 (virtual 0xffffffffff5fe000), IRQs 2, 8, 0, 0 May 16 23:03:20 client-5 kernel: hpet0: 4 64-bit timers, 14318180 Hz May 16 23:03:20 client-5 kernel: DMAR:Host address width 40 May 16 23:03:20 client-5 kernel: DMAR:DRHD base: 0x000000fbffe000 flags: 0x1 May 16 23:03:20 client-5 kernel: IOMMU fbffe000: ver 1:0 cap c90780106f0462 ecap f020f6 May 16 23:03:20 client-5 kernel: DMAR:RMRR base: 0x000000000e6000 end: 0x000000000e9fff May 16 23:03:20 client-5 kernel: DMAR:RMRR base: 0x000000bf7ec000 end: 0x000000bf7fffff May 16 23:03:20 client-5 kernel: DMAR:ATSR flags: 0x0 May 16 23:03:20 client-5 kernel: PCI-GART: No AMD northbridge found. May 16 23:03:20 client-5 kernel: pnp: 00:08: ioport range 0x164e-0x164f has been reserved May 16 23:03:20 client-5 kernel: pnp: 00:08: ioport range 0xa00-0xa0f has been reserved May 16 23:03:20 client-5 kernel: pnp: 00:0c: ioport range 0xca2-0xca3 has been reserved May 16 23:03:20 client-5 kernel: PCI: Bridge: 0000:00:01.0 May 16 23:03:20 client-5 kernel: IO window: e000-efff May 16 23:03:20 client-5 kernel: MEM window: fbc00000-fbcfffff May 16 23:03:20 client-5 kernel: PREFETCH window: disabled. May 16 23:03:20 client-5 kernel: PCI: Bridge: 0000:00:03.0 May 16 23:03:20 client-5 kernel: IO window: disabled. May 16 23:03:20 client-5 kernel: MEM window: fbd00000-fbdfffff May 16 23:03:20 client-5 kernel: PREFETCH window 0x00000000f8800000-0x00000000f8ffffff May 16 23:03:20 client-5 kernel: PCI: Bridge: 0000:00:05.0 May 16 23:03:20 client-5 kernel: IO window: disabled. May 16 23:03:20 client-5 kernel: MEM window: disabled. May 16 23:03:20 client-5 kernel: PREFETCH window: disabled. May 16 23:03:20 client-5 kernel: PCI: Bridge: 0000:00:07.0 May 16 23:03:20 client-5 kernel: IO window: disabled. May 16 23:03:20 client-5 kernel: MEM window: disabled. May 16 23:03:20 client-5 kernel: PREFETCH window: disabled. May 16 23:03:20 client-5 kernel: PCI: Bridge: 0000:00:1e.0 May 16 23:03:20 client-5 kernel: IO window: disabled. May 16 23:03:20 client-5 kernel: MEM window: faf00000-fb7fffff May 16 23:03:20 client-5 kernel: PREFETCH window 0x00000000f9000000-0x00000000f9ffffff May 16 23:03:20 client-5 kernel: NET: Registered protocol family 2 May 16 23:03:20 client-5 kernel: IP route cache hash table entries: 524288 (order: 10, 4194304 bytes) May 16 23:03:20 client-5 kernel: TCP established hash table entries: 262144 (order: 10, 4194304 bytes) May 16 23:03:20 client-5 kernel: TCP bind hash table entries: 65536 (order: 8, 1048576 bytes) May 16 23:03:20 client-5 kernel: TCP: Hash tables configured (established 262144 bind 65536) May 16 23:03:20 client-5 kernel: TCP reno registered May 16 23:03:20 client-5 kernel: audit: initializing netlink socket (disabled) May 16 23:03:20 client-5 kernel: type=2000 audit(1305586948.605:1): initialized May 16 23:03:20 client-5 kernel: Total HugeTLB memory allocated, 0 May 16 23:03:20 client-5 kernel: VFS: Disk quotas dquot_6.5.1 May 16 23:03:20 client-5 kernel: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) May 16 23:03:20 client-5 kernel: Initializing Cryptographic API May 16 23:03:20 client-5 kernel: alg: No test for crc32c (crc32c-generic) May 16 23:03:20 client-5 kernel: ksign: Installing public key data May 16 23:03:20 client-5 kernel: Loading keyring May 16 23:03:20 client-5 kernel: - Added public key 565662CFF69DD0BE May 16 23:03:20 client-5 kernel: - User ID: CentOS (Kernel Module GPG key) May 16 23:03:20 client-5 kernel: io scheduler noop registered May 16 23:03:20 client-5 kernel: io scheduler anticipatory registered May 16 23:03:20 client-5 kernel: io scheduler deadline registered May 16 23:03:20 client-5 kernel: io scheduler cfq registered (default) May 16 23:03:20 client-5 kernel: pci_hotplug: PCI Hot Plug PCI Core version: 0.5 May 16 23:03:20 client-5 kernel: ACPI (exconfig-0456): Dynamic SSDT Load - OemId [DpgPmm] OemTableId [ P001Ist] [20060707] May 16 23:03:20 client-5 kernel: ACPI (exconfig-0456): Dynamic SSDT Load - OemId [ PmRef] OemTableId [ P001Cst] [20060707] May 16 23:03:20 client-5 kernel: ACPI: CPU0 (power states: C1[C1] C2[C2] C3[C3]) May 16 23:03:20 client-5 kernel: ACPI: CPU1 (power states: C1[C1] C2[C2] C3[C3]) May 16 23:03:20 client-5 kernel: ACPI: CPU2 (power states: C1[C1] C2[C2] C3[C3]) May 16 23:03:20 client-5 kernel: ACPI: CPU3 (power states: C1[C1] C2[C2] C3[C3]) May 16 23:03:20 client-5 kernel: Real Time Clock Driver v1.12ac May 16 23:03:20 client-5 kernel: Non-volatile memory driver v1.2 May 16 23:03:20 client-5 kernel: Linux agpgart interface v0.101 (c) Dave Jones May 16 23:03:20 client-5 kernel: Serial: 8250/16550 driver $Revision: 1.90 $ 4 ports, IRQ sharing enabled May 16 23:03:20 client-5 kernel: serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A May 16 23:03:20 client-5 kernel: serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A May 16 23:03:20 client-5 kernel: 00:06: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A May 16 23:03:20 client-5 kernel: 00:07: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A May 16 23:03:20 client-5 kernel: brd: module loaded May 16 23:03:20 client-5 kernel: Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2 May 16 23:03:20 client-5 kernel: ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx May 16 23:03:20 client-5 kernel: ide-floppy driver 0.99.newide May 16 23:03:20 client-5 kernel: usbcore: registered new driver hiddev May 16 23:03:20 client-5 kernel: usbcore: registered new driver usbhid May 16 23:03:20 client-5 kernel: drivers/usb/input/hid-core.c: v2.6:USB HID core driver May 16 23:03:20 client-5 kernel: PNP: No PS/2 controller found. Probing ports directly. May 16 23:03:20 client-5 kernel: serio: i8042 KBD port at 0x60,0x64 irq 1 May 16 23:03:20 client-5 kernel: serio: i8042 AUX port at 0x60,0x64 irq 12 May 16 23:03:20 client-5 kernel: mice: PS/2 mouse device common for all mice May 16 23:03:20 client-5 kernel: md: md driver 0.90.3 MAX_MD_DEVS=256, MD_SB_DISKS=27 May 16 23:03:20 client-5 kernel: md: bitmap version 4.39 May 16 23:03:20 client-5 kernel: TCP bic registered May 16 23:03:20 client-5 kernel: Initializing IPsec netlink socket May 16 23:03:20 client-5 kernel: NET: Registered protocol family 1 May 16 23:03:20 client-5 kernel: NET: Registered protocol family 17 May 16 23:03:20 client-5 kernel: ACPI: (supports S0 S1 S4 S5) May 16 23:03:20 client-5 kernel: Initalizing network drop monitor service May 16 23:03:20 client-5 kernel: Freeing unused kernel memory: 212k freed May 16 23:03:20 client-5 kernel: Write protecting the kernel read-only data: 504k May 16 23:03:20 client-5 kernel: GSI 16 sharing vector 0xD1 and IRQ 16 May 16 23:03:20 client-5 kernel: ACPI: PCI Interrupt 0000:00:1a.7[C] -> GSI 18 (level, low) -> IRQ 209 May 16 23:03:20 client-5 kernel: ehci_hcd 0000:00:1a.7: EHCI Host Controller May 16 23:03:20 client-5 kernel: ehci_hcd 0000:00:1a.7: new USB bus registered, assigned bus number 1 May 16 23:03:20 client-5 kernel: ehci_hcd 0000:00:1a.7: debug port 1 May 16 23:03:20 client-5 kernel: ehci_hcd 0000:00:1a.7: irq 209, io mem 0xfbed6000 May 16 23:03:20 client-5 kernel: ehci_hcd 0000:00:1a.7: USB 2.0 started, EHCI 1.00, driver 10 Dec 2004 May 16 23:03:20 client-5 kernel: usb usb1: configuration #1 chosen from 1 choice May 16 23:03:20 client-5 kernel: hub 1-0:1.0: USB hub found May 16 23:03:20 client-5 kernel: hub 1-0:1.0: 6 ports detected May 16 23:03:20 client-5 kernel: GSI 17 sharing vector 0xD9 and IRQ 17 May 16 23:03:20 client-5 kernel: ACPI: PCI Interrupt 0000:00:1d.7[A] -> GSI 23 (level, low) -> IRQ 217 May 16 23:03:20 client-5 kernel: ehci_hcd 0000:00:1d.7: EHCI Host Controller May 16 23:03:20 client-5 kernel: ehci_hcd 0000:00:1d.7: new USB bus registered, assigned bus number 2 May 16 23:03:20 client-5 kernel: ehci_hcd 0000:00:1d.7: debug port 1 May 16 23:03:20 client-5 kernel: ehci_hcd 0000:00:1d.7: irq 217, io mem 0xfbed4000 May 16 23:03:20 client-5 kernel: ehci_hcd 0000:00:1d.7: USB 2.0 started, EHCI 1.00, driver 10 Dec 2004 May 16 23:03:20 client-5 kernel: usb usb2: configuration #1 chosen from 1 choice May 16 23:03:20 client-5 kernel: hub 2-0:1.0: USB hub found May 16 23:03:20 client-5 kernel: hub 2-0:1.0: 6 ports detected May 16 23:03:20 client-5 kernel: USB Universal Host Controller Interface driver v3.0 May 16 23:03:20 client-5 kernel: GSI 18 sharing vector 0xE1 and IRQ 18 May 16 23:03:20 client-5 kernel: ACPI: PCI Interrupt 0000:00:1a.0[A] -> GSI 16 (level, low) -> IRQ 225 May 16 23:03:20 client-5 kernel: uhci_hcd 0000:00:1a.0: UHCI Host Controller May 16 23:03:20 client-5 kernel: uhci_hcd 0000:00:1a.0: new USB bus registered, assigned bus number 3 May 16 23:03:20 client-5 kernel: uhci_hcd 0000:00:1a.0: irq 225, io base 0x0000c880 May 16 23:03:20 client-5 kernel: usb usb3: configuration #1 chosen from 1 choice May 16 23:03:20 client-5 kernel: hub 3-0:1.0: USB hub found May 16 23:03:20 client-5 kernel: hub 3-0:1.0: 2 ports detected May 16 23:03:20 client-5 kernel: GSI 19 sharing vector 0xE9 and IRQ 19 May 16 23:03:20 client-5 kernel: ACPI: PCI Interrupt 0000:00:1a.1[B] -> GSI 21 (level, low) -> IRQ 233 May 16 23:03:20 client-5 kernel: uhci_hcd 0000:00:1a.1: UHCI Host Controller May 16 23:03:20 client-5 kernel: uhci_hcd 0000:00:1a.1: new USB bus registered, assigned bus number 4 May 16 23:03:20 client-5 kernel: uhci_hcd 0000:00:1a.1: irq 233, io base 0x0000cc00 May 16 23:03:20 client-5 kernel: usb usb4: configuration #1 chosen from 1 choice May 16 23:03:20 client-5 kernel: hub 4-0:1.0: USB hub found May 16 23:03:20 client-5 kernel: hub 4-0:1.0: 2 ports detected May 16 23:03:20 client-5 kernel: GSI 20 sharing vector 0x32 and IRQ 20 May 16 23:03:20 client-5 kernel: ACPI: PCI Interrupt 0000:00:1a.2[D] -> GSI 19 (level, low) -> IRQ 50 May 16 23:03:20 client-5 kernel: uhci_hcd 0000:00:1a.2: UHCI Host Controller May 16 23:03:20 client-5 kernel: uhci_hcd 0000:00:1a.2: new USB bus registered, assigned bus number 5 May 16 23:03:20 client-5 kernel: uhci_hcd 0000:00:1a.2: irq 50, io base 0x0000d000 May 16 23:03:20 client-5 kernel: usb usb5: configuration #1 chosen from 1 choice May 16 23:03:20 client-5 kernel: hub 5-0:1.0: USB hub found May 16 23:03:20 client-5 kernel: hub 5-0:1.0: 2 ports detected May 16 23:03:20 client-5 kernel: ACPI: PCI Interrupt 0000:00:1d.0[A] -> GSI 23 (level, low) -> IRQ 217 May 16 23:03:20 client-5 kernel: uhci_hcd 0000:00:1d.0: UHCI Host Controller May 16 23:03:20 client-5 kernel: uhci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 6 May 16 23:03:20 client-5 kernel: uhci_hcd 0000:00:1d.0: irq 217, io base 0x0000c400 May 16 23:03:20 client-5 kernel: usb usb6: configuration #1 chosen from 1 choice May 16 23:03:20 client-5 kernel: hub 6-0:1.0: USB hub found May 16 23:03:20 client-5 kernel: hub 6-0:1.0: 2 ports detected May 16 23:03:20 client-5 kernel: usb 4-1: new full speed USB device using uhci_hcd and address 2 May 16 23:03:20 client-5 kernel: ACPI: PCI Interrupt 0000:00:1d.1[B] -> GSI 19 (level, low) -> IRQ 50 May 16 23:03:20 client-5 kernel: uhci_hcd 0000:00:1d.1: UHCI Host Controller May 16 23:03:20 client-5 kernel: uhci_hcd 0000:00:1d.1: new USB bus registered, assigned bus number 7 May 16 23:03:20 client-5 kernel: uhci_hcd 0000:00:1d.1: irq 50, io base 0x0000c480 May 16 23:03:20 client-5 kernel: usb usb7: configuration #1 chosen from 1 choice May 16 23:03:20 client-5 kernel: hub 7-0:1.0: USB hub found May 16 23:03:20 client-5 kernel: hub 7-0:1.0: 2 ports detected May 16 23:03:20 client-5 kernel: usb 4-1: configuration #1 chosen from 1 choice May 16 23:03:20 client-5 kernel: ACPI: PCI Interrupt 0000:00:1d.2[C] -> GSI 18 (level, low) -> IRQ 209 May 16 23:03:20 client-5 rpc.statd[3121]: Version 1.0.9 Starting May 16 23:03:20 client-5 kernel: uhci_hcd 0000:00:1d.2: UHCI Host Controller May 16 23:03:20 client-5 kernel: uhci_hcd 0000:00:1d.2: new USB bus registered, assigned bus number 8 May 16 23:03:20 client-5 kernel: uhci_hcd 0000:00:1d.2: irq 209, io base 0x0000c800 May 16 23:03:20 client-5 kernel: usb usb8: configuration #1 chosen from 1 choice May 16 23:03:20 client-5 kernel: hub 8-0:1.0: USB hub found May 16 23:03:20 client-5 kernel: hub 8-0:1.0: 2 ports detected May 16 23:03:20 client-5 kernel: input: American Megatrends Inc. Virtual Keyboard and Mouse as /class/input/input0 May 16 23:03:20 client-5 kernel: input: USB HID v1.10 Keyboard [American Megatrends Inc. Virtual Keyboard and Mouse] on usb-0000:00:1a.1-1 May 16 23:03:20 client-5 kernel: input: American Megatrends Inc. Virtual Keyboard and Mouse as /class/input/input1 May 16 23:03:20 client-5 kernel: input: USB HID v1.10 Mouse [American Megatrends Inc. Virtual Keyboard and Mouse] on usb-0000:00:1a.1-1 May 16 23:03:20 client-5 kernel: SCSI subsystem initialized May 16 23:03:20 client-5 kernel: ACPI: PCI Interrupt 0000:00:1f.2[B] -> GSI 19 (level, low) -> IRQ 50 May 16 23:03:20 client-5 kernel: ahci 0000:00:1f.2: AHCI 0001.0200 32 slots 6 ports 3 Gbps 0x3f impl SATA mode May 16 23:03:20 client-5 kernel: ahci 0000:00:1f.2: flags: 64bit ncq sntf stag pm led clo pio slum part ems May 16 23:03:20 client-5 kernel: scsi0 : ahci May 16 23:03:20 client-5 kernel: scsi1 : ahci May 16 23:03:20 client-5 kernel: scsi2 : ahci May 16 23:03:20 client-5 kernel: scsi3 : ahci May 16 23:03:20 client-5 rpc.statd[3121]: statd running as root. chown /var/lib/nfs/statd/sm to choose different user May 16 23:03:20 client-5 kernel: scsi4 : ahci May 16 23:03:20 client-5 kernel: scsi5 : ahci May 16 23:03:20 client-5 kernel: ata1: SATA max UDMA/133 abar m2048@0xfbefa000 port 0xfbefa100 irq 58 May 16 23:03:20 client-5 kernel: ata2: SATA max UDMA/133 abar m2048@0xfbefa000 port 0xfbefa180 irq 58 May 16 23:03:20 client-5 kernel: ata3: SATA max UDMA/133 abar m2048@0xfbefa000 port 0xfbefa200 irq 58 May 16 23:03:20 client-5 kernel: ata4: SATA max UDMA/133 abar m2048@0xfbefa000 port 0xfbefa280 irq 58 May 16 23:03:20 client-5 kernel: ata5: SATA max UDMA/133 abar m2048@0xfbefa000 port 0xfbefa300 irq 58 May 16 23:03:20 client-5 kernel: ata6: SATA max UDMA/133 abar m2048@0xfbefa000 port 0xfbefa380 irq 58 May 16 23:03:20 client-5 kernel: ata1: SATA link up 3.0 Gbps (SStatus 123 SControl 300) May 16 23:03:20 client-5 kernel: ata1.00: ATA-8: WDC WD2502ABYS-02B7A0, 02.03B03, max UDMA/133 May 16 23:03:20 client-5 kernel: ata1.00: 490350672 sectors, multi 0: LBA48 NCQ (depth 31/32) May 16 23:03:20 client-5 kernel: ata1.00: configured for UDMA/133 May 16 23:03:20 client-5 kernel: ata2: SATA link down (SStatus 0 SControl 300) May 16 23:03:20 client-5 kernel: ata3: SATA link down (SStatus 0 SControl 300) May 16 23:03:20 client-5 kernel: ata4: SATA link down (SStatus 0 SControl 300) May 16 23:03:20 client-5 kernel: ata5: SATA link down (SStatus 0 SControl 300) May 16 23:03:20 client-5 kernel: ata6: SATA link down (SStatus 0 SControl 300) May 16 23:03:20 client-5 kernel: Vendor: ATA Model: WDC WD2502ABYS-0 Rev: 02.0 May 16 23:03:20 client-5 kernel: Type: Direct-Access ANSI SCSI revision: 05 May 16 23:03:20 client-5 kernel: SCSI device sda: 490350672 512-byte hdwr sectors (251060 MB) May 16 23:03:20 client-5 kernel: sda: Write Protect is off May 16 23:03:20 client-5 kernel: SCSI device sda: drive cache: write back May 16 23:03:20 client-5 kernel: SCSI device sda: 490350672 512-byte hdwr sectors (251060 MB) May 16 23:03:20 client-5 kernel: sda: Write Protect is off May 16 23:03:20 client-5 kernel: SCSI device sda: drive cache: write back May 16 23:03:20 client-5 kernel: sda: sda1 sda2 May 16 23:03:20 client-5 kernel: sd 0:0:0:0: Attached scsi disk sda May 16 23:03:20 client-5 kernel: device-mapper: uevent: version 1.0.3 May 16 23:03:20 client-5 kernel: device-mapper: ioctl: 4.11.5-ioctl (2007-12-12) initialised: dm-devel@redhat.com May 16 23:03:20 client-5 kernel: device-mapper: dm-raid45: initialized v0.2594l May 16 23:03:20 client-5 kernel: kjournald starting. Commit interval 5 seconds May 16 23:03:20 client-5 kernel: EXT3-fs: mounted filesystem with ordered data mode. May 16 23:03:20 client-5 kernel: SELinux: Disabled at runtime. May 16 23:03:20 client-5 kernel: type=1404 audit(1305586975.124:2): selinux=0 auid=4294967295 ses=4294967295 May 16 23:03:20 client-5 kernel: input: PC Speaker as /class/input/input2 May 16 23:03:20 client-5 kernel: dca service started, version 1.4 May 16 23:03:20 client-5 kernel: shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 May 16 23:03:20 client-5 kernel: 802.1Q VLAN Support v1.8 Ben Greear May 16 23:03:20 client-5 kernel: All bugs added by David S. Miller May 16 23:03:20 client-5 kernel: sd 0:0:0:0: Attached scsi generic sg0 type 0 May 16 23:03:20 client-5 kernel: mlx4_core: Mellanox ConnectX core driver v1.0-ofed1.5.3 (January 19, 2011) May 16 23:03:20 client-5 kernel: mlx4_core: Initializing 0000:02:00.0 May 16 23:03:20 client-5 kernel: GSI 21 sharing vector 0x42 and IRQ 21 May 16 23:03:20 client-5 kernel: ACPI: PCI Interrupt 0000:02:00.0[A] -> GSI 24 (level, low) -> IRQ 66 May 16 23:03:20 client-5 kernel: Intel(R) Gigabit Ethernet Network Driver - version 2.1.0-k2 May 16 23:03:20 client-5 kernel: Copyright (c) 2007-2009 Intel Corporation. May 16 23:03:20 client-5 kernel: GSI 22 sharing vector 0x53 and IRQ 22 May 16 23:03:20 client-5 kernel: ACPI: PCI Interrupt 0000:01:00.0[A] -> GSI 28 (level, low) -> IRQ 83 May 16 23:03:20 client-5 kernel: mlx4_ib: Mellanox ConnectX InfiniBand driver v1.0-ofed1.5.3 (January 19, 2011) May 16 23:03:20 client-5 kernel: igb 0000:01:00.0: Intel(R) Gigabit Ethernet Network Connection May 16 23:03:20 client-5 kernel: igb 0000:01:00.0: eth0: (PCIe:2.5Gb/s:Width x4) 00:30:48:f7:72:24 May 16 23:03:20 client-5 kernel: igb 0000:01:00.0: eth0: PBA No: ffffff-0ff May 16 23:03:20 client-5 kernel: igb 0000:01:00.0: Using MSI-X interrupts. 4 rx queue(s), 1 tx queue(s) May 16 23:03:20 client-5 kernel: GSI 23 sharing vector 0x8B and IRQ 23 May 16 23:03:20 client-5 kernel: ACPI: PCI Interrupt 0000:01:00.1[B] -> GSI 40 (level, low) -> IRQ 139 May 16 23:03:20 client-5 kernel: igb 0000:01:00.1: Intel(R) Gigabit Ethernet Network Connection May 16 23:03:20 client-5 kernel: igb 0000:01:00.1: eth1: (PCIe:2.5Gb/s:Width x4) 00:30:48:f7:72:25 May 16 23:03:20 client-5 kernel: igb 0000:01:00.1: eth1: PBA No: ffffff-0ff May 16 23:03:20 client-5 kernel: igb 0000:01:00.1: Using MSI-X interrupts. 4 rx queue(s), 1 tx queue(s) May 16 23:03:20 client-5 kernel: ACPI: PCI Interrupt 0000:00:1f.3[C] -> GSI 18 (level, low) -> IRQ 209 May 16 23:03:20 client-5 kernel: floppy0: no floppy controllers found May 16 23:03:20 client-5 kernel: lp: driver loaded but no devices found May 16 23:03:20 client-5 kernel: ACPI: Power Button (FF) [PWRF] May 16 23:03:20 client-5 kernel: ACPI: Power Button (CM) [PWRB] May 16 23:03:20 client-5 kernel: ACPI: Mapper loaded May 16 23:03:20 client-5 kernel: dell-wmi: No known WMI GUID found May 16 23:03:20 client-5 kernel: md: Autodetecting RAID arrays. May 16 23:03:20 client-5 kernel: md: autorun ... May 16 23:03:20 client-5 kernel: md: ... autorun DONE. May 16 23:03:20 client-5 kernel: device-mapper: multipath: version 1.0.5 loaded May 16 23:03:20 client-5 kernel: loop: loaded (max 8 devices) May 16 23:03:20 client-5 kernel: EXT3 FS on sda1, internal journal May 16 23:03:20 client-5 kernel: Adding 14346036k swap on /dev/sda2. Priority:-1 extents:1 across:14346036k May 16 23:03:20 client-5 kernel: IA-32 Microcode Update Driver: v1.14a May 16 23:03:20 client-5 kernel: mlx4_en: Mellanox ConnectX HCA Ethernet driver v1.5.3 (Jan 2011) May 16 23:03:21 client-5 kernel: mlx4_en 0000:02:00.0: UDP RSS is not supported on this device. May 16 23:03:21 client-5 kernel: NET: Registered protocol family 10 May 16 23:03:21 client-5 kernel: lo: Disabled Privacy Extensions May 16 23:03:21 client-5 kernel: IPv6 over IPv4 tunneling driver May 16 23:03:21 client-5 kernel: ADDRCONF(NETDEV_UP): ib0: link is not ready May 16 23:03:21 client-5 kernel: ib0: enabling connected mode will cause multicast packet drops May 16 23:03:21 client-5 kernel: ib0: mtu > 2044 will cause multicast packet drops. May 16 23:03:21 client-5 kernel: ib0: mtu > 2044 will cause multicast packet drops. May 16 23:03:21 client-5 kernel: NET: Registered protocol family 27 May 16 23:03:21 client-5 kernel: NET: Registered protocol family 28 May 16 23:03:21 client-5 kernel: QLGC_VNIC: Initializing QLogic Corp. Virtual NIC (VNIC) driver version 1.3.0.0.4 May 16 23:03:21 client-5 kernel: Loading iSCSI transport class v2.0-871. May 16 23:03:21 client-5 kernel: cxgb3i: disagrees about version of symbol cxgb3_register_client May 16 23:03:21 client-5 kernel: cxgb3i: Unknown symbol cxgb3_register_client May 16 23:03:21 client-5 kernel: cxgb3i: disagrees about version of symbol cxgb3_alloc_atid May 16 23:03:21 client-5 kernel: cxgb3i: Unknown symbol cxgb3_alloc_atid May 16 23:03:21 client-5 kernel: cxgb3i: disagrees about version of symbol t3_l2t_get May 16 23:03:21 client-5 kernel: cxgb3i: Unknown symbol t3_l2t_get May 16 23:03:21 client-5 kernel: cxgb3i: disagrees about version of symbol cxgb3_insert_tid May 16 23:03:21 client-5 kernel: cxgb3i: Unknown symbol cxgb3_insert_tid May 16 23:03:21 client-5 kernel: cxgb3i: disagrees about version of symbol t3_l2e_free May 16 23:03:21 client-5 kernel: cxgb3i: Unknown symbol t3_l2e_free May 16 23:03:21 client-5 kernel: cxgb3i: disagrees about version of symbol t3_l2t_send_slow May 16 23:03:21 client-5 kernel: cxgb3i: Unknown symbol t3_l2t_send_slow May 16 23:03:21 client-5 kernel: cxgb3i: disagrees about version of symbol cxgb3_unregister_client May 16 23:03:21 client-5 kernel: cxgb3i: Unknown symbol cxgb3_unregister_client May 16 23:03:21 client-5 kernel: Broadcom NetXtreme II CNIC Driver cnic v2.1.0 (Oct 10, 2009) May 16 23:03:21 client-5 kernel: Broadcom NetXtreme II iSCSI Driver bnx2i v2.1.0 (Dec 06, 2009) May 16 23:03:21 client-5 kernel: iscsi: registered transport (bnx2i) May 16 23:03:21 client-5 kernel: iscsi: registered transport (tcp) May 16 23:03:21 client-5 kernel: ib_iser: disagrees about version of symbol ib_fmr_pool_unmap May 16 23:03:21 client-5 kernel: ib_iser: Unknown symbol ib_fmr_pool_unmap May 16 23:03:21 client-5 kernel: ib_iser: disagrees about version of symbol ib_create_cq May 16 23:03:21 client-5 kernel: ib_iser: Unknown symbol ib_create_cq May 16 23:03:21 client-5 kernel: ib_iser: disagrees about version of symbol rdma_resolve_addr May 16 23:03:21 client-5 kernel: ib_iser: Unknown symbol rdma_resolve_addr May 16 23:03:21 client-5 kernel: ib_iser: disagrees about version of symbol ib_create_fmr_pool May 16 23:03:21 client-5 kernel: ib_iser: Unknown symbol ib_create_fmr_pool May 16 23:03:21 client-5 kernel: ib_iser: disagrees about version of symbol ib_dereg_mr May 16 23:03:21 client-5 kernel: ib_iser: Unknown symbol ib_dereg_mr May 16 23:03:21 client-5 kernel: ib_iser: disagrees about version of symbol rdma_disconnect May 16 23:03:21 client-5 kernel: ib_iser: Unknown symbol rdma_disconnect May 16 23:03:21 client-5 kernel: ib_iser: disagrees about version of symbol rdma_resolve_route May 16 23:03:21 client-5 kernel: ib_iser: Unknown symbol rdma_resolve_route May 16 23:03:21 client-5 kernel: ib_iser: disagrees about version of symbol rdma_create_qp May 16 23:03:21 client-5 kernel: ib_iser: Unknown symbol rdma_create_qp May 16 23:03:21 client-5 kernel: ib_iser: disagrees about version of symbol ib_destroy_cq May 16 23:03:21 client-5 kernel: ib_iser: Unknown symbol ib_destroy_cq May 16 23:03:21 client-5 kernel: ib_iser: disagrees about version of symbol rdma_create_id May 16 23:03:21 client-5 kernel: ib_iser: Unknown symbol rdma_create_id May 16 23:03:21 client-5 kernel: ib_iser: disagrees about version of symbol rdma_destroy_qp May 16 23:03:21 client-5 kernel: ib_iser: Unknown symbol rdma_destroy_qp May 16 23:03:21 client-5 kernel: ib_iser: disagrees about version of symbol ib_get_dma_mr May 16 23:03:21 client-5 kernel: ib_iser: Unknown symbol ib_get_dma_mr May 16 23:03:21 client-5 kernel: ib_iser: disagrees about version of symbol ib_alloc_pd May 16 23:03:21 client-5 kernel: ib_iser: Unknown symbol ib_alloc_pd May 16 23:03:21 client-5 kernel: ib_iser: disagrees about version of symbol rdma_connect May 16 23:03:21 client-5 kernel: ib_iser: Unknown symbol rdma_connect May 16 23:03:21 client-5 kernel: ib_iser: disagrees about version of symbol rdma_destroy_id May 16 23:03:21 client-5 kernel: ib_iser: Unknown symbol rdma_destroy_id May 16 23:03:21 client-5 kernel: ib_iser: disagrees about version of symbol ib_dealloc_pd May 16 23:03:21 client-5 kernel: ib_iser: Unknown symbol ib_dealloc_pd May 16 23:03:21 client-5 kernel: ib_iser: disagrees about version of symbol ib_fmr_pool_map_phys May 16 23:03:21 client-5 kernel: ib_iser: Unknown symbol ib_fmr_pool_map_phys May 16 23:03:21 client-5 kernel: iscsi: registered transport (be2iscsi) May 16 23:03:21 client-5 kernel: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX May 16 23:03:21 client-5 kernel: ADDRCONF(NETDEV_UP): eth1: link is not ready May 16 23:03:21 client-5 hcid[3164]: Bluetooth HCI daemon May 16 23:03:21 client-5 kernel: Bluetooth: Core ver 2.10 May 16 23:03:21 client-5 kernel: NET: Registered protocol family 31 May 16 23:03:21 client-5 kernel: Bluetooth: HCI device and connection manager initialized May 16 23:03:21 client-5 kernel: Bluetooth: HCI socket layer initialized May 16 23:03:21 client-5 hcid[3164]: Register path:/org/bluez fallback:1 May 16 23:03:21 client-5 kernel: Bluetooth: L2CAP ver 2.8 May 16 23:03:21 client-5 kernel: Bluetooth: L2CAP socket layer initialized May 16 23:03:21 client-5 sdpd[3170]: Bluetooth SDP daemon May 16 23:03:21 client-5 kernel: Bluetooth: RFCOMM socket layer initialized May 16 23:03:21 client-5 kernel: Bluetooth: RFCOMM TTY layer initialized May 16 23:03:21 client-5 kernel: Bluetooth: RFCOMM ver 1.8 May 16 23:03:21 client-5 pcscd: pcscdaemon.c:507:main() pcsc-lite 1.4.4 daemon ready. May 16 23:03:21 client-5 pcscd: hotplug_libusb.c:402:HPEstablishUSBNotifications() Driver ifd-egate.bundle does not support IFD_GENERATE_HOTPLUG. Using active polling instead. May 16 23:03:21 client-5 pcscd: hotplug_libusb.c:411:HPEstablishUSBNotifications() Polling forced every 1 second(s) May 16 23:03:22 client-5 kernel: ADDRCONF(NETDEV_CHANGE): ib0: link becomes ready May 16 23:03:22 client-5 kernel: Bluetooth: HIDP (Human Interface Emulation) ver 1.1 May 16 23:03:22 client-5 hidd[3294]: Bluetooth HID daemon May 16 23:03:23 client-5 automount[3323]: lookup_read_master: lookup(nisplus): couldn't locate nis+ table auto.master May 16 23:03:23 client-5 automount[3323]: lookup_read_map: lookup(nisplus): couldn't locate nis+ table auto.direct May 16 23:03:24 client-5 sssd: Starting up May 16 23:03:24 client-5 sssd[be[LAB.WHAMCLOUD.COM]]: Starting up May 16 23:03:24 client-5 sssd[nss]: Starting up May 16 23:03:24 client-5 sssd[pam]: Starting up May 16 23:03:25 client-5 ntpd[3395]: ntpd 4.2.2p1@1.1570-o Sat Dec 19 00:56:13 UTC 2009 (1) May 16 23:03:25 client-5 ntpd[3396]: precision = 1.000 usec May 16 23:03:25 client-5 ntpd[3396]: Listening on interface wildcard, 0.0.0.0#123 Disabled May 16 23:03:25 client-5 ntpd[3396]: Listening on interface wildcard, ::#123 Disabled May 16 23:03:25 client-5 ntpd[3396]: Listening on interface lo, ::1#123 Enabled May 16 23:03:25 client-5 ntpd[3396]: Listening on interface eth0, fe80::230:48ff:fef7:7224#123 Enabled May 16 23:03:25 client-5 ntpd[3396]: Listening on interface ib0, fe80::230:48ff:fff4:f571#123 Enabled May 16 23:03:25 client-5 ntpd[3396]: Listening on interface lo, 127.0.0.1#123 Enabled May 16 23:03:25 client-5 ntpd[3396]: Listening on interface eth0, 10.10.4.5#123 Enabled May 16 23:03:25 client-5 ntpd[3396]: Listening on interface eth1, 172.16.8.5#123 Enabled May 16 23:03:25 client-5 ntpd[3396]: Listening on interface ib0, 192.168.4.5#123 Enabled May 16 23:03:25 client-5 ntpd[3396]: kernel time sync status 0040 May 16 23:03:25 client-5 xinetd[3385]: xinetd Version 2.3.14 started with libwrap loadavg labeled-networking options compiled in. May 16 23:03:25 client-5 xinetd[3385]: Started working: 2 available services May 16 23:03:25 client-5 gpm[3431]: *** info [startup.c(95)]: May 16 23:03:25 client-5 gpm[3431]: Started gpm successfully. Entered daemon mode. May 16 23:03:25 client-5 smartd[3496]: smartd version 5.38 [x86_64-redhat-linux-gnu] Copyright (C) 2002-8 Bruce Allen May 16 23:03:25 client-5 smartd[3496]: Home page is http://smartmontools.sourceforge.net/ May 16 23:03:25 client-5 smartd[3496]: Opened configuration file /etc/smartd.conf May 16 23:03:25 client-5 smartd[3496]: Configuration file /etc/smartd.conf was parsed, found DEVICESCAN, scanning devices May 16 23:03:25 client-5 smartd[3496]: Problem creating device name scan list May 16 23:03:25 client-5 smartd[3496]: Device: /dev/sda, opened May 16 23:03:25 client-5 smartd[3496]: Device /dev/sda: using '-d sat' for ATA disk behind SAT layer. May 16 23:03:25 client-5 smartd[3496]: Device: /dev/sda, opened May 16 23:03:25 client-5 smartd[3496]: Device: /dev/sda, not found in smartd database. May 16 23:03:25 client-5 smartd[3496]: Device: /dev/sda, is SMART capable. Adding to "monitor" list. May 16 23:03:25 client-5 smartd[3496]: Monitoring 0 ATA and 1 SCSI devices May 16 23:03:25 client-5 smartd[3503]: smartd has fork()ed into background mode. New PID=3503. May 16 23:07:43 client-5 ntpd[3396]: synchronized to 10.10.0.2, stratum 3 May 17 00:07:43 client-5 ntpd[3396]: time reset +3600.016690 s May 17 00:12:50 client-5 ntpd[3396]: synchronized to 10.10.0.1, stratum 2 May 17 00:21:29 client-5 ntpd[3396]: kernel time sync enabled 0001 May 17 01:06:45 client-5 xinetd[3385]: START: shell pid=3820 from=10.10.0.1 May 17 01:06:45 client-5 rshd[3821]: sarah@brent.lab.whamcloud.com as root: cmd='cp /home/sarah/.maloorc /root' May 17 01:06:45 client-5 kernel: FS-Cache: Loaded May 17 01:06:45 client-5 xinetd[3385]: EXIT: shell status=0 pid=3820 duration=0(sec) May 17 01:07:02 client-5 xinetd[3385]: START: shell pid=3879 from=10.10.0.1 May 17 01:07:02 client-5 rshd[3880]: sarah@brent.lab.whamcloud.com as root: cmd='cp /home/sarah/backup/sarah.sh /usr/lib64/lustre/tests/cfg' May 17 01:07:02 client-5 xinetd[3385]: EXIT: shell status=0 pid=3879 duration=0(sec) May 17 01:07:20 client-5 xinetd[3385]: START: shell pid=3912 from=10.10.0.1 May 17 01:07:20 client-5 rshd[3913]: sarah@brent.lab.whamcloud.com as root: cmd='cp /home/sarah/backup/modprobe.conf /etc/' May 17 01:07:20 client-5 xinetd[3385]: EXIT: shell status=0 pid=3912 duration=0(sec) May 17 01:08:38 client-5 shutdown[8545]: shutting down for system halt May 17 01:08:38 client-5 init: Switching to runlevel: 0 May 17 01:08:39 client-5 smartd[3503]: smartd received signal 15: Terminated May 17 01:08:39 client-5 smartd[3503]: smartd is exiting (exit status 0) May 17 00:11:20 client-5 syslogd 1.4.1: restart. May 17 00:11:20 client-5 kernel: klogd 1.4.1, log source = /proc/kmsg started. May 17 00:11:20 client-5 kernel: Linux version 2.6.18-194.17.1.el5 (mockbuild@builder10.centos.org) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-48)) #1 SMP Wed Sep 29 12:50:31 EDT 2010 May 17 00:11:20 client-5 kernel: Command line: ro root=LABEL=/ console=ttyS0,115200 May 17 00:11:20 client-5 kernel: BIOS-provided physical RAM map: May 17 00:11:20 client-5 kernel: BIOS-e820: 0000000000010000 - 000000000009cc00 (usable) May 17 00:11:20 client-5 kernel: BIOS-e820: 000000000009cc00 - 00000000000a0000 (reserved) May 17 00:11:20 client-5 kernel: BIOS-e820: 00000000000e6000 - 0000000000100000 (reserved) May 17 00:11:20 client-5 kernel: BIOS-e820: 0000000000100000 - 00000000bf760000 (usable) May 17 00:11:20 client-5 kernel: BIOS-e820: 00000000bf76e000 - 00000000bf770000 type 9 May 17 00:11:20 client-5 kernel: BIOS-e820: 00000000bf770000 - 00000000bf77e000 (ACPI data) May 17 00:11:20 client-5 kernel: BIOS-e820: 00000000bf77e000 - 00000000bf7d0000 (ACPI NVS) May 17 00:11:20 client-5 kernel: BIOS-e820: 00000000bf7d0000 - 00000000bf7e0000 (reserved) May 17 00:11:20 client-5 kernel: BIOS-e820: 00000000bf7ec000 - 00000000c0000000 (reserved) May 17 00:11:20 client-5 kernel: BIOS-e820: 00000000e0000000 - 00000000f0000000 (reserved) May 17 00:11:20 client-5 kernel: BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved) May 17 00:11:20 client-5 kernel: BIOS-e820: 00000000ffc00000 - 0000000100000000 (reserved) May 17 00:11:20 client-5 kernel: BIOS-e820: 0000000100000000 - 0000000340000000 (usable) May 17 00:11:20 client-5 kernel: DMI present. May 17 00:11:20 client-5 kernel: No NUMA configuration found May 17 00:11:20 client-5 kernel: Faking a node at 0000000000000000-0000000340000000 May 17 00:11:20 client-5 kernel: Bootmem setup node 0 0000000000000000-0000000340000000 May 17 00:11:20 client-5 kernel: Memory for crash kernel (0x0 to 0x0) notwithin permissible range May 17 00:11:20 client-5 kernel: disabling kdump May 17 00:11:20 client-5 kernel: ACPI: PM-Timer IO Port: 0x808 May 17 00:11:20 client-5 kernel: ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled) May 17 00:11:20 client-5 kernel: Processor #0 7:10 APIC version 21 May 17 00:11:20 client-5 kernel: ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled) May 17 00:11:20 client-5 kernel: Processor #2 7:10 APIC version 21 May 17 00:11:20 client-5 kernel: ACPI: LAPIC (acpi_id[0x03] lapic_id[0x04] enabled) May 17 00:11:20 client-5 kernel: Processor #4 7:10 APIC version 21 May 17 00:11:20 client-5 kernel: ACPI: LAPIC (acpi_id[0x04] lapic_id[0x06] enabled) May 17 00:11:20 client-5 kernel: Processor #6 7:10 APIC version 21 May 17 00:11:20 client-5 kernel: ACPI: LAPIC (acpi_id[0x05] lapic_id[0x84] disabled) May 17 00:11:20 client-5 kernel: ACPI: LAPIC (acpi_id[0x06] lapic_id[0x85] disabled) May 17 00:11:20 client-5 kernel: ACPI: LAPIC (acpi_id[0x07] lapic_id[0x86] disabled) May 17 00:11:20 client-5 kernel: ACPI: LAPIC (acpi_id[0x08] lapic_id[0x87] disabled) May 17 00:11:20 client-5 kernel: ACPI: LAPIC (acpi_id[0x09] lapic_id[0x88] disabled) May 17 00:11:20 client-5 kernel: ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x89] disabled) May 17 00:11:20 client-5 kernel: ACPI: LAPIC (acpi_id[0x0b] lapic_id[0x8a] disabled) May 17 00:11:20 client-5 kernel: ACPI: LAPIC (acpi_id[0x0c] lapic_id[0x8b] disabled) May 17 00:11:20 client-5 kernel: ACPI: LAPIC (acpi_id[0x0d] lapic_id[0x8c] disabled) May 17 00:11:20 client-5 kernel: ACPI: LAPIC (acpi_id[0x0e] lapic_id[0x8d] disabled) May 17 00:11:20 client-5 kernel: ACPI: LAPIC (acpi_id[0x0f] lapic_id[0x8e] disabled) May 17 00:11:20 client-5 kernel: ACPI: LAPIC (acpi_id[0x10] lapic_id[0x8f] disabled) May 17 00:11:20 client-5 kernel: ACPI: LAPIC (acpi_id[0x11] lapic_id[0x90] disabled) May 17 00:11:20 client-5 kernel: ACPI: LAPIC (acpi_id[0x12] lapic_id[0x91] disabled) May 17 00:11:20 client-5 kernel: ACPI: LAPIC (acpi_id[0x13] lapic_id[0x92] disabled) May 17 00:11:20 client-5 kernel: ACPI: LAPIC (acpi_id[0x14] lapic_id[0x93] disabled) May 17 00:11:20 client-5 kernel: ACPI: LAPIC (acpi_id[0x15] lapic_id[0x94] disabled) May 17 00:11:20 client-5 kernel: ACPI: LAPIC (acpi_id[0x16] lapic_id[0x95] disabled) May 17 00:11:20 client-5 kernel: ACPI: LAPIC (acpi_id[0x17] lapic_id[0x96] disabled) May 17 00:11:20 client-5 kernel: ACPI: LAPIC (acpi_id[0x18] lapic_id[0x97] disabled) May 17 00:11:20 client-5 kernel: ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) May 17 00:11:20 client-5 kernel: ACPI: IOAPIC (id[0x01] address[0xfec00000] gsi_base[0]) May 17 00:11:20 client-5 kernel: IOAPIC[0]: apic_id 1, version 32, address 0xfec00000, GSI 0-23 May 17 00:11:20 client-5 kernel: ACPI: IOAPIC (id[0x03] address[0xfec8a000] gsi_base[24]) May 17 00:11:20 client-5 kernel: IOAPIC[1]: apic_id 3, version 32, address 0xfec8a000, GSI 24-47 May 17 00:11:20 client-5 cpuspeed: Enabling ondemand cpu frequency scaling governor May 17 00:11:20 client-5 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) May 17 00:11:20 client-5 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) May 17 00:11:20 client-5 kernel: Setting APIC routing to physical flat May 17 00:11:20 client-5 kernel: ACPI: HPET id: 0x8086a301 base: 0xfed00000 May 17 00:11:20 client-5 kernel: Using ACPI (MADT) for SMP configuration information May 17 00:11:20 client-5 kernel: Nosave address range: 000000000009c000 - 000000000009d000 May 17 00:11:20 client-5 kernel: Nosave address range: 000000000009d000 - 00000000000a0000 May 17 00:11:20 client-5 kernel: Nosave address range: 00000000000a0000 - 00000000000e6000 May 17 00:11:20 client-5 kernel: Nosave address range: 00000000000e6000 - 0000000000100000 May 17 00:11:20 client-5 kernel: Nosave address range: 00000000bf760000 - 00000000bf76e000 May 17 00:11:20 client-5 kernel: Nosave address range: 00000000bf76e000 - 00000000bf770000 May 17 00:11:20 client-5 kernel: Nosave address range: 00000000bf770000 - 00000000bf77e000 May 17 00:11:20 client-5 kernel: Nosave address range: 00000000bf77e000 - 00000000bf7d0000 May 17 00:11:20 client-5 kernel: Nosave address range: 00000000bf7d0000 - 00000000bf7e0000 May 17 00:11:20 client-5 kernel: Nosave address range: 00000000bf7e0000 - 00000000bf7ec000 May 17 00:11:20 client-5 kernel: Nosave address range: 00000000bf7ec000 - 00000000c0000000 May 17 00:11:20 client-5 kernel: Nosave address range: 00000000c0000000 - 00000000e0000000 May 17 00:11:20 client-5 kernel: Nosave address range: 00000000e0000000 - 00000000f0000000 May 17 00:11:20 client-5 kernel: Nosave address range: 00000000f0000000 - 00000000fee00000 May 17 00:11:20 client-5 kernel: Nosave address range: 00000000fee00000 - 00000000fee01000 May 17 00:11:20 client-5 kernel: Nosave address range: 00000000fee01000 - 00000000ffc00000 May 17 00:11:20 client-5 kernel: Nosave address range: 00000000ffc00000 - 0000000100000000 May 17 00:11:20 client-5 kernel: Allocating PCI resources starting at c2000000 (gap: c0000000:20000000) May 17 00:11:20 client-5 kernel: SMP: Allowing 24 CPUs, 20 hotplug CPUs May 17 00:11:20 client-5 kernel: Built 1 zonelists. Total pages: 3095502 May 17 00:11:20 client-5 kernel: Kernel command line: ro root=LABEL=/ console=ttyS0,115200 May 17 00:11:20 client-5 kernel: Initializing CPU#0 May 17 00:11:20 client-5 kernel: PID hash table entries: 4096 (order: 12, 32768 bytes) May 17 00:11:20 client-5 kernel: Console: colour VGA+ 80x25 May 17 00:11:20 client-5 kernel: Dentry cache hash table entries: 2097152 (order: 12, 16777216 bytes) May 17 00:11:20 client-5 kernel: Inode-cache hash table entries: 1048576 (order: 11, 8388608 bytes) May 17 00:11:20 client-5 kernel: Checking aperture... May 17 00:11:20 client-5 kernel: PCI-DMA: Using software bounce buffering for IO (SWIOTLB) May 17 00:11:20 client-5 kernel: Placing software IO TLB between 0x28e4000 - 0x68e4000 May 17 00:11:20 client-5 kernel: Memory: 12287648k/13631488k available (2574k kernel code, 285968k reserved, 1304k data, 212k init) May 17 00:11:20 client-5 kernel: Calibrating delay loop (skipped), value calculated using timer frequency.. 4533.60 BogoMIPS (lpj=2266804) May 17 00:11:20 client-5 kernel: Security Framework v1.0.0 initialized May 17 00:11:20 client-5 kernel: SELinux: Initializing. May 17 00:11:20 client-5 kernel: selinux_register_security: Registering secondary module capability May 17 00:11:20 client-5 kernel: Capability LSM initialized as secondary May 17 00:11:20 client-5 kernel: Mount-cache hash table entries: 256 May 17 00:11:20 client-5 kernel: CPU: L1 I cache: 32K, L1 D cache: 32K May 17 00:11:20 client-5 kernel: CPU: L2 cache: 256K May 17 00:11:20 client-5 kernel: CPU: L3 cache: 4096K May 17 00:11:20 client-5 kernel: using mwait in idle threads. May 17 00:11:20 client-5 kernel: CPU: Physical Processor ID: 0 May 17 00:11:20 client-5 kernel: CPU: Processor Core ID: 0 May 17 00:11:20 client-5 kernel: CPU0: Thermal monitoring enabled (TM1) May 17 00:11:20 client-5 kernel: SMP alternatives: switching to UP code May 17 00:11:20 client-5 kernel: ACPI: Core revision 20060707 May 17 00:11:20 client-5 kernel: Using local APIC timer interrupts. May 17 00:11:20 client-5 kernel: Detected 8.333 MHz APIC timer. May 17 00:11:20 client-5 kernel: SMP alternatives: switching to SMP code May 17 00:11:20 client-5 kernel: Booting processor 1/4 APIC 0x2 May 17 00:11:20 client-5 kernel: Initializing CPU#1 May 17 00:11:20 client-5 kernel: Calibrating delay using timer specific routine.. 4533.40 BogoMIPS (lpj=2266704) May 17 00:11:20 client-5 kernel: CPU: L1 I cache: 32K, L1 D cache: 32K May 17 00:11:20 client-5 kernel: CPU: L2 cache: 256K May 17 00:11:20 client-5 kernel: CPU: L3 cache: 4096K May 17 00:11:20 client-5 kernel: CPU: Physical Processor ID: 0 May 17 00:11:20 client-5 kernel: CPU: Processor Core ID: 1 May 17 00:11:20 client-5 kernel: CPU1: Thermal monitoring enabled (TM1) May 17 00:11:20 client-5 kernel: Intel(R) Xeon(R) CPU E5507 @ 2.27GHz stepping 05 May 17 00:11:20 client-5 kernel: SMP alternatives: switching to SMP code May 17 00:11:20 client-5 kernel: Booting processor 2/4 APIC 0x4 May 17 00:11:20 client-5 kernel: Initializing CPU#2 May 17 00:11:20 client-5 kernel: Calibrating delay using timer specific routine.. 4533.66 BogoMIPS (lpj=2266834) May 17 00:11:20 client-5 kernel: CPU: L1 I cache: 32K, L1 D cache: 32K May 17 00:11:20 client-5 kernel: CPU: L2 cache: 256K May 17 00:11:20 client-5 kernel: CPU: L3 cache: 4096K May 17 00:11:20 client-5 kernel: CPU: Physical Processor ID: 0 May 17 00:11:20 client-5 kernel: CPU: Processor Core ID: 2 May 17 00:11:20 client-5 kernel: CPU2: Thermal monitoring enabled (TM1) May 17 00:11:20 client-5 kernel: Intel(R) Xeon(R) CPU E5507 @ 2.27GHz stepping 05 May 17 00:11:20 client-5 kernel: SMP alternatives: switching to SMP code May 17 00:11:20 client-5 kernel: Booting processor 3/4 APIC 0x6 May 17 00:11:20 client-5 kernel: Initializing CPU#3 May 17 00:11:20 client-5 kernel: Calibrating delay using timer specific routine.. 4533.40 BogoMIPS (lpj=2266702) May 17 00:11:20 client-5 kernel: CPU: L1 I cache: 32K, L1 D cache: 32K May 17 00:11:20 client-5 kernel: CPU: L2 cache: 256K May 17 00:11:20 client-5 kernel: CPU: L3 cache: 4096K May 17 00:11:20 client-5 kernel: CPU: Physical Processor ID: 0 May 17 00:11:20 client-5 kernel: CPU: Processor Core ID: 3 May 17 00:11:20 client-5 kernel: CPU3: Thermal monitoring enabled (TM1) May 17 00:11:20 client-5 kernel: Intel(R) Xeon(R) CPU E5507 @ 2.27GHz stepping 05 May 17 00:11:20 client-5 kernel: Brought up 4 CPUs May 17 00:11:20 client-5 kernel: testing NMI watchdog ... OK. May 17 00:11:20 client-5 kernel: time.c: Using 14.318180 MHz WALL HPET GTOD HPET/TSC timer. May 17 00:11:20 client-5 kernel: time.c: Detected 2266.804 MHz processor. May 17 00:11:20 client-5 kernel: migration_cost=20 May 17 00:11:20 client-5 kernel: checking if image is initramfs... it is May 17 00:11:20 client-5 kernel: Freeing initrd memory: 2611k freed May 17 00:11:20 client-5 kernel: NET: Registered protocol family 16 May 17 00:11:20 client-5 kernel: ACPI: bus type pci registered May 17 00:11:20 client-5 kernel: PCI: Using MMCONFIG at e0000000 May 17 00:11:20 client-5 kernel: ACPI: Interpreter enabled May 17 00:11:20 client-5 kernel: ACPI: Using IOAPIC for interrupt routing May 17 00:11:20 client-5 kernel: ACPI: No dock devices found. May 17 00:11:20 client-5 kernel: ACPI: PCI Root Bridge [PCI0] (0000:00) May 17 00:11:20 client-5 kernel: PCI: Transparent bridge - 0000:00:1e.0 May 17 00:11:20 client-5 kernel: ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 7 10 11 12 14 *15) May 17 00:11:20 client-5 kernel: ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 *5 6 7 10 11 12 14 15) May 17 00:11:20 client-5 kernel: ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 5 6 7 *10 11 12 14 15) May 17 00:11:20 client-5 kernel: ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 6 7 10 *11 12 14 15) May 17 00:11:20 client-5 kernel: ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 5 6 7 10 11 12 14 15) *0, disabled. May 17 00:11:20 client-5 kernel: ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 5 6 7 10 11 12 *14 15) May 17 00:11:20 client-5 kernel: ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 5 6 7 10 11 12 14 15) *0, disabled. May 17 00:11:20 client-5 kernel: ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 5 6 *7 10 11 12 14 15) May 17 00:11:20 client-5 kernel: Linux Plug and Play Support v0.97 (c) Adam Belay May 17 00:11:20 client-5 kernel: pnp: PnP ACPI init May 17 00:11:20 client-5 kernel: pnp: PnP ACPI: found 15 devices May 17 00:11:20 client-5 kernel: usbcore: registered new driver usbfs May 17 00:11:20 client-5 kernel: usbcore: registered new driver hub May 17 00:11:20 client-5 kernel: PCI: Using ACPI for IRQ routing May 17 00:11:20 client-5 kernel: PCI: If a device doesn't work, try "pci=routeirq". If it helps, post a report May 17 00:11:20 client-5 kernel: NetLabel: Initializing May 17 00:11:20 client-5 kernel: NetLabel: domain hash size = 128 May 17 00:11:20 client-5 kernel: NetLabel: protocols = UNLABELED CIPSOv4 May 17 00:11:20 client-5 kernel: NetLabel: unlabeled traffic allowed by default May 17 00:11:20 client-5 kernel: hpet0: at MMIO 0xfed00000 (virtual 0xffffffffff5fe000), IRQs 2, 8, 0, 0 May 17 00:11:20 client-5 kernel: hpet0: 4 64-bit timers, 14318180 Hz May 17 00:11:20 client-5 kernel: DMAR:Host address width 40 May 17 00:11:20 client-5 kernel: DMAR:DRHD base: 0x000000fbffe000 flags: 0x1 May 17 00:11:20 client-5 kernel: IOMMU fbffe000: ver 1:0 cap c90780106f0462 ecap f020f6 May 17 00:11:20 client-5 kernel: DMAR:RMRR base: 0x000000000e6000 end: 0x000000000e9fff May 17 00:11:20 client-5 kernel: DMAR:RMRR base: 0x000000bf7ec000 end: 0x000000bf7fffff May 17 00:11:20 client-5 kernel: DMAR:ATSR flags: 0x0 May 17 00:11:20 client-5 kernel: PCI-GART: No AMD northbridge found. May 17 00:11:20 client-5 kernel: pnp: 00:08: ioport range 0x164e-0x164f has been reserved May 17 00:11:20 client-5 kernel: pnp: 00:08: ioport range 0xa00-0xa0f has been reserved May 17 00:11:20 client-5 kernel: pnp: 00:0c: ioport range 0xca2-0xca3 has been reserved May 17 00:11:20 client-5 kernel: PCI: Bridge: 0000:00:01.0 May 17 00:11:20 client-5 kernel: IO window: e000-efff May 17 00:11:20 client-5 kernel: MEM window: fbc00000-fbcfffff May 17 00:11:20 client-5 kernel: PREFETCH window: disabled. May 17 00:11:20 client-5 kernel: PCI: Bridge: 0000:00:03.0 May 17 00:11:20 client-5 kernel: IO window: disabled. May 17 00:11:20 client-5 kernel: MEM window: fbd00000-fbdfffff May 17 00:11:20 client-5 kernel: PREFETCH window 0x00000000f8800000-0x00000000f8ffffff May 17 00:11:20 client-5 kernel: PCI: Bridge: 0000:00:05.0 May 17 00:11:20 client-5 kernel: IO window: disabled. May 17 00:11:20 client-5 kernel: MEM window: disabled. May 17 00:11:20 client-5 kernel: PREFETCH window: disabled. May 17 00:11:20 client-5 kernel: PCI: Bridge: 0000:00:07.0 May 17 00:11:20 client-5 kernel: IO window: disabled. May 17 00:11:20 client-5 kernel: MEM window: disabled. May 17 00:11:20 client-5 kernel: PREFETCH window: disabled. May 17 00:11:20 client-5 kernel: PCI: Bridge: 0000:00:1e.0 May 17 00:11:20 client-5 kernel: IO window: disabled. May 17 00:11:20 client-5 kernel: MEM window: faf00000-fb7fffff May 17 00:11:20 client-5 kernel: PREFETCH window 0x00000000f9000000-0x00000000f9ffffff May 17 00:11:20 client-5 rpc.statd[3141]: Version 1.0.9 Starting May 17 00:11:20 client-5 kernel: NET: Registered protocol family 2 May 17 00:11:20 client-5 kernel: IP route cache hash table entries: 524288 (order: 10, 4194304 bytes) May 17 00:11:20 client-5 kernel: TCP established hash table entries: 262144 (order: 10, 4194304 bytes) May 17 00:11:20 client-5 kernel: TCP bind hash table entries: 65536 (order: 8, 1048576 bytes) May 17 00:11:20 client-5 kernel: TCP: Hash tables configured (established 262144 bind 65536) May 17 00:11:20 client-5 kernel: TCP reno registered May 17 00:11:20 client-5 kernel: audit: initializing netlink socket (disabled) May 17 00:11:20 client-5 kernel: type=2000 audit(1305591030.605:1): initialized May 17 00:11:20 client-5 kernel: Total HugeTLB memory allocated, 0 May 17 00:11:20 client-5 kernel: VFS: Disk quotas dquot_6.5.1 May 17 00:11:20 client-5 kernel: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) May 17 00:11:20 client-5 kernel: Initializing Cryptographic API May 17 00:11:20 client-5 kernel: alg: No test for crc32c (crc32c-generic) May 17 00:11:20 client-5 rpc.statd[3141]: statd running as root. chown /var/lib/nfs/statd/sm to choose different user May 17 00:11:20 client-5 kernel: ksign: Installing public key data May 17 00:11:20 client-5 kernel: Loading keyring May 17 00:11:20 client-5 kernel: - Added public key 565662CFF69DD0BE May 17 00:11:20 client-5 kernel: - User ID: CentOS (Kernel Module GPG key) May 17 00:11:20 client-5 kernel: io scheduler noop registered May 17 00:11:20 client-5 kernel: io scheduler anticipatory registered May 17 00:11:20 client-5 kernel: io scheduler deadline registered May 17 00:11:20 client-5 kernel: io scheduler cfq registered (default) May 17 00:11:20 client-5 kernel: pci_hotplug: PCI Hot Plug PCI Core version: 0.5 May 17 00:11:20 client-5 kernel: ACPI (exconfig-0456): Dynamic SSDT Load - OemId [DpgPmm] OemTableId [ P001Ist] [20060707] May 17 00:11:20 client-5 kernel: ACPI (exconfig-0456): Dynamic SSDT Load - OemId [ PmRef] OemTableId [ P001Cst] [20060707] May 17 00:11:20 client-5 kernel: ACPI: CPU0 (power states: C1[C1] C2[C2] C3[C3]) May 17 00:11:20 client-5 kernel: ACPI: CPU1 (power states: C1[C1] C2[C2] C3[C3]) May 17 00:11:20 client-5 kernel: ACPI: CPU2 (power states: C1[C1] C2[C2] C3[C3]) May 17 00:11:20 client-5 kernel: ACPI: CPU3 (power states: C1[C1] C2[C2] C3[C3]) May 17 00:11:20 client-5 kernel: Real Time Clock Driver v1.12ac May 17 00:11:20 client-5 kernel: Non-volatile memory driver v1.2 May 17 00:11:20 client-5 kernel: Linux agpgart interface v0.101 (c) Dave Jones May 17 00:11:20 client-5 kernel: Serial: 8250/16550 driver $Revision: 1.90 $ 4 ports, IRQ sharing enabled May 17 00:11:20 client-5 kernel: serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A May 17 00:11:20 client-5 kernel: serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A May 17 00:11:20 client-5 kernel: 00:06: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A May 17 00:11:20 client-5 kernel: 00:07: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A May 17 00:11:20 client-5 kernel: brd: module loaded May 17 00:11:20 client-5 kernel: Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2 May 17 00:11:20 client-5 kernel: ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx May 17 00:11:20 client-5 kernel: ide-floppy driver 0.99.newide May 17 00:11:20 client-5 kernel: usbcore: registered new driver hiddev May 17 00:11:20 client-5 kernel: usbcore: registered new driver usbhid May 17 00:11:20 client-5 kernel: drivers/usb/input/hid-core.c: v2.6:USB HID core driver May 17 00:11:20 client-5 kernel: PNP: No PS/2 controller found. Probing ports directly. May 17 00:11:20 client-5 kernel: serio: i8042 KBD port at 0x60,0x64 irq 1 May 17 00:11:20 client-5 kernel: serio: i8042 AUX port at 0x60,0x64 irq 12 May 17 00:11:20 client-5 kernel: mice: PS/2 mouse device common for all mice May 17 00:11:20 client-5 kernel: md: md driver 0.90.3 MAX_MD_DEVS=256, MD_SB_DISKS=27 May 17 00:11:20 client-5 kernel: md: bitmap version 4.39 May 17 00:11:20 client-5 kernel: TCP bic registered May 17 00:11:20 client-5 kernel: Initializing IPsec netlink socket May 17 00:11:20 client-5 kernel: NET: Registered protocol family 1 May 17 00:11:20 client-5 kernel: NET: Registered protocol family 17 May 17 00:11:20 client-5 kernel: ACPI: (supports S0 S1 S4 S5) May 17 00:11:20 client-5 kernel: Initalizing network drop monitor service May 17 00:11:20 client-5 kernel: Freeing unused kernel memory: 212k freed May 17 00:11:20 client-5 kernel: Write protecting the kernel read-only data: 504k May 17 00:11:20 client-5 kernel: GSI 16 sharing vector 0xD1 and IRQ 16 May 17 00:11:20 client-5 kernel: ACPI: PCI Interrupt 0000:00:1a.7[C] -> GSI 18 (level, low) -> IRQ 209 May 17 00:11:20 client-5 kernel: ehci_hcd 0000:00:1a.7: EHCI Host Controller May 17 00:11:20 client-5 kernel: ehci_hcd 0000:00:1a.7: new USB bus registered, assigned bus number 1 May 17 00:11:20 client-5 kernel: ehci_hcd 0000:00:1a.7: debug port 1 May 17 00:11:20 client-5 kernel: ehci_hcd 0000:00:1a.7: irq 209, io mem 0xfbed6000 May 17 00:11:20 client-5 kernel: ehci_hcd 0000:00:1a.7: USB 2.0 started, EHCI 1.00, driver 10 Dec 2004 May 17 00:11:20 client-5 kernel: usb usb1: configuration #1 chosen from 1 choice May 17 00:11:20 client-5 kernel: hub 1-0:1.0: USB hub found May 17 00:11:20 client-5 kernel: hub 1-0:1.0: 6 ports detected May 17 00:11:20 client-5 kernel: GSI 17 sharing vector 0xD9 and IRQ 17 May 17 00:11:20 client-5 kernel: ACPI: PCI Interrupt 0000:00:1d.7[A] -> GSI 23 (level, low) -> IRQ 217 May 17 00:11:20 client-5 kernel: ehci_hcd 0000:00:1d.7: EHCI Host Controller May 17 00:11:20 client-5 kernel: ehci_hcd 0000:00:1d.7: new USB bus registered, assigned bus number 2 May 17 00:11:20 client-5 kernel: ehci_hcd 0000:00:1d.7: debug port 1 May 17 00:11:20 client-5 kernel: ehci_hcd 0000:00:1d.7: irq 217, io mem 0xfbed4000 May 17 00:11:20 client-5 kernel: ehci_hcd 0000:00:1d.7: USB 2.0 started, EHCI 1.00, driver 10 Dec 2004 May 17 00:11:20 client-5 kernel: usb usb2: configuration #1 chosen from 1 choice May 17 00:11:20 client-5 kernel: hub 2-0:1.0: USB hub found May 17 00:11:20 client-5 kernel: hub 2-0:1.0: 6 ports detected May 17 00:11:20 client-5 kernel: USB Universal Host Controller Interface driver v3.0 May 17 00:11:20 client-5 kernel: GSI 18 sharing vector 0xE1 and IRQ 18 May 17 00:11:20 client-5 kernel: ACPI: PCI Interrupt 0000:00:1a.0[A] -> GSI 16 (level, low) -> IRQ 225 May 17 00:11:20 client-5 kernel: uhci_hcd 0000:00:1a.0: UHCI Host Controller May 17 00:11:20 client-5 kernel: uhci_hcd 0000:00:1a.0: new USB bus registered, assigned bus number 3 May 17 00:11:20 client-5 kernel: uhci_hcd 0000:00:1a.0: irq 225, io base 0x0000c880 May 17 00:11:20 client-5 kernel: usb usb3: configuration #1 chosen from 1 choice May 17 00:11:20 client-5 kernel: hub 3-0:1.0: USB hub found May 17 00:11:20 client-5 kernel: hub 3-0:1.0: 2 ports detected May 17 00:11:20 client-5 kernel: GSI 19 sharing vector 0xE9 and IRQ 19 May 17 00:11:20 client-5 kernel: ACPI: PCI Interrupt 0000:00:1a.1[B] -> GSI 21 (level, low) -> IRQ 233 May 17 00:11:20 client-5 kernel: uhci_hcd 0000:00:1a.1: UHCI Host Controller May 17 00:11:20 client-5 kernel: uhci_hcd 0000:00:1a.1: new USB bus registered, assigned bus number 4 May 17 00:11:20 client-5 kernel: uhci_hcd 0000:00:1a.1: irq 233, io base 0x0000cc00 May 17 00:11:20 client-5 kernel: usb usb4: configuration #1 chosen from 1 choice May 17 00:11:20 client-5 kernel: hub 4-0:1.0: USB hub found May 17 00:11:20 client-5 kernel: hub 4-0:1.0: 2 ports detected May 17 00:11:20 client-5 kernel: GSI 20 sharing vector 0x32 and IRQ 20 May 17 00:11:20 client-5 kernel: ACPI: PCI Interrupt 0000:00:1a.2[D] -> GSI 19 (level, low) -> IRQ 50 May 17 00:11:20 client-5 kernel: uhci_hcd 0000:00:1a.2: UHCI Host Controller May 17 00:11:20 client-5 kernel: uhci_hcd 0000:00:1a.2: new USB bus registered, assigned bus number 5 May 17 00:11:20 client-5 kernel: uhci_hcd 0000:00:1a.2: irq 50, io base 0x0000d000 May 17 00:11:20 client-5 kernel: usb usb5: configuration #1 chosen from 1 choice May 17 00:11:20 client-5 kernel: hub 5-0:1.0: USB hub found May 17 00:11:20 client-5 kernel: hub 5-0:1.0: 2 ports detected May 17 00:11:20 client-5 kernel: ACPI: PCI Interrupt 0000:00:1d.0[A] -> GSI 23 (level, low) -> IRQ 217 May 17 00:11:20 client-5 kernel: uhci_hcd 0000:00:1d.0: UHCI Host Controller May 17 00:11:20 client-5 kernel: uhci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 6 May 17 00:11:20 client-5 kernel: uhci_hcd 0000:00:1d.0: irq 217, io base 0x0000c400 May 17 00:11:20 client-5 kernel: usb usb6: configuration #1 chosen from 1 choice May 17 00:11:20 client-5 kernel: hub 6-0:1.0: USB hub found May 17 00:11:20 client-5 kernel: hub 6-0:1.0: 2 ports detected May 17 00:11:20 client-5 kernel: usb 4-1: new full speed USB device using uhci_hcd and address 2 May 17 00:11:20 client-5 kernel: ACPI: PCI Interrupt 0000:00:1d.1[B] -> GSI 19 (level, low) -> IRQ 50 May 17 00:11:20 client-5 kernel: uhci_hcd 0000:00:1d.1: UHCI Host Controller May 17 00:11:20 client-5 kernel: uhci_hcd 0000:00:1d.1: new USB bus registered, assigned bus number 7 May 17 00:11:20 client-5 kernel: uhci_hcd 0000:00:1d.1: irq 50, io base 0x0000c480 May 17 00:11:20 client-5 kernel: usb usb7: configuration #1 chosen from 1 choice May 17 00:11:20 client-5 kernel: hub 7-0:1.0: USB hub found May 17 00:11:20 client-5 kernel: hub 7-0:1.0: 2 ports detected May 17 00:11:20 client-5 kernel: usb 4-1: configuration #1 chosen from 1 choice May 17 00:11:20 client-5 kernel: ACPI: PCI Interrupt 0000:00:1d.2[C] -> GSI 18 (level, low) -> IRQ 209 May 17 00:11:20 client-5 kernel: uhci_hcd 0000:00:1d.2: UHCI Host Controller May 17 00:11:20 client-5 kernel: uhci_hcd 0000:00:1d.2: new USB bus registered, assigned bus number 8 May 17 00:11:20 client-5 kernel: input: American Megatrends Inc. Virtual Keyboard and Mouse as /class/input/input0 May 17 00:11:20 client-5 kernel: input: USB HID v1.10 Keyboard [American Megatrends Inc. Virtual Keyboard and Mouse] on usb-0000:00:1a.1-1 May 17 00:11:20 client-5 kernel: input: American Megatrends Inc. Virtual Keyboard and Mouse as /class/input/input1 May 17 00:11:20 client-5 kernel: input: USB HID v1.10 Mouse [American Megatrends Inc. Virtual Keyboard and Mouse] on usb-0000:00:1a.1-1 May 17 00:11:20 client-5 kernel: uhci_hcd 0000:00:1d.2: irq 209, io base 0x0000c800 May 17 00:11:20 client-5 kernel: usb usb8: configuration #1 chosen from 1 choice May 17 00:11:20 client-5 kernel: hub 8-0:1.0: USB hub found May 17 00:11:20 client-5 kernel: hub 8-0:1.0: 2 ports detected May 17 00:11:20 client-5 kernel: SCSI subsystem initialized May 17 00:11:20 client-5 kernel: ACPI: PCI Interrupt 0000:00:1f.2[B] -> GSI 19 (level, low) -> IRQ 50 May 17 00:11:20 client-5 kernel: ahci 0000:00:1f.2: AHCI 0001.0200 32 slots 6 ports 3 Gbps 0x3f impl SATA mode May 17 00:11:20 client-5 kernel: ahci 0000:00:1f.2: flags: 64bit ncq sntf stag pm led clo pio slum part ems May 17 00:11:20 client-5 kernel: scsi0 : ahci May 17 00:11:20 client-5 kernel: scsi1 : ahci May 17 00:11:20 client-5 kernel: scsi2 : ahci May 17 00:11:20 client-5 kernel: scsi3 : ahci May 17 00:11:20 client-5 kernel: scsi4 : ahci May 17 00:11:20 client-5 kernel: scsi5 : ahci May 17 00:11:20 client-5 kernel: ata1: SATA max UDMA/133 abar m2048@0xfbefa000 port 0xfbefa100 irq 58 May 17 00:11:20 client-5 kernel: ata2: SATA max UDMA/133 abar m2048@0xfbefa000 port 0xfbefa180 irq 58 May 17 00:11:20 client-5 kernel: ata3: SATA max UDMA/133 abar m2048@0xfbefa000 port 0xfbefa200 irq 58 May 17 00:11:20 client-5 kernel: ata4: SATA max UDMA/133 abar m2048@0xfbefa000 port 0xfbefa280 irq 58 May 17 00:11:20 client-5 kernel: ata5: SATA max UDMA/133 abar m2048@0xfbefa000 port 0xfbefa300 irq 58 May 17 00:11:20 client-5 kernel: ata6: SATA max UDMA/133 abar m2048@0xfbefa000 port 0xfbefa380 irq 58 May 17 00:11:20 client-5 kernel: ata1: SATA link up 3.0 Gbps (SStatus 123 SControl 300) May 17 00:11:20 client-5 kernel: ata1.00: ATA-8: WDC WD2502ABYS-02B7A0, 02.03B03, max UDMA/133 May 17 00:11:20 client-5 kernel: ata1.00: 490350672 sectors, multi 0: LBA48 NCQ (depth 31/32) May 17 00:11:20 client-5 kernel: ata1.00: configured for UDMA/133 May 17 00:11:20 client-5 kernel: ata2: SATA link down (SStatus 0 SControl 300) May 17 00:11:20 client-5 kernel: ata3: SATA link down (SStatus 0 SControl 300) May 17 00:11:20 client-5 kernel: ata4: SATA link down (SStatus 0 SControl 300) May 17 00:11:20 client-5 kernel: ata5: SATA link down (SStatus 0 SControl 300) May 17 00:11:20 client-5 kernel: ata6: SATA link down (SStatus 0 SControl 300) May 17 00:11:20 client-5 kernel: Vendor: ATA Model: WDC WD2502ABYS-0 Rev: 02.0 May 17 00:11:20 client-5 kernel: Type: Direct-Access ANSI SCSI revision: 05 May 17 00:11:20 client-5 kernel: SCSI device sda: 490350672 512-byte hdwr sectors (251060 MB) May 17 00:11:21 client-5 kernel: sda: Write Protect is off May 17 00:11:21 client-5 kernel: SCSI device sda: drive cache: write back May 17 00:11:21 client-5 kernel: SCSI device sda: 490350672 512-byte hdwr sectors (251060 MB) May 17 00:11:21 client-5 kernel: sda: Write Protect is off May 17 00:11:21 client-5 kernel: SCSI device sda: drive cache: write back May 17 00:11:21 client-5 kernel: sda: sda1 sda2 May 17 00:11:21 client-5 kernel: sd 0:0:0:0: Attached scsi disk sda May 17 00:11:21 client-5 kernel: device-mapper: uevent: version 1.0.3 May 17 00:11:21 client-5 kernel: device-mapper: ioctl: 4.11.5-ioctl (2007-12-12) initialised: dm-devel@redhat.com May 17 00:11:21 client-5 kernel: device-mapper: dm-raid45: initialized v0.2594l May 17 00:11:21 client-5 kernel: EXT3-fs: INFO: recovery required on readonly filesystem. May 17 00:11:21 client-5 kernel: EXT3-fs: write access will be enabled during recovery. May 17 00:11:21 client-5 kernel: kjournald starting. Commit interval 5 seconds May 17 00:11:21 client-5 kernel: EXT3-fs: sda1: orphan cleanup on readonly fs May 17 00:11:21 client-5 kernel: EXT3-fs: sda1: 1 orphan inode deleted May 17 00:11:21 client-5 kernel: EXT3-fs: recovery complete. May 17 00:11:21 client-5 kernel: EXT3-fs: mounted filesystem with ordered data mode. May 17 00:11:21 client-5 kernel: SELinux: Disabled at runtime. May 17 00:11:21 client-5 kernel: type=1404 audit(1305591057.557:2): selinux=0 auid=4294967295 ses=4294967295 May 17 00:11:21 client-5 kernel: dca service started, version 1.4 May 17 00:11:21 client-5 kernel: 802.1Q VLAN Support v1.8 Ben Greear May 17 00:11:21 client-5 kernel: All bugs added by David S. Miller May 17 00:11:21 client-5 kernel: input: PC Speaker as /class/input/input2 May 17 00:11:21 client-5 kernel: Intel(R) Gigabit Ethernet Network Driver - version 2.1.0-k2 May 17 00:11:21 client-5 kernel: Copyright (c) 2007-2009 Intel Corporation. May 17 00:11:21 client-5 kernel: GSI 21 sharing vector 0x42 and IRQ 21 May 17 00:11:21 client-5 kernel: ACPI: PCI Interrupt 0000:01:00.0[A] -> GSI 28 (level, low) -> IRQ 66 May 17 00:11:21 client-5 kernel: sd 0:0:0:0: Attached scsi generic sg0 type 0 May 17 00:11:21 client-5 kernel: igb 0000:01:00.0: Intel(R) Gigabit Ethernet Network Connection May 17 00:11:21 client-5 kernel: igb 0000:01:00.0: eth0: (PCIe:2.5Gb/s:Width x4) 00:30:48:f7:72:24 May 17 00:11:21 client-5 kernel: igb 0000:01:00.0: eth0: PBA No: ffffff-0ff May 17 00:11:21 client-5 kernel: igb 0000:01:00.0: Using MSI-X interrupts. 4 rx queue(s), 1 tx queue(s) May 17 00:11:21 client-5 kernel: GSI 22 sharing vector 0x7A and IRQ 22 May 17 00:11:21 client-5 kernel: ACPI: PCI Interrupt 0000:01:00.1[B] -> GSI 40 (level, low) -> IRQ 122 May 17 00:11:21 client-5 kernel: igb 0000:01:00.1: Intel(R) Gigabit Ethernet Network Connection May 17 00:11:21 client-5 kernel: igb 0000:01:00.1: eth1: (PCIe:2.5Gb/s:Width x4) 00:30:48:f7:72:25 May 17 00:11:21 client-5 kernel: igb 0000:01:00.1: eth1: PBA No: ffffff-0ff May 17 00:11:21 client-5 kernel: igb 0000:01:00.1: Using MSI-X interrupts. 4 rx queue(s), 1 tx queue(s) May 17 00:11:21 client-5 kernel: shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 May 17 00:11:21 client-5 kernel: ACPI: PCI Interrupt 0000:00:1f.3[C] -> GSI 18 (level, low) -> IRQ 209 May 17 00:11:21 client-5 kernel: mlx4_core: Mellanox ConnectX core driver v1.0-ofed1.5.3 (January 19, 2011) May 17 00:11:21 client-5 kernel: mlx4_core: Initializing 0000:02:00.0 May 17 00:11:21 client-5 kernel: GSI 23 sharing vector 0xB2 and IRQ 23 May 17 00:11:21 client-5 kernel: ACPI: PCI Interrupt 0000:02:00.0[A] -> GSI 24 (level, low) -> IRQ 178 May 17 00:11:21 client-5 kernel: mlx4_ib: Mellanox ConnectX InfiniBand driver v1.0-ofed1.5.3 (January 19, 2011) May 17 00:11:21 client-5 kernel: floppy0: no floppy controllers found May 17 00:11:21 client-5 kernel: lp: driver loaded but no devices found May 17 00:11:21 client-5 kernel: ACPI: Power Button (FF) [PWRF] May 17 00:11:21 client-5 kernel: ACPI: Power Button (CM) [PWRB] May 17 00:11:21 client-5 kernel: ACPI: Mapper loaded May 17 00:11:21 client-5 kernel: dell-wmi: No known WMI GUID found May 17 00:11:21 client-5 kernel: md: Autodetecting RAID arrays. May 17 00:11:21 client-5 kernel: md: autorun ... May 17 00:11:21 client-5 kernel: md: ... autorun DONE. May 17 00:11:21 client-5 kernel: device-mapper: multipath: version 1.0.5 loaded May 17 00:11:21 client-5 kernel: loop: loaded (max 8 devices) May 17 00:11:21 client-5 kernel: EXT3 FS on sda1, internal journal May 17 00:11:21 client-5 kernel: Adding 14346036k swap on /dev/sda2. Priority:-1 extents:1 across:14346036k May 17 00:11:21 client-5 kernel: IA-32 Microcode Update Driver: v1.14a May 17 00:11:21 client-5 kernel: mlx4_en: Mellanox ConnectX HCA Ethernet driver v1.5.3 (Jan 2011) May 17 00:11:21 client-5 kernel: mlx4_en 0000:02:00.0: UDP RSS is not supported on this device. May 17 00:11:21 client-5 kernel: NET: Registered protocol family 10 May 17 00:11:21 client-5 kernel: lo: Disabled Privacy Extensions May 17 00:11:21 client-5 kernel: IPv6 over IPv4 tunneling driver May 17 00:11:21 client-5 kernel: ADDRCONF(NETDEV_UP): ib0: link is not ready May 17 00:11:21 client-5 kernel: ib0: enabling connected mode will cause multicast packet drops May 17 00:11:21 client-5 kernel: ib0: mtu > 2044 will cause multicast packet drops. May 17 00:11:21 client-5 kernel: ib0: mtu > 2044 will cause multicast packet drops. May 17 00:11:21 client-5 kernel: NET: Registered protocol family 27 May 17 00:11:21 client-5 kernel: NET: Registered protocol family 28 May 17 00:11:21 client-5 kernel: QLGC_VNIC: Initializing QLogic Corp. Virtual NIC (VNIC) driver version 1.3.0.0.4 May 17 00:11:21 client-5 kernel: Loading iSCSI transport class v2.0-871. May 17 00:11:21 client-5 kernel: cxgb3i: disagrees about version of symbol cxgb3_register_client May 17 00:11:21 client-5 kernel: cxgb3i: Unknown symbol cxgb3_register_client May 17 00:11:21 client-5 kernel: cxgb3i: disagrees about version of symbol cxgb3_alloc_atid May 17 00:11:21 client-5 kernel: cxgb3i: Unknown symbol cxgb3_alloc_atid May 17 00:11:21 client-5 kernel: cxgb3i: disagrees about version of symbol t3_l2t_get May 17 00:11:21 client-5 kernel: cxgb3i: Unknown symbol t3_l2t_get May 17 00:11:21 client-5 kernel: cxgb3i: disagrees about version of symbol cxgb3_insert_tid May 17 00:11:21 client-5 kernel: cxgb3i: Unknown symbol cxgb3_insert_tid May 17 00:11:21 client-5 kernel: cxgb3i: disagrees about version of symbol t3_l2e_free May 17 00:11:21 client-5 kernel: cxgb3i: Unknown symbol t3_l2e_free May 17 00:11:21 client-5 kernel: cxgb3i: disagrees about version of symbol t3_l2t_send_slow May 17 00:11:21 client-5 kernel: cxgb3i: Unknown symbol t3_l2t_send_slow May 17 00:11:21 client-5 kernel: cxgb3i: disagrees about version of symbol cxgb3_unregister_client May 17 00:11:21 client-5 kernel: cxgb3i: Unknown symbol cxgb3_unregister_client May 17 00:11:21 client-5 kernel: Broadcom NetXtreme II CNIC Driver cnic v2.1.0 (Oct 10, 2009) May 17 00:11:21 client-5 kernel: Broadcom NetXtreme II iSCSI Driver bnx2i v2.1.0 (Dec 06, 2009) May 17 00:11:21 client-5 kernel: iscsi: registered transport (bnx2i) May 17 00:11:21 client-5 kernel: iscsi: registered transport (tcp) May 17 00:11:21 client-5 kernel: ib_iser: disagrees about version of symbol ib_fmr_pool_unmap May 17 00:11:21 client-5 kernel: ib_iser: Unknown symbol ib_fmr_pool_unmap May 17 00:11:21 client-5 kernel: ib_iser: disagrees about version of symbol ib_create_cq May 17 00:11:21 client-5 kernel: ib_iser: Unknown symbol ib_create_cq May 17 00:11:21 client-5 kernel: ib_iser: disagrees about version of symbol rdma_resolve_addr May 17 00:11:21 client-5 kernel: ib_iser: Unknown symbol rdma_resolve_addr May 17 00:11:21 client-5 kernel: ib_iser: disagrees about version of symbol ib_create_fmr_pool May 17 00:11:21 client-5 kernel: ib_iser: Unknown symbol ib_create_fmr_pool May 17 00:11:21 client-5 kernel: ib_iser: disagrees about version of symbol ib_dereg_mr May 17 00:11:21 client-5 kernel: ib_iser: Unknown symbol ib_dereg_mr May 17 00:11:21 client-5 kernel: ib_iser: disagrees about version of symbol rdma_disconnect May 17 00:11:21 client-5 kernel: ib_iser: Unknown symbol rdma_disconnect May 17 00:11:21 client-5 kernel: ib_iser: disagrees about version of symbol rdma_resolve_route May 17 00:11:21 client-5 kernel: ib_iser: Unknown symbol rdma_resolve_route May 17 00:11:21 client-5 kernel: ib_iser: disagrees about version of symbol rdma_create_qp May 17 00:11:21 client-5 kernel: ib_iser: Unknown symbol rdma_create_qp May 17 00:11:21 client-5 kernel: ib_iser: disagrees about version of symbol ib_destroy_cq May 17 00:11:21 client-5 kernel: ib_iser: Unknown symbol ib_destroy_cq May 17 00:11:21 client-5 kernel: ib_iser: disagrees about version of symbol rdma_create_id May 17 00:11:21 client-5 kernel: ib_iser: Unknown symbol rdma_create_id May 17 00:11:21 client-5 kernel: ib_iser: disagrees about version of symbol rdma_destroy_qp May 17 00:11:21 client-5 kernel: ib_iser: Unknown symbol rdma_destroy_qp May 17 00:11:21 client-5 kernel: ib_iser: disagrees about version of symbol ib_get_dma_mr May 17 00:11:21 client-5 kernel: ib_iser: Unknown symbol ib_get_dma_mr May 17 00:11:21 client-5 kernel: ib_iser: disagrees about version of symbol ib_alloc_pd May 17 00:11:21 client-5 kernel: ib_iser: Unknown symbol ib_alloc_pd May 17 00:11:21 client-5 kernel: ib_iser: disagrees about version of symbol rdma_connect May 17 00:11:21 client-5 kernel: ib_iser: Unknown symbol rdma_connect May 17 00:11:21 client-5 kernel: ib_iser: disagrees about version of symbol rdma_destroy_id May 17 00:11:21 client-5 kernel: ib_iser: Unknown symbol rdma_destroy_id May 17 00:11:21 client-5 kernel: ib_iser: disagrees about version of symbol ib_dealloc_pd May 17 00:11:21 client-5 kernel: ib_iser: Unknown symbol ib_dealloc_pd May 17 00:11:21 client-5 kernel: ib_iser: disagrees about version of symbol ib_fmr_pool_map_phys May 17 00:11:21 client-5 kernel: ib_iser: Unknown symbol ib_fmr_pool_map_phys May 17 00:11:21 client-5 kernel: iscsi: registered transport (be2iscsi) May 17 00:11:21 client-5 kernel: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX May 17 00:11:21 client-5 kernel: ADDRCONF(NETDEV_UP): eth1: link is not ready May 17 00:11:21 client-5 hcid[3184]: Bluetooth HCI daemon May 17 00:11:21 client-5 kernel: Bluetooth: Core ver 2.10 May 17 00:11:21 client-5 kernel: NET: Registered protocol family 31 May 17 00:11:21 client-5 kernel: Bluetooth: HCI device and connection manager initialized May 17 00:11:21 client-5 kernel: Bluetooth: HCI socket layer initialized May 17 00:11:21 client-5 hcid[3184]: Register path:/org/bluez fallback:1 May 17 00:11:21 client-5 sdpd[3190]: Bluetooth SDP daemon May 17 00:11:21 client-5 kernel: Bluetooth: L2CAP ver 2.8 May 17 00:11:21 client-5 kernel: Bluetooth: L2CAP socket layer initialized May 17 00:11:21 client-5 kernel: Bluetooth: RFCOMM socket layer initialized May 17 00:11:21 client-5 kernel: Bluetooth: RFCOMM TTY layer initialized May 17 00:11:21 client-5 kernel: Bluetooth: RFCOMM ver 1.8 May 17 00:11:21 client-5 pcscd: pcscdaemon.c:507:main() pcsc-lite 1.4.4 daemon ready. May 17 00:11:21 client-5 pcscd: hotplug_libusb.c:402:HPEstablishUSBNotifications() Driver ifd-egate.bundle does not support IFD_GENERATE_HOTPLUG. Using active polling instead. May 17 00:11:21 client-5 pcscd: hotplug_libusb.c:411:HPEstablishUSBNotifications() Polling forced every 1 second(s) May 17 00:11:22 client-5 kernel: Bluetooth: HIDP (Human Interface Emulation) ver 1.1 May 17 00:11:22 client-5 hidd[3315]: Bluetooth HID daemon May 17 00:11:22 client-5 kernel: ADDRCONF(NETDEV_CHANGE): ib0: link becomes ready May 17 00:11:23 client-5 automount[3344]: lookup_read_master: lookup(nisplus): couldn't locate nis+ table auto.master May 17 00:11:23 client-5 automount[3344]: lookup_read_map: lookup(nisplus): couldn't locate nis+ table auto.direct May 17 00:11:23 client-5 sssd: Starting up May 17 00:11:23 client-5 sssd[be[LAB.WHAMCLOUD.COM]]: Starting up May 17 00:11:23 client-5 sssd[pam]: Starting up May 17 00:11:23 client-5 sssd[nss]: Starting up May 17 00:11:24 client-5 ntpd[3401]: ntpd 4.2.2p1@1.1570-o Sat Dec 19 00:56:13 UTC 2009 (1) May 17 00:11:24 client-5 ntpd[3402]: precision = 1.000 usec May 17 00:11:24 client-5 ntpd[3402]: Listening on interface wildcard, 0.0.0.0#123 Disabled May 17 00:11:24 client-5 ntpd[3402]: Listening on interface wildcard, ::#123 Disabled May 17 00:11:24 client-5 ntpd[3402]: Listening on interface lo, ::1#123 Enabled May 17 00:11:24 client-5 ntpd[3402]: Listening on interface eth0, fe80::230:48ff:fef7:7224#123 Enabled May 17 00:11:24 client-5 ntpd[3402]: bind() fd 20, family 10, port 123, scope 5, addr fe80::230:48ff:fff4:f571, in6_is_addr_multicast=0 flags=1 fails: Cannot assign requested address May 17 00:11:24 client-5 ntpd[3402]: Listening on interface lo, 127.0.0.1#123 Enabled May 17 00:11:24 client-5 ntpd[3402]: Listening on interface eth0, 10.10.4.5#123 Enabled May 17 00:11:24 client-5 ntpd[3402]: Listening on interface eth1, 172.16.8.5#123 Enabled May 17 00:11:24 client-5 ntpd[3402]: Listening on interface ib0, 192.168.4.5#123 Enabled May 17 00:11:24 client-5 ntpd[3402]: kernel time sync status 0040 May 17 00:11:24 client-5 ntpd[3402]: frequency initialized -24.602 PPM from /var/lib/ntp/ntp.drift May 17 00:11:24 client-5 xinetd[3391]: xinetd Version 2.3.14 started with libwrap loadavg labeled-networking options compiled in. May 17 00:11:24 client-5 xinetd[3391]: Started working: 2 available services May 17 00:11:24 client-5 gpm[3437]: *** info [startup.c(95)]: May 17 00:11:24 client-5 gpm[3437]: Started gpm successfully. Entered daemon mode. May 17 00:11:24 client-5 smartd[3501]: smartd version 5.38 [x86_64-redhat-linux-gnu] Copyright (C) 2002-8 Bruce Allen May 17 00:11:24 client-5 smartd[3501]: Home page is http://smartmontools.sourceforge.net/ May 17 00:11:24 client-5 smartd[3501]: Opened configuration file /etc/smartd.conf May 17 00:11:24 client-5 smartd[3501]: Configuration file /etc/smartd.conf was parsed, found DEVICESCAN, scanning devices May 17 00:11:24 client-5 smartd[3501]: Problem creating device name scan list May 17 00:11:24 client-5 smartd[3501]: Device: /dev/sda, opened May 17 00:11:24 client-5 smartd[3501]: Device /dev/sda: using '-d sat' for ATA disk behind SAT layer. May 17 00:11:24 client-5 smartd[3501]: Device: /dev/sda, opened May 17 00:11:24 client-5 smartd[3501]: Device: /dev/sda, not found in smartd database. May 17 00:11:24 client-5 smartd[3501]: Device: /dev/sda, is SMART capable. Adding to "monitor" list. May 17 00:11:24 client-5 smartd[3501]: Monitoring 0 ATA and 1 SCSI devices May 17 00:11:24 client-5 smartd[3509]: smartd has fork()ed into background mode. New PID=3509. May 17 00:13:01 client-5 xinetd[3391]: START: shell pid=3751 from=192.168.4.5 May 17 00:13:02 client-5 rshd[3752]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:; lsof -t /mnt/lustre || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre is still busy, wait one second && sleep 1; umount /mnt/lustre; fi fi");echo XXRETCODE:$?' May 17 00:13:02 client-5 xinetd[3391]: EXIT: shell status=0 pid=3751 duration=1(sec) May 17 00:13:02 client-5 xinetd[3391]: START: shell pid=3795 from=192.168.4.5 May 17 00:13:02 client-5 rshd[3796]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:; lsof -t /mnt/lustre2 || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre2 | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre2 is still busy, wait one second && sleep 1; umount /mnt/lustre2; fi fi");echo XXRETCODE:$?' May 17 00:13:02 client-5 xinetd[3391]: EXIT: shell status=0 pid=3795 duration=0(sec) May 17 00:13:04 client-5 kernel: Lustre: 4015:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 00:13:05 client-5 kernel: Lustre: 4017:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 00:13:05 client-5 kernel: Lustre: OBD class driver, http://www.lustre.org/ May 17 00:13:05 client-5 kernel: Lustre: Lustre Version: 2.0.60 May 17 00:13:05 client-5 kernel: Lustre: Build Version: jenkins-g985ac5b-PRISTINE-2.6.18-194.17.1.el5 May 17 00:13:05 client-5 kernel: Lustre: Lustre LU module (ffffffff888eb1c0). May 17 00:13:05 client-5 kernel: Lustre: Register global MR array, MR size: 0xffffffffffffffff, array size: 1 May 17 00:13:05 client-5 kernel: Lustre: Added LNI 192.168.4.5@o2ib [8/64/0/180] May 17 00:13:05 client-5 kernel: Lustre: Added LNI 10.10.4.5@tcp [8/256/0/180] May 17 00:13:05 client-5 kernel: Lustre: Accept all, port 7988 May 17 00:13:05 client-5 kernel: Lustre: Lustre OSC module (ffffffff88b3c6c0). May 17 00:13:05 client-5 kernel: Lustre: Lustre LOV module (ffffffff88bab540). May 17 00:13:06 client-5 kernel: Lustre: Lustre client module (ffffffff88c643a0). May 17 00:14:04 client-5 xinetd[3391]: START: shell pid=4744 from=192.168.4.5 May 17 00:14:04 client-5 rshd[4745]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); mpts=\$(mount | grep -w -c /mnt/lustre); if [ \$running -ne \$mpts ]; then echo \$(hostname) env are INSANE!; exit 1; fi");echo XXRETCODE:$?' May 17 00:14:04 client-5 xinetd[3391]: EXIT: shell status=0 pid=4744 duration=0(sec) May 17 00:14:04 client-5 xinetd[3391]: START: shell pid=4790 from=192.168.4.5 May 17 00:14:04 client-5 rshd[4792]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); mpts=\$(mount | grep -w -c /mnt/lustre2); if [ \$running -ne \$mpts ]; then echo \$(hostname) env are INSANE!; exit 1; fi");echo XXRETCODE:$?' May 17 00:14:04 client-5 xinetd[3391]: EXIT: shell status=0 pid=4790 duration=0(sec) May 17 00:14:04 client-5 kernel: Lustre: 4834:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 00:14:12 client-5 kernel: Lustre: 5171:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 00:14:12 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 00:14:12 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 00:14:12 client-5 kernel: Lustre: 5171:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff81033c45d000->192.168.4.128@o2ib netid 50000: select flavor null May 17 00:14:12 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 00:14:17 client-5 kernel: Lustre: 5171:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-OST0000-osc-ffff81033c45d000->192.168.4.129@o2ib netid 50000: select flavor null May 17 00:14:17 client-5 kernel: Lustre: Client lustre-client has started May 17 00:14:17 client-5 kernel: Lustre: 5189:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 00:14:17 client-5 kernel: Lustre: 5189:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message May 17 00:14:17 client-5 xinetd[3391]: START: shell pid=5199 from=192.168.4.5 May 17 00:14:17 client-5 rshd[5200]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c " running=\$(mount | grep -c /mnt/lustre' '); rc=0; if [ \$running -eq 0 ] ; then mkdir -p /mnt/lustre; mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre; rc=\$?; fi; exit \$rc");echo XXRETCODE:$?' May 17 00:14:17 client-5 xinetd[3391]: EXIT: shell status=0 pid=5199 duration=0(sec) May 17 00:14:18 client-5 xinetd[3391]: START: shell pid=5244 from=192.168.4.5 May 17 00:14:18 client-5 rshd[5245]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mount | grep -w /mnt/lustre");echo XXRETCODE:$?' May 17 00:14:18 client-5 xinetd[3391]: EXIT: shell status=0 pid=5244 duration=0(sec) May 17 00:14:18 client-5 xinetd[3391]: START: shell pid=5287 from=192.168.4.5 May 17 00:14:18 client-5 rshd[5289]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param debug=\"0x33f0406\"; lctl set_param subsystem_debug=\"0xffb7e3ff\"; lctl set_param debug_mb=32;");echo XXRETCODE:$?' May 17 00:14:18 client-5 xinetd[3391]: EXIT: shell status=0 pid=5287 duration=0(sec) May 17 00:14:19 client-5 xinetd[3391]: START: shell pid=5458 from=192.168.4.5 May 17 00:14:19 client-5 rshd[5459]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark Using TIMEOUT=20");echo XXRETCODE:$?' May 17 00:14:19 client-5 kernel: Lustre: DEBUG MARKER: Using TIMEOUT=20 May 17 00:14:19 client-5 xinetd[3391]: EXIT: shell status=0 pid=5458 duration=0(sec) May 17 00:14:19 client-5 kernel: LustreError: 5512:0:(quota_ctl.c:328:client_quota_ctl()) ptlrpc_queue_wait failed, rc: -114 May 17 00:14:24 client-5 xinetd[3391]: START: shell pid=5590 from=192.168.4.5 May 17 00:14:24 client-5 rshd[5591]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/tests:/usr/lib64/openmpi/1.4-gcc/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/usr/lib64/lustre/utils:/usr/lib64/lustre/utils/gss:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey::/usr/lib64/lustre/tests/mpi NAME=sarah sh rpc.sh check_logdir /tmp/test_logs/2011-05-17/001301 ");echo XXRETCODE:$?' May 17 00:14:24 client-5 xinetd[3391]: EXIT: shell status=0 pid=5590 duration=0(sec) May 17 00:14:24 client-5 xinetd[3391]: START: shell pid=5772 from=192.168.4.5 May 17 00:14:25 client-5 rshd[5775]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/tests:/usr/lib64/openmpi/1.4-gcc/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/usr/lib64/lustre/utils:/usr/lib64/lustre/utils/gss:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey::/usr/lib64/lustre/tests/mpi NAME=sarah sh rpc.sh yml_node ");echo XXRETCODE:$?' May 17 00:14:25 client-5 xinetd[3391]: EXIT: shell status=0 pid=5772 duration=1(sec) May 17 00:14:25 client-5 xinetd[3391]: START: shell pid=6209 from=192.168.4.5 May 17 00:14:25 client-5 rshd[6210]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark -----============= acceptance-small: sanity ============----- Tue May 17 00:14:25 PDT 2011");echo XXRETCODE:$?' May 17 00:14:25 client-5 kernel: Lustre: DEBUG MARKER: -----============= acceptance-small: sanity ============----- Tue May 17 00:14:25 PDT 2011 May 17 00:14:25 client-5 xinetd[3391]: EXIT: shell status=0 pid=6209 duration=0(sec) May 17 00:14:25 client-5 xinetd[3391]: START: shell pid=6303 from=192.168.4.5 May 17 00:14:25 client-5 rshd[6304]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/tests:/usr/lib64/openmpi/1.4-gcc/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/usr/lib64/lustre/utils:/usr/lib64/lustre/utils/gss:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey::/usr/lib64/lustre/tests/mpi:/sbin NAME=sarah sh rpc.sh check_config_client /mnt/lustre ");echo XXRETCODE:$?' May 17 00:14:25 client-5 xinetd[3391]: EXIT: shell status=0 pid=6303 duration=0(sec) May 17 00:14:26 client-5 xinetd[3391]: START: shell pid=6503 from=192.168.4.5 May 17 00:14:26 client-5 rshd[6504]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); mpts=\$(mount | grep -w -c /mnt/lustre); if [ \$running -ne \$mpts ]; then echo \$(hostname) env are INSANE!; exit 1; fi");echo XXRETCODE:$?' May 17 00:14:26 client-5 xinetd[3391]: EXIT: shell status=0 pid=6503 duration=0(sec) May 17 00:14:26 client-5 xinetd[3391]: START: shell pid=6550 from=192.168.4.5 May 17 00:14:26 client-5 rshd[6551]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); mpts=\$(mount | grep -w -c /mnt/lustre2); if [ \$running -ne \$mpts ]; then echo \$(hostname) env are INSANE!; exit 1; fi");echo XXRETCODE:$?' May 17 00:14:26 client-5 xinetd[3391]: EXIT: shell status=0 pid=6550 duration=0(sec) May 17 00:14:28 client-5 xinetd[3391]: START: shell pid=6911 from=192.168.4.5 May 17 00:14:28 client-5 rshd[6913]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark Using TIMEOUT=20");echo XXRETCODE:$?' May 17 00:14:28 client-5 kernel: Lustre: DEBUG MARKER: Using TIMEOUT=20 May 17 00:14:28 client-5 xinetd[3391]: EXIT: shell status=0 pid=6911 duration=0(sec) May 17 00:14:29 client-5 kernel: LustreError: 6966:0:(quota_ctl.c:328:client_quota_ctl()) ptlrpc_queue_wait failed, rc: -114 May 17 00:14:29 client-5 xinetd[3391]: START: shell pid=7023 from=192.168.4.5 May 17 00:14:29 client-5 rshd[7024]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param debug=\"0x33f0406\"; lctl set_param subsystem_debug=\"0xffb7e3ff\"; lctl set_param debug_mb=32; sync");echo XXRETCODE:$?' May 17 00:14:29 client-5 kernel: Lustre: 7058:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 00:14:29 client-5 kernel: Lustre: 7058:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 3 previous similar messages May 17 00:14:29 client-5 xinetd[3391]: EXIT: shell status=0 pid=7023 duration=0(sec) May 17 00:14:29 client-5 xinetd[3391]: START: shell pid=7075 from=192.168.4.5 May 17 00:14:29 client-5 rshd[7076]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "cat /proc/mounts");echo XXRETCODE:$?' May 17 00:14:29 client-5 xinetd[3391]: EXIT: shell status=0 pid=7075 duration=0(sec) May 17 00:14:30 client-5 xinetd[3391]: START: shell pid=7356 from=192.168.4.5 May 17 00:14:30 client-5 rshd[7357]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark excepting tests: 27u 42a 42b 42c 42d 45 51d 65a 65e 68b");echo XXRETCODE:$?' May 17 00:14:30 client-5 kernel: Lustre: DEBUG MARKER: excepting tests: 27u 42a 42b 42c 42d 45 51d 65a 65e 68b May 17 00:14:30 client-5 xinetd[3391]: EXIT: shell status=0 pid=7356 duration=0(sec) May 17 00:14:30 client-5 xinetd[3391]: START: shell pid=7464 from=192.168.4.5 May 17 00:14:30 client-5 rshd[7465]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 0: touch ...\/ \; rm ...\/ =============================================================== 00:14:30 \(1305616470\)");echo XXRETCODE:$?' May 17 00:14:30 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 0: touch .../ ; rm .../ =============================================================== 00:14:30 (1305616470) May 17 00:14:30 client-5 xinetd[3391]: EXIT: shell status=0 pid=7464 duration=0(sec) May 17 00:14:30 client-5 xinetd[3391]: START: shell pid=7558 from=192.168.4.5 May 17 00:14:30 client-5 rshd[7560]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 00:14:30 client-5 xinetd[3391]: EXIT: shell status=0 pid=7558 duration=0(sec) May 17 00:14:31 client-5 xinetd[3391]: START: shell pid=7731 from=192.168.4.5 May 17 00:14:31 client-5 rshd[7732]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 0b: chmod 0755 \/mnt\/lustre ============================================================ 00:14:31 \(1305616471\)");echo XXRETCODE:$?' May 17 00:14:31 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 0b: chmod 0755 /mnt/lustre ============================================================ 00:14:31 (1305616471) May 17 00:14:31 client-5 xinetd[3391]: EXIT: shell status=0 pid=7731 duration=0(sec) May 17 00:14:31 client-5 xinetd[3391]: START: shell pid=7822 from=192.168.4.5 May 17 00:14:31 client-5 rshd[7825]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 00:14:31 client-5 xinetd[3391]: EXIT: shell status=0 pid=7822 duration=0(sec) May 17 00:14:31 client-5 xinetd[3391]: START: shell pid=7996 from=192.168.4.5 May 17 00:14:31 client-5 rshd[7997]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 0c: check import proc ================================================================= 00:14:31 \(1305616471\)");echo XXRETCODE:$?' May 17 00:14:31 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 0c: check import proc ================================================================= 00:14:31 (1305616471) May 17 00:14:31 client-5 xinetd[3391]: EXIT: shell status=0 pid=7996 duration=0(sec) May 17 00:14:31 client-5 xinetd[3391]: START: shell pid=8090 from=192.168.4.5 May 17 00:14:31 client-5 rshd[8092]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 00:14:31 client-5 xinetd[3391]: EXIT: shell status=0 pid=8090 duration=0(sec) May 17 00:14:32 client-5 xinetd[3391]: START: shell pid=8263 from=192.168.4.5 May 17 00:14:32 client-5 rshd[8264]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 1a: mkdir ...\/d1\; mkdir ...\/d1\/d2 ===================================================== 00:14:32 \(1305616472\)");echo XXRETCODE:$?' May 17 00:14:32 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 1a: mkdir .../d1; mkdir .../d1/d2 ===================================================== 00:14:32 (1305616472) May 17 00:14:32 client-5 xinetd[3391]: EXIT: shell status=0 pid=8263 duration=0(sec) May 17 00:14:32 client-5 xinetd[3391]: START: shell pid=8358 from=192.168.4.5 May 17 00:14:32 client-5 rshd[8359]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 00:14:32 client-5 xinetd[3391]: EXIT: shell status=0 pid=8358 duration=0(sec) May 17 00:14:32 client-5 xinetd[3391]: START: shell pid=8530 from=192.168.4.5 May 17 00:14:32 client-5 rshd[8531]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 1b: rmdir ...\/d1\/d2\; rmdir ...\/d1 ===================================================== 00:14:32 \(1305616472\)");echo XXRETCODE:$?' May 17 00:14:32 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 1b: rmdir .../d1/d2; rmdir .../d1 ===================================================== 00:14:32 (1305616472) May 17 00:14:32 client-5 xinetd[3391]: EXIT: shell status=0 pid=8530 duration=0(sec) May 17 00:14:32 client-5 xinetd[3391]: START: shell pid=8625 from=192.168.4.5 May 17 00:14:32 client-5 rshd[8626]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 00:14:32 client-5 xinetd[3391]: EXIT: shell status=0 pid=8625 duration=0(sec) May 17 00:14:33 client-5 xinetd[3391]: START: shell pid=8797 from=192.168.4.5 May 17 00:14:33 client-5 rshd[8798]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 2a: mkdir ...\/d2\; touch ...\/d2\/f ====================================================== 00:14:33 \(1305616473\)");echo XXRETCODE:$?' May 17 00:14:33 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 2a: mkdir .../d2; touch .../d2/f ====================================================== 00:14:33 (1305616473) May 17 00:14:33 client-5 xinetd[3391]: EXIT: shell status=0 pid=8797 duration=0(sec) May 17 00:14:33 client-5 xinetd[3391]: START: shell pid=8891 from=192.168.4.5 May 17 00:14:33 client-5 rshd[8892]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 00:14:33 client-5 xinetd[3391]: EXIT: shell status=0 pid=8891 duration=0(sec) May 17 00:14:33 client-5 xinetd[3391]: START: shell pid=9063 from=192.168.4.5 May 17 00:14:33 client-5 rshd[9064]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 2b: rm -r ...\/d2\; checkstat ...\/d2\/f ================================================== 00:14:33 \(1305616473\)");echo XXRETCODE:$?' May 17 00:14:33 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 2b: rm -r .../d2; checkstat .../d2/f ================================================== 00:14:33 (1305616473) May 17 00:14:33 client-5 xinetd[3391]: EXIT: shell status=0 pid=9063 duration=0(sec) May 17 00:14:33 client-5 xinetd[3391]: START: shell pid=9156 from=192.168.4.5 May 17 00:14:33 client-5 rshd[9159]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 00:14:33 client-5 xinetd[3391]: EXIT: shell status=0 pid=9156 duration=0(sec) May 17 00:14:34 client-5 xinetd[3391]: START: shell pid=9330 from=192.168.4.5 May 17 00:14:34 client-5 rshd[9331]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 3a: mkdir ...\/d3 ====================================================================== 00:14:34 \(1305616474\)");echo XXRETCODE:$?' May 17 00:14:34 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 3a: mkdir .../d3 ====================================================================== 00:14:34 (1305616474) May 17 00:14:34 client-5 xinetd[3391]: EXIT: shell status=0 pid=9330 duration=0(sec) May 17 00:14:34 client-5 xinetd[3391]: START: shell pid=9423 from=192.168.4.5 May 17 00:14:34 client-5 rshd[9424]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 00:14:34 client-5 xinetd[3391]: EXIT: shell status=0 pid=9423 duration=0(sec) May 17 00:14:34 client-5 xinetd[3391]: START: shell pid=9594 from=192.168.4.5 May 17 00:14:34 client-5 rshd[9596]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 3b: touch ...\/d3\/f ==================================================================== 00:14:34 \(1305616474\)");echo XXRETCODE:$?' May 17 00:14:34 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 3b: touch .../d3/f ==================================================================== 00:14:34 (1305616474) May 17 00:14:34 client-5 xinetd[3391]: EXIT: shell status=0 pid=9594 duration=0(sec) May 17 00:14:34 client-5 xinetd[3391]: START: shell pid=9688 from=192.168.4.5 May 17 00:14:34 client-5 rshd[9689]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 00:14:34 client-5 xinetd[3391]: EXIT: shell status=0 pid=9688 duration=0(sec) May 17 00:14:35 client-5 xinetd[3391]: START: shell pid=9859 from=192.168.4.5 May 17 00:14:35 client-5 rshd[9861]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 3c: rm -r ...\/d3 ====================================================================== 00:14:35 \(1305616475\)");echo XXRETCODE:$?' May 17 00:14:35 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 3c: rm -r .../d3 ====================================================================== 00:14:35 (1305616475) May 17 00:14:35 client-5 xinetd[3391]: EXIT: shell status=0 pid=9859 duration=0(sec) May 17 00:14:35 client-5 xinetd[3391]: START: shell pid=9954 from=192.168.4.5 May 17 00:14:35 client-5 rshd[9955]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 00:14:35 client-5 xinetd[3391]: EXIT: shell status=0 pid=9954 duration=0(sec) May 17 00:14:35 client-5 xinetd[3391]: START: shell pid=10124 from=192.168.4.5 May 17 00:14:35 client-5 rshd[10127]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 4a: mkdir ...\/d4 ====================================================================== 00:14:35 \(1305616475\)");echo XXRETCODE:$?' May 17 00:14:35 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 4a: mkdir .../d4 ====================================================================== 00:14:35 (1305616475) May 17 00:14:35 client-5 xinetd[3391]: EXIT: shell status=0 pid=10124 duration=0(sec) May 17 00:14:35 client-5 xinetd[3391]: START: shell pid=10219 from=192.168.4.5 May 17 00:14:35 client-5 rshd[10220]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 00:14:35 client-5 xinetd[3391]: EXIT: shell status=0 pid=10219 duration=0(sec) May 17 00:14:36 client-5 xinetd[3391]: START: shell pid=10391 from=192.168.4.5 May 17 00:14:36 client-5 rshd[10392]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 4b: mkdir ...\/d4\/d2 =================================================================== 00:14:36 \(1305616476\)");echo XXRETCODE:$?' May 17 00:14:36 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 4b: mkdir .../d4/d2 =================================================================== 00:14:36 (1305616476) May 17 00:14:36 client-5 xinetd[3391]: EXIT: shell status=0 pid=10391 duration=0(sec) May 17 00:14:36 client-5 xinetd[3391]: START: shell pid=10484 from=192.168.4.5 May 17 00:14:36 client-5 rshd[10485]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 00:14:36 client-5 xinetd[3391]: EXIT: shell status=0 pid=10484 duration=0(sec) May 17 00:14:36 client-5 xinetd[3391]: START: shell pid=10655 from=192.168.4.5 May 17 00:14:36 client-5 rshd[10657]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 5: mkdir ...\/d5 ...\/d5\/d2\; chmod ...\/d5\/d2 ============================================ 00:14:36 \(1305616476\)");echo XXRETCODE:$?' May 17 00:14:36 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 5: mkdir .../d5 .../d5/d2; chmod .../d5/d2 ============================================ 00:14:36 (1305616476) May 17 00:14:36 client-5 xinetd[3391]: EXIT: shell status=0 pid=10655 duration=0(sec) May 17 00:14:36 client-5 xinetd[3391]: START: shell pid=10751 from=192.168.4.5 May 17 00:14:36 client-5 rshd[10752]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 00:14:36 client-5 xinetd[3391]: EXIT: shell status=0 pid=10751 duration=0(sec) May 17 00:14:37 client-5 xinetd[3391]: START: shell pid=10923 from=192.168.4.5 May 17 00:14:37 client-5 rshd[10924]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 6a: touch ...\/f6a\; chmod ...\/f6a ====================================================== 00:14:37 \(1305616477\)");echo XXRETCODE:$?' May 17 00:14:37 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 6a: touch .../f6a; chmod .../f6a ====================================================== 00:14:37 (1305616477) May 17 00:14:37 client-5 xinetd[3391]: EXIT: shell status=0 pid=10923 duration=0(sec) May 17 00:14:37 client-5 xinetd[3391]: START: shell pid=11017 from=192.168.4.5 May 17 00:14:37 client-5 rshd[11018]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 00:14:37 client-5 xinetd[3391]: EXIT: shell status=0 pid=11017 duration=0(sec) May 17 00:14:37 client-5 xinetd[3391]: START: shell pid=11189 from=192.168.4.5 May 17 00:14:37 client-5 rshd[11190]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 6b: runas -u 500 chmod ...\/f6a \(should return error\) ================================== 00:14:37 \(1305616477\)");echo XXRETCODE:$?' May 17 00:14:37 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 6b: runas -u 500 chmod .../f6a (should return error) ================================== 00:14:37 (1305616477) May 17 00:14:37 client-5 xinetd[3391]: EXIT: shell status=0 pid=11189 duration=0(sec) May 17 00:14:37 client-5 xinetd[3391]: START: shell pid=11282 from=192.168.4.5 May 17 00:14:37 client-5 rshd[11283]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 00:14:37 client-5 xinetd[3391]: EXIT: shell status=0 pid=11282 duration=0(sec) May 17 00:14:38 client-5 xinetd[3391]: START: shell pid=11452 from=192.168.4.5 May 17 00:14:38 client-5 rshd[11455]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 6c: touch ...\/f6c\; chown ...\/f6c ====================================================== 00:14:38 \(1305616478\)");echo XXRETCODE:$?' May 17 00:14:38 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 6c: touch .../f6c; chown .../f6c ====================================================== 00:14:38 (1305616478) May 17 00:14:38 client-5 xinetd[3391]: EXIT: shell status=0 pid=11452 duration=0(sec) May 17 00:14:38 client-5 xinetd[3391]: START: shell pid=11547 from=192.168.4.5 May 17 00:14:38 client-5 rshd[11549]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 00:14:38 client-5 xinetd[3391]: EXIT: shell status=0 pid=11547 duration=0(sec) May 17 00:14:38 client-5 xinetd[3391]: START: shell pid=11718 from=192.168.4.5 May 17 00:14:38 client-5 rshd[11721]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 6d: runas -u 500 chown ...\/f6c \(should return error\) ================================== 00:14:38 \(1305616478\)");echo XXRETCODE:$?' May 17 00:14:38 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 6d: runas -u 500 chown .../f6c (should return error) ================================== 00:14:38 (1305616478) May 17 00:14:38 client-5 xinetd[3391]: EXIT: shell status=0 pid=11718 duration=0(sec) May 17 00:14:38 client-5 xinetd[3391]: START: shell pid=11813 from=192.168.4.5 May 17 00:14:38 client-5 rshd[11814]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 00:14:38 client-5 xinetd[3391]: EXIT: shell status=0 pid=11813 duration=0(sec) May 17 00:14:39 client-5 xinetd[3391]: START: shell pid=11985 from=192.168.4.5 May 17 00:14:39 client-5 rshd[11986]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 6e: touch ...\/f6e\; chgrp ...\/f6e ====================================================== 00:14:39 \(1305616479\)");echo XXRETCODE:$?' May 17 00:14:39 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 6e: touch .../f6e; chgrp .../f6e ====================================================== 00:14:39 (1305616479) May 17 00:14:39 client-5 xinetd[3391]: EXIT: shell status=0 pid=11985 duration=0(sec) May 17 00:14:39 client-5 xinetd[3391]: START: shell pid=12079 from=192.168.4.5 May 17 00:14:39 client-5 rshd[12080]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 00:14:39 client-5 xinetd[3391]: EXIT: shell status=0 pid=12079 duration=0(sec) May 17 00:14:39 client-5 xinetd[3391]: START: shell pid=12251 from=192.168.4.5 May 17 00:14:39 client-5 rshd[12252]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 6f: runas -u 500 chgrp ...\/f6e \(should return error\) ================================== 00:14:39 \(1305616479\)");echo XXRETCODE:$?' May 17 00:14:39 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 6f: runas -u 500 chgrp .../f6e (should return error) ================================== 00:14:39 (1305616479) May 17 00:14:39 client-5 xinetd[3391]: EXIT: shell status=0 pid=12251 duration=0(sec) May 17 00:14:39 client-5 xinetd[3391]: START: shell pid=12343 from=192.168.4.5 May 17 00:14:39 client-5 rshd[12345]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 00:14:39 client-5 xinetd[3391]: EXIT: shell status=0 pid=12343 duration=0(sec) May 17 00:14:40 client-5 xinetd[3391]: START: shell pid=12516 from=192.168.4.5 May 17 00:14:40 client-5 rshd[12517]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 6g: Is new dir in sgid dir inheriting group? ========================================== 00:14:40 \(1305616480\)");echo XXRETCODE:$?' May 17 00:14:40 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 6g: Is new dir in sgid dir inheriting group? ========================================== 00:14:40 (1305616480) May 17 00:14:40 client-5 xinetd[3391]: EXIT: shell status=0 pid=12516 duration=0(sec) May 17 00:14:40 client-5 xinetd[3391]: START: shell pid=12612 from=192.168.4.5 May 17 00:14:40 client-5 rshd[12614]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 00:14:40 client-5 xinetd[3391]: EXIT: shell status=0 pid=12612 duration=0(sec) May 17 00:14:40 client-5 xinetd[3391]: START: shell pid=12785 from=192.168.4.5 May 17 00:14:40 client-5 rshd[12786]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 6h: runas -u 500 chown RUNAS_ID.0 ...\/f6h \(should return error\) ======================= 00:14:40 \(1305616480\)");echo XXRETCODE:$?' May 17 00:14:40 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 6h: runas -u 500 chown RUNAS_ID.0 .../f6h (should return error) ======================= 00:14:40 (1305616480) May 17 00:14:40 client-5 xinetd[3391]: EXIT: shell status=0 pid=12785 duration=0(sec) May 17 00:14:40 client-5 xinetd[3391]: START: shell pid=12879 from=192.168.4.5 May 17 00:14:40 client-5 rshd[12881]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 00:14:40 client-5 xinetd[3391]: EXIT: shell status=0 pid=12879 duration=0(sec) May 17 00:14:41 client-5 xinetd[3391]: START: shell pid=13052 from=192.168.4.5 May 17 00:14:41 client-5 rshd[13053]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 7a: mkdir ...\/d7\; mcreate ...\/d7\/f\; chmod ...\/d7\/f ==================================== 00:14:41 \(1305616481\)");echo XXRETCODE:$?' May 17 00:14:41 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 7a: mkdir .../d7; mcreate .../d7/f; chmod .../d7/f ==================================== 00:14:41 (1305616481) May 17 00:14:41 client-5 xinetd[3391]: EXIT: shell status=0 pid=13052 duration=0(sec) May 17 00:14:41 client-5 xinetd[3391]: START: shell pid=13147 from=192.168.4.5 May 17 00:14:41 client-5 rshd[13148]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 00:14:41 client-5 xinetd[3391]: EXIT: shell status=0 pid=13147 duration=0(sec) May 17 00:14:41 client-5 xinetd[3391]: START: shell pid=13317 from=192.168.4.5 May 17 00:14:41 client-5 rshd[13320]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 7b: mkdir ...\/d7\; mcreate d7\/f2\; echo foo \> d7\/f2 ===================================== 00:14:41 \(1305616481\)");echo XXRETCODE:$?' May 17 00:14:41 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 7b: mkdir .../d7; mcreate d7/f2; echo foo > d7/f2 ===================================== 00:14:41 (1305616481) May 17 00:14:41 client-5 xinetd[3391]: EXIT: shell status=0 pid=13317 duration=0(sec) May 17 00:14:41 client-5 xinetd[3391]: START: shell pid=13413 from=192.168.4.5 May 17 00:14:41 client-5 rshd[13414]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 00:14:42 client-5 xinetd[3391]: EXIT: shell status=0 pid=13413 duration=1(sec) May 17 00:14:42 client-5 xinetd[3391]: START: shell pid=13585 from=192.168.4.5 May 17 00:14:42 client-5 rshd[13586]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 8: mkdir ...\/d8\; touch ...\/d8\/f\; chmod ...\/d8\/f ======================================= 00:14:42 \(1305616482\)");echo XXRETCODE:$?' May 17 00:14:42 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 8: mkdir .../d8; touch .../d8/f; chmod .../d8/f ======================================= 00:14:42 (1305616482) May 17 00:14:42 client-5 xinetd[3391]: EXIT: shell status=0 pid=13585 duration=0(sec) May 17 00:14:42 client-5 xinetd[3391]: START: shell pid=13680 from=192.168.4.5 May 17 00:14:42 client-5 rshd[13681]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 00:14:42 client-5 xinetd[3391]: EXIT: shell status=0 pid=13680 duration=0(sec) May 17 00:14:42 client-5 xinetd[3391]: START: shell pid=13851 from=192.168.4.5 May 17 00:14:42 client-5 rshd[13853]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 9: mkdir ...\/d9 ...\/d9\/d2 ...\/d9\/d2\/d3 ================================================ 00:14:42 \(1305616482\)");echo XXRETCODE:$?' May 17 00:14:42 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 9: mkdir .../d9 .../d9/d2 .../d9/d2/d3 ================================================ 00:14:42 (1305616482) May 17 00:14:42 client-5 xinetd[3391]: EXIT: shell status=0 pid=13851 duration=0(sec) May 17 00:14:42 client-5 xinetd[3391]: START: shell pid=13947 from=192.168.4.5 May 17 00:14:42 client-5 rshd[13948]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 00:14:43 client-5 xinetd[3391]: EXIT: shell status=0 pid=13947 duration=1(sec) May 17 00:14:43 client-5 xinetd[3391]: START: shell pid=14119 from=192.168.4.5 May 17 00:14:43 client-5 rshd[14120]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 10: mkdir ...\/d10 ...\/d10\/d2\; touch ...\/d10\/d2\/f ====================================== 00:14:43 \(1305616483\)");echo XXRETCODE:$?' May 17 00:14:43 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 10: mkdir .../d10 .../d10/d2; touch .../d10/d2/f ====================================== 00:14:43 (1305616483) May 17 00:14:43 client-5 xinetd[3391]: EXIT: shell status=0 pid=14119 duration=0(sec) May 17 00:14:43 client-5 xinetd[3391]: START: shell pid=14212 from=192.168.4.5 May 17 00:14:43 client-5 rshd[14215]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 00:14:43 client-5 xinetd[3391]: EXIT: shell status=0 pid=14212 duration=0(sec) May 17 00:14:43 client-5 xinetd[3391]: START: shell pid=14384 from=192.168.4.5 May 17 00:14:43 client-5 rshd[14387]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 11: mkdir ...\/d11 d11\/d2\; chmod ...\/d11\/d2 ============================================ 00:14:43 \(1305616483\)");echo XXRETCODE:$?' May 17 00:14:43 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 11: mkdir .../d11 d11/d2; chmod .../d11/d2 ============================================ 00:14:43 (1305616483) May 17 00:14:43 client-5 xinetd[3391]: EXIT: shell status=0 pid=14384 duration=0(sec) May 17 00:14:43 client-5 xinetd[3391]: START: shell pid=14482 from=192.168.4.5 May 17 00:14:43 client-5 rshd[14483]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 00:14:44 client-5 xinetd[3391]: EXIT: shell status=0 pid=14482 duration=1(sec) May 17 00:14:44 client-5 xinetd[3391]: START: shell pid=14654 from=192.168.4.5 May 17 00:14:44 client-5 rshd[14655]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 12: touch ...\/d12\/f\; chmod ...\/d12\/f ...\/d12\/f ======================================== 00:14:44 \(1305616484\)");echo XXRETCODE:$?' May 17 00:14:44 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 12: touch .../d12/f; chmod .../d12/f .../d12/f ======================================== 00:14:44 (1305616484) May 17 00:14:44 client-5 xinetd[3391]: EXIT: shell status=0 pid=14654 duration=0(sec) May 17 00:14:44 client-5 xinetd[3391]: START: shell pid=14749 from=192.168.4.5 May 17 00:14:44 client-5 rshd[14751]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 00:14:44 client-5 xinetd[3391]: EXIT: shell status=0 pid=14749 duration=0(sec) May 17 00:14:44 client-5 xinetd[3391]: START: shell pid=14921 from=192.168.4.5 May 17 00:14:44 client-5 rshd[14924]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 13: creat ...\/d13\/f\; dd ...\/d13\/f\; \> ...\/d13\/f ======================================== 00:14:44 \(1305616484\)");echo XXRETCODE:$?' May 17 00:14:44 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 13: creat .../d13/f; dd .../d13/f; > .../d13/f ======================================== 00:14:44 (1305616484) May 17 00:14:44 client-5 xinetd[3391]: EXIT: shell status=0 pid=14921 duration=0(sec) May 17 00:14:44 client-5 xinetd[3391]: START: shell pid=15017 from=192.168.4.5 May 17 00:14:44 client-5 rshd[15018]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 00:14:45 client-5 xinetd[3391]: EXIT: shell status=0 pid=15017 duration=1(sec) May 17 00:14:45 client-5 xinetd[3391]: START: shell pid=15189 from=192.168.4.5 May 17 00:14:45 client-5 rshd[15190]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 14: touch ...\/d14\/f\; rm ...\/d14\/f\; rm ...\/d14\/f ======================================= 00:14:45 \(1305616485\)");echo XXRETCODE:$?' May 17 00:14:45 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 14: touch .../d14/f; rm .../d14/f; rm .../d14/f ======================================= 00:14:45 (1305616485) May 17 00:14:45 client-5 xinetd[3391]: EXIT: shell status=0 pid=15189 duration=0(sec) May 17 00:14:45 client-5 xinetd[3391]: START: shell pid=15284 from=192.168.4.5 May 17 00:14:45 client-5 rshd[15285]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 00:14:45 client-5 xinetd[3391]: EXIT: shell status=0 pid=15284 duration=0(sec) May 17 00:14:45 client-5 xinetd[3391]: START: shell pid=15456 from=192.168.4.5 May 17 00:14:45 client-5 rshd[15457]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 15: touch ...\/d15\/f\; mv ...\/d15\/f ...\/d15\/f2 ========================================== 00:14:45 \(1305616485\)");echo XXRETCODE:$?' May 17 00:14:45 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 15: touch .../d15/f; mv .../d15/f .../d15/f2 ========================================== 00:14:45 (1305616485) May 17 00:14:45 client-5 xinetd[3391]: EXIT: shell status=0 pid=15456 duration=0(sec) May 17 00:14:45 client-5 xinetd[3391]: START: shell pid=15550 from=192.168.4.5 May 17 00:14:45 client-5 rshd[15552]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 00:14:46 client-5 xinetd[3391]: EXIT: shell status=0 pid=15550 duration=1(sec) May 17 00:14:46 client-5 xinetd[3391]: START: shell pid=15723 from=192.168.4.5 May 17 00:14:46 client-5 rshd[15724]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 16: touch ...\/d16\/f\; rm -rf ...\/d16\/f ================================================= 00:14:46 \(1305616486\)");echo XXRETCODE:$?' May 17 00:14:46 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 16: touch .../d16/f; rm -rf .../d16/f ================================================= 00:14:46 (1305616486) May 17 00:14:46 client-5 xinetd[3391]: EXIT: shell status=0 pid=15723 duration=0(sec) May 17 00:14:46 client-5 xinetd[3391]: START: shell pid=15816 from=192.168.4.5 May 17 00:14:46 client-5 rshd[15819]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 00:14:46 client-5 xinetd[3391]: EXIT: shell status=0 pid=15816 duration=0(sec) May 17 00:14:46 client-5 xinetd[3391]: START: shell pid=15990 from=192.168.4.5 May 17 00:14:46 client-5 rshd[15991]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 17a: symlinks: create, remove \(real\) ================================================== 00:14:46 \(1305616486\)");echo XXRETCODE:$?' May 17 00:14:46 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 17a: symlinks: create, remove (real) ================================================== 00:14:46 (1305616486) May 17 00:14:46 client-5 xinetd[3391]: EXIT: shell status=0 pid=15990 duration=0(sec) May 17 00:14:47 client-5 xinetd[3391]: START: shell pid=16090 from=192.168.4.5 May 17 00:14:47 client-5 rshd[16091]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 00:14:47 client-5 xinetd[3391]: EXIT: shell status=0 pid=16090 duration=0(sec) May 17 00:14:47 client-5 xinetd[3391]: START: shell pid=16260 from=192.168.4.5 May 17 00:14:47 client-5 rshd[16263]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 17b: symlinks: create, remove \(dangling\) ============================================== 00:14:47 \(1305616487\)");echo XXRETCODE:$?' May 17 00:14:47 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 17b: symlinks: create, remove (dangling) ============================================== 00:14:47 (1305616487) May 17 00:14:47 client-5 xinetd[3391]: EXIT: shell status=0 pid=16260 duration=0(sec) May 17 00:14:47 client-5 xinetd[3391]: START: shell pid=16360 from=192.168.4.5 May 17 00:14:47 client-5 rshd[16362]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 00:14:47 client-5 xinetd[3391]: EXIT: shell status=0 pid=16360 duration=0(sec) May 17 00:14:47 client-5 xinetd[3391]: START: shell pid=16533 from=192.168.4.5 May 17 00:14:47 client-5 rshd[16534]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 17c: symlinks: open dangling \(should return error\) ==================================== 00:14:47 \(1305616487\)");echo XXRETCODE:$?' May 17 00:14:47 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 17c: symlinks: open dangling (should return error) ==================================== 00:14:47 (1305616487) May 17 00:14:47 client-5 xinetd[3391]: EXIT: shell status=0 pid=16533 duration=0(sec) May 17 00:14:48 client-5 xinetd[3391]: START: shell pid=16626 from=192.168.4.5 May 17 00:14:48 client-5 rshd[16628]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 00:14:48 client-5 xinetd[3391]: EXIT: shell status=0 pid=16626 duration=0(sec) May 17 00:14:48 client-5 xinetd[3391]: START: shell pid=16799 from=192.168.4.5 May 17 00:14:48 client-5 rshd[16800]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 17d: symlinks: create dangling ======================================================== 00:14:48 \(1305616488\)");echo XXRETCODE:$?' May 17 00:14:48 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 17d: symlinks: create dangling ======================================================== 00:14:48 (1305616488) May 17 00:14:48 client-5 xinetd[3391]: EXIT: shell status=0 pid=16799 duration=0(sec) May 17 00:14:48 client-5 xinetd[3391]: START: shell pid=16893 from=192.168.4.5 May 17 00:14:48 client-5 rshd[16894]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 00:14:48 client-5 xinetd[3391]: EXIT: shell status=0 pid=16893 duration=0(sec) May 17 00:14:48 client-5 xinetd[3391]: START: shell pid=17065 from=192.168.4.5 May 17 00:14:48 client-5 rshd[17066]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 17e: symlinks: create recursive symlink \(should return error\) ========================= 00:14:48 \(1305616488\)");echo XXRETCODE:$?' May 17 00:14:48 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 17e: symlinks: create recursive symlink (should return error) ========================= 00:14:48 (1305616488) May 17 00:14:48 client-5 xinetd[3391]: EXIT: shell status=0 pid=17065 duration=0(sec) May 17 00:14:49 client-5 xinetd[3391]: START: shell pid=17159 from=192.168.4.5 May 17 00:14:49 client-5 rshd[17161]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 00:14:49 client-5 xinetd[3391]: EXIT: shell status=0 pid=17159 duration=0(sec) May 17 00:14:49 client-5 xinetd[3391]: START: shell pid=17332 from=192.168.4.5 May 17 00:14:49 client-5 rshd[17333]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 17f: symlinks: long and very long symlink name ======================================== 00:14:49 \(1305616489\)");echo XXRETCODE:$?' May 17 00:14:49 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 17f: symlinks: long and very long symlink name ======================================== 00:14:49 (1305616489) May 17 00:14:49 client-5 xinetd[3391]: EXIT: shell status=0 pid=17332 duration=0(sec) May 17 00:14:49 client-5 xinetd[3391]: START: shell pid=17432 from=192.168.4.5 May 17 00:14:49 client-5 rshd[17433]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 00:14:49 client-5 xinetd[3391]: EXIT: shell status=0 pid=17432 duration=0(sec) May 17 00:14:49 client-5 xinetd[3391]: START: shell pid=17604 from=192.168.4.5 May 17 00:14:49 client-5 rshd[17605]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 17g: symlinks: really long symlink name =============================================== 00:14:49 \(1305616489\)");echo XXRETCODE:$?' May 17 00:14:50 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 17g: symlinks: really long symlink name =============================================== 00:14:49 (1305616489) May 17 00:14:50 client-5 xinetd[3391]: EXIT: shell status=0 pid=17604 duration=1(sec) May 17 00:14:50 client-5 xinetd[3391]: START: shell pid=17702 from=192.168.4.5 May 17 00:14:50 client-5 rshd[17703]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 00:14:50 client-5 xinetd[3391]: EXIT: shell status=0 pid=17702 duration=0(sec) May 17 00:14:50 client-5 xinetd[3391]: START: shell pid=17874 from=192.168.4.5 May 17 00:14:50 client-5 rshd[17875]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 17h: create objects: lov_free_memmd\(\) doesn\'t lbug ==================================== 00:14:50 \(1305616490\)");echo XXRETCODE:$?' May 17 00:14:50 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 17h: create objects: lov_free_memmd() doesn't lbug ==================================== 00:14:50 (1305616490) May 17 00:14:50 client-5 xinetd[3391]: EXIT: shell status=0 pid=17874 duration=0(sec) May 17 00:14:50 client-5 xinetd[3391]: START: shell pid=17976 from=192.168.4.5 May 17 00:14:50 client-5 rshd[17977]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 00:14:50 client-5 xinetd[3391]: EXIT: shell status=0 pid=17976 duration=0(sec) May 17 00:14:51 client-5 xinetd[3391]: START: shell pid=18148 from=192.168.4.5 May 17 00:14:51 client-5 rshd[18149]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 17i: don\'t panic on short symlink ===================================================== 00:14:51 \(1305616491\)");echo XXRETCODE:$?' May 17 00:14:51 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 17i: don't panic on short symlink ===================================================== 00:14:51 (1305616491) May 17 00:14:51 client-5 xinetd[3391]: EXIT: shell status=0 pid=18148 duration=0(sec) May 17 00:14:51 client-5 kernel: LustreError: 18193:0:(symlink.c:86:ll_readlink_internal()) inode 144115205255725124: symlink length 28 not expected 30 May 17 00:14:51 client-5 xinetd[3391]: START: shell pid=18250 from=192.168.4.5 May 17 00:14:51 client-5 rshd[18251]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 00:14:51 client-5 xinetd[3391]: EXIT: shell status=0 pid=18250 duration=0(sec) May 17 00:14:51 client-5 xinetd[3391]: START: shell pid=18422 from=192.168.4.5 May 17 00:14:51 client-5 rshd[18423]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 17k: symlinks: rsync with xattrs enabled ============================================== 00:14:51 \(1305616491\)");echo XXRETCODE:$?' May 17 00:14:51 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 17k: symlinks: rsync with xattrs enabled ============================================== 00:14:51 (1305616491) May 17 00:14:51 client-5 xinetd[3391]: EXIT: shell status=0 pid=18422 duration=0(sec) May 17 00:14:51 client-5 kernel: LustreError: 18462:0:(xattr.c:473:ll_getxattr()) server bug: replied size 176 > 100 for f17k (trusted.lov) May 17 00:14:51 client-5 kernel: cowardly refusing to write 4197 bytes in a page May 17 00:14:51 client-5 kernel: LustreError: 18462:0:(symlink.c:61:ll_readlink_internal()) using cached symlink xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx May 17 00:14:52 client-5 xinetd[3391]: START: shell pid=18522 from=192.168.4.5 May 17 00:14:52 client-5 rshd[18523]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 00:14:52 client-5 xinetd[3391]: EXIT: shell status=0 pid=18522 duration=0(sec) May 17 00:14:52 client-5 xinetd[3391]: START: shell pid=18694 from=192.168.4.5 May 17 00:14:52 client-5 rshd[18695]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 18: touch ...\/f \; ls ... ============================================================== 00:14:52 \(1305616492\)");echo XXRETCODE:$?' May 17 00:14:52 client-5 kernel: xxxxxxxxxxxxxxxxxxxxxxxx<4>Lustre: DEBUG MARKER: == sanity test 18: touch .../f ; ls ... ============================================================== 00:14:52 (1305616492) May 17 00:14:52 client-5 xinetd[3391]: EXIT: shell status=0 pid=18694 duration=0(sec) May 17 00:14:52 client-5 xinetd[3391]: START: shell pid=18785 from=192.168.4.5 May 17 00:14:52 client-5 rshd[18788]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 00:14:52 client-5 xinetd[3391]: EXIT: shell status=0 pid=18785 duration=0(sec) May 17 00:14:53 client-5 xinetd[3391]: START: shell pid=18957 from=192.168.4.5 May 17 00:14:53 client-5 rshd[18960]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 19a: touch ...\/f19 \; ls -l ... \; rm ...\/f19 =========================================== 00:14:52 \(1305616492\)");echo XXRETCODE:$?' May 17 00:14:53 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 19a: touch .../f19 ; ls -l ... ; rm .../f19 =========================================== 00:14:52 (1305616492) May 17 00:14:53 client-5 xinetd[3391]: EXIT: shell status=0 pid=18957 duration=0(sec) May 17 00:14:53 client-5 xinetd[3391]: START: shell pid=19055 from=192.168.4.5 May 17 00:14:53 client-5 rshd[19056]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 00:14:53 client-5 xinetd[3391]: EXIT: shell status=0 pid=19055 duration=0(sec) May 17 00:14:53 client-5 xinetd[3391]: START: shell pid=19227 from=192.168.4.5 May 17 00:14:53 client-5 rshd[19228]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 19b: ls -l ...\/f19 \(should return error\) ============================================== 00:14:53 \(1305616493\)");echo XXRETCODE:$?' May 17 00:14:53 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 19b: ls -l .../f19 (should return error) ============================================== 00:14:53 (1305616493) May 17 00:14:53 client-5 xinetd[3391]: EXIT: shell status=0 pid=19227 duration=0(sec) May 17 00:14:53 client-5 xinetd[3391]: START: shell pid=19319 from=192.168.4.5 May 17 00:14:53 client-5 rshd[19320]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 00:14:53 client-5 xinetd[3391]: EXIT: shell status=0 pid=19319 duration=0(sec) May 17 00:14:54 client-5 xinetd[3391]: START: shell pid=19491 from=192.168.4.5 May 17 00:14:54 client-5 rshd[19492]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 19c: runas -u 500 touch ...\/f19 \(should return error\) ================================= 00:14:53 \(1305616493\)");echo XXRETCODE:$?' May 17 00:14:54 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 19c: runas -u 500 touch .../f19 (should return error) ================================= 00:14:53 (1305616493) May 17 00:14:54 client-5 xinetd[3391]: EXIT: shell status=0 pid=19491 duration=0(sec) May 17 00:14:54 client-5 xinetd[3391]: START: shell pid=19583 from=192.168.4.5 May 17 00:14:54 client-5 rshd[19584]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 00:14:54 client-5 xinetd[3391]: EXIT: shell status=0 pid=19583 duration=0(sec) May 17 00:14:54 client-5 xinetd[3391]: START: shell pid=19755 from=192.168.4.5 May 17 00:14:54 client-5 rshd[19756]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 19d: cat ...\/f19 \(should return error\) ================================================ 00:14:54 \(1305616494\)");echo XXRETCODE:$?' May 17 00:14:54 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 19d: cat .../f19 (should return error) ================================================ 00:14:54 (1305616494) May 17 00:14:54 client-5 xinetd[3391]: EXIT: shell status=0 pid=19755 duration=0(sec) May 17 00:14:54 client-5 xinetd[3391]: START: shell pid=19846 from=192.168.4.5 May 17 00:14:54 client-5 rshd[19848]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 00:14:54 client-5 xinetd[3391]: EXIT: shell status=0 pid=19846 duration=0(sec) May 17 00:14:55 client-5 xinetd[3391]: START: shell pid=20019 from=192.168.4.5 May 17 00:14:55 client-5 rshd[20020]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 20: touch ...\/f \; ls -l ... =========================================================== 00:14:54 \(1305616494\)");echo XXRETCODE:$?' May 17 00:14:55 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 20: touch .../f ; ls -l ... =========================================================== 00:14:54 (1305616494) May 17 00:14:55 client-5 xinetd[3391]: EXIT: shell status=0 pid=20019 duration=0(sec) May 17 00:14:55 client-5 xinetd[3391]: START: shell pid=20114 from=192.168.4.5 May 17 00:14:55 client-5 rshd[20115]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark 1 done");echo XXRETCODE:$?' May 17 00:14:55 client-5 kernel: Lustre: DEBUG MARKER: 1 done May 17 00:14:55 client-5 xinetd[3391]: EXIT: shell status=0 pid=20114 duration=0(sec) May 17 00:14:55 client-5 xinetd[3391]: START: shell pid=20209 from=192.168.4.5 May 17 00:14:55 client-5 rshd[20210]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark 2 done");echo XXRETCODE:$?' May 17 00:14:55 client-5 kernel: Lustre: DEBUG MARKER: 2 done May 17 00:14:55 client-5 xinetd[3391]: EXIT: shell status=0 pid=20209 duration=0(sec) May 17 00:14:55 client-5 xinetd[3391]: START: shell pid=20304 from=192.168.4.5 May 17 00:14:55 client-5 rshd[20305]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark 3 done");echo XXRETCODE:$?' May 17 00:14:55 client-5 kernel: Lustre: DEBUG MARKER: 3 done May 17 00:14:55 client-5 xinetd[3391]: EXIT: shell status=0 pid=20304 duration=0(sec) May 17 00:14:55 client-5 xinetd[3391]: START: shell pid=20396 from=192.168.4.5 May 17 00:14:55 client-5 rshd[20398]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 00:14:55 client-5 xinetd[3391]: EXIT: shell status=0 pid=20396 duration=0(sec) May 17 00:14:56 client-5 xinetd[3391]: START: shell pid=20569 from=192.168.4.5 May 17 00:14:56 client-5 rshd[20570]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 21: write to dangling link ============================================================ 00:14:55 \(1305616495\)");echo XXRETCODE:$?' May 17 00:14:56 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 21: write to dangling link ============================================================ 00:14:55 (1305616495) May 17 00:14:56 client-5 xinetd[3391]: EXIT: shell status=0 pid=20569 duration=0(sec) May 17 00:14:56 client-5 xinetd[3391]: START: shell pid=20665 from=192.168.4.5 May 17 00:14:56 client-5 rshd[20666]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 00:14:56 client-5 xinetd[3391]: EXIT: shell status=0 pid=20665 duration=0(sec) May 17 00:14:56 client-5 xinetd[3391]: START: shell pid=20836 from=192.168.4.5 May 17 00:14:56 client-5 rshd[20838]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 22: unpack tar archive as non-root user =============================================== 00:14:56 \(1305616496\)");echo XXRETCODE:$?' May 17 00:14:56 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 22: unpack tar archive as non-root user =============================================== 00:14:56 (1305616496) May 17 00:14:56 client-5 xinetd[3391]: EXIT: shell status=0 pid=20836 duration=0(sec) May 17 00:14:56 client-5 xinetd[3391]: START: shell pid=20938 from=192.168.4.5 May 17 00:14:56 client-5 rshd[20939]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 00:14:56 client-5 xinetd[3391]: EXIT: shell status=0 pid=20938 duration=0(sec) May 17 00:14:57 client-5 xinetd[3391]: START: shell pid=21110 from=192.168.4.5 May 17 00:14:57 client-5 rshd[21111]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 23a: O_CREAT\|O_EXCL in subdir ========================================================= 00:14:57 \(1305616497\)");echo XXRETCODE:$?' May 17 00:14:57 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 23a: O_CREAT|O_EXCL in subdir ========================================================= 00:14:57 (1305616497) May 17 00:14:57 client-5 xinetd[3391]: EXIT: shell status=0 pid=21110 duration=0(sec) May 17 00:14:57 client-5 xinetd[3391]: START: shell pid=21204 from=192.168.4.5 May 17 00:14:57 client-5 rshd[21205]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 00:14:57 client-5 xinetd[3391]: EXIT: shell status=0 pid=21204 duration=0(sec) May 17 00:14:57 client-5 xinetd[3391]: START: shell pid=21376 from=192.168.4.5 May 17 00:14:57 client-5 rshd[21377]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 23b: O_APPEND check =================================================================== 00:14:57 \(1305616497\)");echo XXRETCODE:$?' May 17 00:14:57 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 23b: O_APPEND check =================================================================== 00:14:57 (1305616497) May 17 00:14:57 client-5 xinetd[3391]: EXIT: shell status=0 pid=21376 duration=0(sec) May 17 00:14:57 client-5 xinetd[3391]: START: shell pid=21470 from=192.168.4.5 May 17 00:14:57 client-5 rshd[21472]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 00:14:57 client-5 xinetd[3391]: EXIT: shell status=0 pid=21470 duration=0(sec) May 17 00:14:58 client-5 xinetd[3391]: START: shell pid=21643 from=192.168.4.5 May 17 00:14:58 client-5 rshd[21644]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 24a: touch ...\/R1\/f\; rename ...\/R1\/f ...\/R1\/g ========================================= 00:14:58 \(1305616498\)");echo XXRETCODE:$?' May 17 00:14:58 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 24a: touch .../R1/f; rename .../R1/f .../R1/g ========================================= 00:14:58 (1305616498) May 17 00:14:58 client-5 xinetd[3391]: EXIT: shell status=0 pid=21643 duration=0(sec) May 17 00:14:58 client-5 xinetd[3391]: START: shell pid=21736 from=192.168.4.5 May 17 00:14:58 client-5 rshd[21739]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 00:14:58 client-5 xinetd[3391]: EXIT: shell status=0 pid=21736 duration=0(sec) May 17 00:14:58 client-5 xinetd[3391]: START: shell pid=21910 from=192.168.4.5 May 17 00:14:58 client-5 rshd[21911]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 24b: touch ...\/R2\/{f,g}\; rename ...\/R2\/f ...\/R2\/g ===================================== 00:14:58 \(1305616498\)");echo XXRETCODE:$?' May 17 00:14:58 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 24b: touch .../R2/f; .../R2/g; rename .../R2/f .../R2/g ===================================== 00:14:58 (1305616498) May 17 00:14:58 client-5 xinetd[3391]: EXIT: shell status=0 pid=21910 duration=0(sec) May 17 00:14:58 client-5 xinetd[3391]: START: shell pid=22006 from=192.168.4.5 May 17 00:14:58 client-5 rshd[22007]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 00:14:58 client-5 xinetd[3391]: EXIT: shell status=0 pid=22006 duration=0(sec) May 17 00:14:59 client-5 xinetd[3391]: START: shell pid=22178 from=192.168.4.5 May 17 00:14:59 client-5 rshd[22179]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 24c: mkdir ...\/R3\/f\; rename ...\/R3\/f ...\/R3\/g ========================================= 00:14:59 \(1305616499\)");echo XXRETCODE:$?' May 17 00:14:59 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 24c: mkdir .../R3/f; rename .../R3/f .../R3/g ========================================= 00:14:59 (1305616499) May 17 00:14:59 client-5 xinetd[3391]: EXIT: shell status=0 pid=22178 duration=0(sec) May 17 00:14:59 client-5 xinetd[3391]: START: shell pid=22272 from=192.168.4.5 May 17 00:14:59 client-5 rshd[22275]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 00:14:59 client-5 xinetd[3391]: EXIT: shell status=0 pid=22272 duration=0(sec) May 17 00:14:59 client-5 xinetd[3391]: START: shell pid=22446 from=192.168.4.5 May 17 00:14:59 client-5 rshd[22447]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 24d: mkdir ...\/R4\/{f,g}\; rename ...\/R4\/f ...\/R4\/g ===================================== 00:14:59 \(1305616499\)");echo XXRETCODE:$?' May 17 00:14:59 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 24d: mkdir .../R4/f; .../R4/g; rename .../R4/f .../R4/g ===================================== 00:14:59 (1305616499) May 17 00:14:59 client-5 xinetd[3391]: EXIT: shell status=0 pid=22446 duration=0(sec) May 17 00:14:59 client-5 xinetd[3391]: START: shell pid=22542 from=192.168.4.5 May 17 00:14:59 client-5 rshd[22543]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 00:15:00 client-5 xinetd[3391]: EXIT: shell status=0 pid=22542 duration=1(sec) May 17 00:15:00 client-5 xinetd[3391]: START: shell pid=22713 from=192.168.4.5 May 17 00:15:00 client-5 rshd[22715]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 24e: touch ...\/R5a\/f\; rename ...\/R5a\/f ...\/R5b\/g ====================================== 00:15:00 \(1305616500\)");echo XXRETCODE:$?' May 17 00:15:00 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 24e: touch .../R5a/f; rename .../R5a/f .../R5b/g ====================================== 00:15:00 (1305616500) May 17 00:15:00 client-5 xinetd[3391]: EXIT: shell status=0 pid=22713 duration=0(sec) May 17 00:15:00 client-5 xinetd[3391]: START: shell pid=22809 from=192.168.4.5 May 17 00:15:00 client-5 rshd[22811]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 00:15:00 client-5 xinetd[3391]: EXIT: shell status=0 pid=22809 duration=0(sec) May 17 00:15:00 client-5 xinetd[3391]: START: shell pid=22981 from=192.168.4.5 May 17 00:15:00 client-5 rshd[22983]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 24f: touch ...\/R6a\/f R6b\/g\; mv ...\/R6a\/f ...\/R6b\/g ==================================== 00:15:00 \(1305616500\)");echo XXRETCODE:$?' May 17 00:15:00 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 24f: touch .../R6a/f R6b/g; mv .../R6a/f .../R6b/g ==================================== 00:15:00 (1305616500) May 17 00:15:00 client-5 xinetd[3391]: EXIT: shell status=0 pid=22981 duration=0(sec) May 17 00:15:00 client-5 xinetd[3391]: START: shell pid=23078 from=192.168.4.5 May 17 00:15:00 client-5 rshd[23079]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 00:15:01 client-5 xinetd[3391]: EXIT: shell status=0 pid=23078 duration=1(sec) May 17 00:15:01 client-5 xinetd[3391]: START: shell pid=23250 from=192.168.4.5 May 17 00:15:01 client-5 rshd[23251]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 24g: mkdir ...\/R7{a,b}\/d\; mv ...\/R7a\/d ...\/R7b\/e ====================================== 00:15:01 \(1305616501\)");echo XXRETCODE:$?' May 17 00:15:01 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 24g: mkdir .../R7a/d; .../R7b/d; mv .../R7a/d .../R7b/e ====================================== 00:15:01 (1305616501) May 17 00:15:01 client-5 xinetd[3391]: EXIT: shell status=0 pid=23250 duration=0(sec) May 17 00:15:01 client-5 xinetd[3391]: START: shell pid=23345 from=192.168.4.5 May 17 00:15:01 client-5 rshd[23347]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 00:15:01 client-5 xinetd[3391]: EXIT: shell status=0 pid=23345 duration=0(sec) May 17 00:15:01 client-5 xinetd[3391]: START: shell pid=23516 from=192.168.4.5 May 17 00:15:01 client-5 rshd[23519]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 24h: mkdir ...\/R8{a,b}\/{d,e}\; rename ...\/R8a\/d ...\/R8b\/e ============================== 00:15:01 \(1305616501\)");echo XXRETCODE:$?' May 17 00:15:01 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 24h: mkdir .../R8a/d; .../R8a/e; .../R8b/d; .../R8b/e; rename .../R8a/d .../R8b/e ============================== 00:15:01 (1305616501) May 17 00:15:01 client-5 xinetd[3391]: EXIT: shell status=0 pid=23516 duration=0(sec) May 17 00:15:01 client-5 xinetd[3391]: START: shell pid=23613 from=192.168.4.5 May 17 00:15:02 client-5 rshd[23615]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 00:15:02 client-5 xinetd[3391]: EXIT: shell status=0 pid=23613 duration=1(sec) May 17 00:15:02 client-5 xinetd[3391]: START: shell pid=23785 from=192.168.4.5 May 17 00:15:02 client-5 rshd[23787]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 24i: rename file to dir error: touch f \; mkdir a \; rename f a ========================= 00:15:02 \(1305616502\)");echo XXRETCODE:$?' May 17 00:15:02 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 24i: rename file to dir error: touch f ; mkdir a ; rename f a ========================= 00:15:02 (1305616502) May 17 00:15:02 client-5 xinetd[3391]: EXIT: shell status=0 pid=23785 duration=0(sec) May 17 00:15:02 client-5 xinetd[3391]: START: shell pid=23884 from=192.168.4.5 May 17 00:15:02 client-5 rshd[23885]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 00:15:02 client-5 xinetd[3391]: EXIT: shell status=0 pid=23884 duration=0(sec) May 17 00:15:02 client-5 xinetd[3391]: START: shell pid=24056 from=192.168.4.5 May 17 00:15:02 client-5 rshd[24057]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 24j: source does not exist ============================================================ 00:15:02 \(1305616502\)");echo XXRETCODE:$?' May 17 00:15:02 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 24j: source does not exist ============================================================ 00:15:02 (1305616502) May 17 00:15:02 client-5 xinetd[3391]: EXIT: shell status=0 pid=24056 duration=0(sec) May 17 00:15:03 client-5 xinetd[3391]: START: shell pid=24152 from=192.168.4.5 May 17 00:15:03 client-5 rshd[24153]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 00:15:03 client-5 xinetd[3391]: EXIT: shell status=0 pid=24152 duration=0(sec) May 17 00:15:03 client-5 xinetd[3391]: START: shell pid=24324 from=192.168.4.5 May 17 00:15:03 client-5 rshd[24325]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 24k: touch ...\/R11a\/f\; mv ...\/R11a\/f ...\/R11a\/d ======================================= 00:15:03 \(1305616503\)");echo XXRETCODE:$?' May 17 00:15:03 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 24k: touch .../R11a/f; mv .../R11a/f .../R11a/d ======================================= 00:15:03 (1305616503) May 17 00:15:03 client-5 xinetd[3391]: EXIT: shell status=0 pid=24324 duration=0(sec) May 17 00:15:03 client-5 xinetd[3391]: START: shell pid=24419 from=192.168.4.5 May 17 00:15:03 client-5 rshd[24421]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 00:15:03 client-5 xinetd[3391]: EXIT: shell status=0 pid=24419 duration=0(sec) May 17 00:15:03 client-5 xinetd[3391]: START: shell pid=24592 from=192.168.4.5 May 17 00:15:03 client-5 rshd[24593]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 24l: Renaming a file to itself ======================================================== 00:15:03 \(1305616503\)");echo XXRETCODE:$?' May 17 00:15:03 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 24l: Renaming a file to itself ======================================================== 00:15:03 (1305616503) May 17 00:15:03 client-5 xinetd[3391]: EXIT: shell status=0 pid=24592 duration=0(sec) May 17 00:15:04 client-5 xinetd[3391]: START: shell pid=24683 from=192.168.4.5 May 17 00:15:04 client-5 rshd[24685]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 00:15:04 client-5 xinetd[3391]: EXIT: shell status=0 pid=24683 duration=0(sec) May 17 00:15:04 client-5 xinetd[3391]: START: shell pid=24856 from=192.168.4.5 May 17 00:15:04 client-5 rshd[24857]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 24m: Renaming a file to a hard link to itself ========================================= 00:15:04 \(1305616504\)");echo XXRETCODE:$?' May 17 00:15:04 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 24m: Renaming a file to a hard link to itself ========================================= 00:15:04 (1305616504) May 17 00:15:04 client-5 xinetd[3391]: EXIT: shell status=0 pid=24856 duration=0(sec) May 17 00:15:04 client-5 xinetd[3391]: START: shell pid=24948 from=192.168.4.5 May 17 00:15:04 client-5 rshd[24951]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 00:15:04 client-5 xinetd[3391]: EXIT: shell status=0 pid=24948 duration=0(sec) May 17 00:15:04 client-5 xinetd[3391]: START: shell pid=25121 from=192.168.4.5 May 17 00:15:04 client-5 rshd[25123]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 24n: Statting the old file after renaming \(Posix rename 2\) ============================ 00:15:04 \(1305616504\)");echo XXRETCODE:$?' May 17 00:15:05 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 24n: Statting the old file after renaming (Posix rename 2) ============================ 00:15:04 (1305616504) May 17 00:15:05 client-5 xinetd[3391]: EXIT: shell status=0 pid=25121 duration=1(sec) May 17 00:15:05 client-5 xinetd[3391]: START: shell pid=25218 from=192.168.4.5 May 17 00:15:05 client-5 rshd[25219]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 00:15:05 client-5 xinetd[3391]: EXIT: shell status=0 pid=25218 duration=0(sec) May 17 00:15:05 client-5 xinetd[3391]: START: shell pid=25388 from=192.168.4.5 May 17 00:15:05 client-5 rshd[25391]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 24o: rename of files during htree split =============================================== 00:15:05 \(1305616505\)");echo XXRETCODE:$?' May 17 00:15:05 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 24o: rename of files during htree split =============================================== 00:15:05 (1305616505) May 17 00:15:05 client-5 xinetd[3391]: EXIT: shell status=0 pid=25388 duration=0(sec) May 17 00:15:40 client-5 ntpd[3402]: synchronized to 10.10.0.1, stratum 2 May 17 01:15:40 client-5 ntpd[3402]: time reset +3600.013438 s May 17 01:15:40 client-5 ntpd[3402]: kernel time sync enabled 0001 May 17 01:16:05 client-5 xinetd[3391]: START: shell pid=25492 from=192.168.4.5 May 17 01:16:05 client-5 rshd[25493]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:16:05 client-5 xinetd[3391]: EXIT: shell status=0 pid=25492 duration=0(sec) May 17 01:16:05 client-5 xinetd[3391]: START: shell pid=25664 from=192.168.4.5 May 17 01:16:05 client-5 rshd[25665]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 24p: mkdir ...\/R12{a,b}\; rename ...\/R12a ...\/R12b ===================================== 01:16:05 \(1305620165\)");echo XXRETCODE:$?' May 17 01:16:05 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 24p: mkdir .../R12a; .../R12b; rename .../R12a .../R12b ===================================== 01:16:05 (1305620165) May 17 01:16:05 client-5 xinetd[3391]: EXIT: shell status=0 pid=25664 duration=0(sec) May 17 01:16:05 client-5 xinetd[3391]: START: shell pid=25765 from=192.168.4.5 May 17 01:16:05 client-5 rshd[25766]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:16:06 client-5 xinetd[3391]: EXIT: shell status=0 pid=25765 duration=1(sec) May 17 01:16:06 client-5 xinetd[3391]: START: shell pid=25937 from=192.168.4.5 May 17 01:16:06 client-5 rshd[25938]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 24q: mkdir ...\/R13{a,b}\; open R13b rename R13a R13b =================================== 01:16:06 \(1305620166\)");echo XXRETCODE:$?' May 17 01:16:06 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 24q: mkdir .../R13a; .../R13b; open R13b rename R13a R13b =================================== 01:16:06 (1305620166) May 17 01:16:06 client-5 xinetd[3391]: EXIT: shell status=0 pid=25937 duration=0(sec) May 17 01:16:06 client-5 xinetd[3391]: START: shell pid=26041 from=192.168.4.5 May 17 01:16:06 client-5 rshd[26042]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:16:06 client-5 xinetd[3391]: EXIT: shell status=0 pid=26041 duration=0(sec) May 17 01:16:06 client-5 xinetd[3391]: START: shell pid=26213 from=192.168.4.5 May 17 01:16:06 client-5 rshd[26214]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 24r: mkdir ...\/R14a\/b\; rename ...\/R14a ...\/R14a\/b ===================================== 01:16:06 \(1305620166\)");echo XXRETCODE:$?' May 17 01:16:06 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 24r: mkdir .../R14a/b; rename .../R14a .../R14a/b ===================================== 01:16:06 (1305620166) May 17 01:16:06 client-5 xinetd[3391]: EXIT: shell status=0 pid=26213 duration=0(sec) May 17 01:16:07 client-5 xinetd[3391]: START: shell pid=26308 from=192.168.4.5 May 17 01:16:07 client-5 rshd[26309]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:16:07 client-5 xinetd[3391]: EXIT: shell status=0 pid=26308 duration=0(sec) May 17 01:16:07 client-5 xinetd[3391]: START: shell pid=26480 from=192.168.4.5 May 17 01:16:07 client-5 rshd[26481]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 24s: mkdir ...\/R15a\/b\/c\; rename ...\/R15a ...\/R15a\/b\/c ================================= 01:16:07 \(1305620167\)");echo XXRETCODE:$?' May 17 01:16:07 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 24s: mkdir .../R15a/b/c; rename .../R15a .../R15a/b/c ================================= 01:16:07 (1305620167) May 17 01:16:07 client-5 xinetd[3391]: EXIT: shell status=0 pid=26480 duration=0(sec) May 17 01:16:07 client-5 xinetd[3391]: START: shell pid=26574 from=192.168.4.5 May 17 01:16:07 client-5 rshd[26576]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:16:07 client-5 xinetd[3391]: EXIT: shell status=0 pid=26574 duration=0(sec) May 17 01:16:07 client-5 xinetd[3391]: START: shell pid=26747 from=192.168.4.5 May 17 01:16:07 client-5 rshd[26748]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 24t: mkdir ...\/R16a\/b\/c\; rename ...\/R16a\/b\/c ...\/R16a ================================= 01:16:07 \(1305620167\)");echo XXRETCODE:$?' May 17 01:16:07 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 24t: mkdir .../R16a/b/c; rename .../R16a/b/c .../R16a ================================= 01:16:07 (1305620167) May 17 01:16:07 client-5 xinetd[3391]: EXIT: shell status=0 pid=26747 duration=0(sec) May 17 01:16:08 client-5 xinetd[3391]: START: shell pid=26840 from=192.168.4.5 May 17 01:16:08 client-5 rshd[26843]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:16:08 client-5 xinetd[3391]: EXIT: shell status=0 pid=26840 duration=0(sec) May 17 01:16:08 client-5 xinetd[3391]: START: shell pid=27012 from=192.168.4.5 May 17 01:16:08 client-5 rshd[27015]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 24u: create stripe file =============================================================== 01:16:08 \(1305620168\)");echo XXRETCODE:$?' May 17 01:16:08 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 24u: create stripe file =============================================================== 01:16:08 (1305620168) May 17 01:16:08 client-5 xinetd[3391]: EXIT: shell status=0 pid=27012 duration=0(sec) May 17 01:16:08 client-5 xinetd[3391]: START: shell pid=27107 from=192.168.4.5 May 17 01:16:08 client-5 rshd[27108]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:16:08 client-5 xinetd[3391]: EXIT: shell status=0 pid=27107 duration=0(sec) May 17 01:16:08 client-5 xinetd[3391]: START: shell pid=27278 from=192.168.4.5 May 17 01:16:09 client-5 rshd[27280]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 24v: list directory with large files \(handle hash collision, bug: 17560\) ============== 01:16:08 \(1305620168\)");echo XXRETCODE:$?' May 17 01:16:09 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 24v: list directory with large files (handle hash collision, bug: 17560) ============== 01:16:08 (1305620168) May 17 01:16:09 client-5 xinetd[3391]: EXIT: shell status=0 pid=27278 duration=1(sec) May 17 01:20:38 client-5 ntpd[3402]: synchronized to 10.10.0.2, stratum 3 May 17 01:21:23 client-5 xinetd[3391]: START: shell pid=27393 from=192.168.4.5 May 17 01:21:23 client-5 rshd[27396]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:21:23 client-5 xinetd[3391]: EXIT: shell status=0 pid=27393 duration=0(sec) May 17 01:21:23 client-5 xinetd[3391]: START: shell pid=27567 from=192.168.4.5 May 17 01:21:23 client-5 rshd[27568]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 24w: Reading a file larger than 4Gb =================================================== 01:21:23 \(1305620483\)");echo XXRETCODE:$?' May 17 01:21:24 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 24w: Reading a file larger than 4Gb =================================================== 01:21:23 (1305620483) May 17 01:21:24 client-5 xinetd[3391]: EXIT: shell status=0 pid=27567 duration=1(sec) May 17 01:21:24 client-5 xinetd[3391]: START: shell pid=27664 from=192.168.4.5 May 17 01:21:24 client-5 rshd[27665]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:21:24 client-5 xinetd[3391]: EXIT: shell status=0 pid=27664 duration=0(sec) May 17 01:21:24 client-5 xinetd[3391]: START: shell pid=27836 from=192.168.4.5 May 17 01:21:24 client-5 rshd[27837]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 25a: create file in symlinked directory =============================================== 01:21:24 \(1305620484\)");echo XXRETCODE:$?' May 17 01:21:24 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 25a: create file in symlinked directory =============================================== 01:21:24 (1305620484) May 17 01:21:24 client-5 xinetd[3391]: EXIT: shell status=0 pid=27836 duration=0(sec) May 17 01:21:24 client-5 xinetd[3391]: START: shell pid=27929 from=192.168.4.5 May 17 01:21:24 client-5 rshd[27931]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:21:24 client-5 xinetd[3391]: EXIT: shell status=0 pid=27929 duration=0(sec) May 17 01:21:25 client-5 xinetd[3391]: START: shell pid=28102 from=192.168.4.5 May 17 01:21:25 client-5 rshd[28103]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 25b: lookup file in symlinked directory =============================================== 01:21:24 \(1305620484\)");echo XXRETCODE:$?' May 17 01:21:25 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 25b: lookup file in symlinked directory =============================================== 01:21:24 (1305620484) May 17 01:21:25 client-5 xinetd[3391]: EXIT: shell status=0 pid=28102 duration=0(sec) May 17 01:21:25 client-5 xinetd[3391]: START: shell pid=28193 from=192.168.4.5 May 17 01:21:25 client-5 rshd[28195]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:21:25 client-5 xinetd[3391]: EXIT: shell status=0 pid=28193 duration=0(sec) May 17 01:21:25 client-5 xinetd[3391]: START: shell pid=28364 from=192.168.4.5 May 17 01:21:25 client-5 rshd[28367]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 26a: multiple component symlink ======================================================= 01:21:25 \(1305620485\)");echo XXRETCODE:$?' May 17 01:21:25 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 26a: multiple component symlink ======================================================= 01:21:25 (1305620485) May 17 01:21:25 client-5 xinetd[3391]: EXIT: shell status=0 pid=28364 duration=0(sec) May 17 01:21:25 client-5 xinetd[3391]: START: shell pid=28460 from=192.168.4.5 May 17 01:21:25 client-5 rshd[28462]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:21:25 client-5 xinetd[3391]: EXIT: shell status=0 pid=28460 duration=0(sec) May 17 01:21:26 client-5 xinetd[3391]: START: shell pid=28633 from=192.168.4.5 May 17 01:21:26 client-5 rshd[28634]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 26b: multiple component symlink at end of lookup ====================================== 01:21:25 \(1305620485\)");echo XXRETCODE:$?' May 17 01:21:26 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 26b: multiple component symlink at end of lookup ====================================== 01:21:25 (1305620485) May 17 01:21:26 client-5 xinetd[3391]: EXIT: shell status=0 pid=28633 duration=0(sec) May 17 01:21:26 client-5 xinetd[3391]: START: shell pid=28726 from=192.168.4.5 May 17 01:21:26 client-5 rshd[28728]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:21:26 client-5 xinetd[3391]: EXIT: shell status=0 pid=28726 duration=0(sec) May 17 01:21:26 client-5 xinetd[3391]: START: shell pid=28899 from=192.168.4.5 May 17 01:21:26 client-5 rshd[28900]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 26c: chain of symlinks ================================================================ 01:21:26 \(1305620486\)");echo XXRETCODE:$?' May 17 01:21:26 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 26c: chain of symlinks ================================================================ 01:21:26 (1305620486) May 17 01:21:26 client-5 xinetd[3391]: EXIT: shell status=0 pid=28899 duration=0(sec) May 17 01:21:26 client-5 xinetd[3391]: START: shell pid=28996 from=192.168.4.5 May 17 01:21:26 client-5 rshd[28997]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:21:26 client-5 xinetd[3391]: EXIT: shell status=0 pid=28996 duration=0(sec) May 17 01:21:27 client-5 xinetd[3391]: START: shell pid=29168 from=192.168.4.5 May 17 01:21:27 client-5 rshd[29169]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 26d: create multiple component recursive symlink ====================================== 01:21:27 \(1305620487\)");echo XXRETCODE:$?' May 17 01:21:27 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 26d: create multiple component recursive symlink ====================================== 01:21:27 (1305620487) May 17 01:21:27 client-5 xinetd[3391]: EXIT: shell status=0 pid=29168 duration=0(sec) May 17 01:21:27 client-5 xinetd[3391]: START: shell pid=29259 from=192.168.4.5 May 17 01:21:27 client-5 rshd[29261]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:21:27 client-5 xinetd[3391]: EXIT: shell status=0 pid=29259 duration=0(sec) May 17 01:21:27 client-5 xinetd[3391]: START: shell pid=29432 from=192.168.4.5 May 17 01:21:27 client-5 rshd[29433]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 26e: unlink multiple component recursive symlink ====================================== 01:21:27 \(1305620487\)");echo XXRETCODE:$?' May 17 01:21:27 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 26e: unlink multiple component recursive symlink ====================================== 01:21:27 (1305620487) May 17 01:21:27 client-5 xinetd[3391]: EXIT: shell status=0 pid=29432 duration=0(sec) May 17 01:21:27 client-5 xinetd[3391]: START: shell pid=29524 from=192.168.4.5 May 17 01:21:27 client-5 rshd[29525]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:21:27 client-5 xinetd[3391]: EXIT: shell status=0 pid=29524 duration=0(sec) May 17 01:21:28 client-5 xinetd[3391]: START: shell pid=29694 from=192.168.4.5 May 17 01:21:28 client-5 rshd[29697]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 26f: rm -r of a directory which has recursive symlink ================================= 01:21:28 \(1305620488\)");echo XXRETCODE:$?' May 17 01:21:28 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 26f: rm -r of a directory which has recursive symlink ================================= 01:21:28 (1305620488) May 17 01:21:28 client-5 xinetd[3391]: EXIT: shell status=0 pid=29694 duration=0(sec) May 17 01:21:28 client-5 xinetd[3391]: START: shell pid=29801 from=192.168.4.5 May 17 01:21:28 client-5 rshd[29802]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:21:28 client-5 xinetd[3391]: EXIT: shell status=0 pid=29801 duration=0(sec) May 17 01:21:28 client-5 xinetd[3391]: START: shell pid=29973 from=192.168.4.5 May 17 01:21:28 client-5 rshd[29974]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 27a: one stripe file ================================================================== 01:21:28 \(1305620488\)");echo XXRETCODE:$?' May 17 01:21:28 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 27a: one stripe file ================================================================== 01:21:28 (1305620488) May 17 01:21:28 client-5 xinetd[3391]: EXIT: shell status=0 pid=29973 duration=0(sec) May 17 01:21:28 client-5 xinetd[3391]: START: shell pid=30072 from=192.168.4.5 May 17 01:21:28 client-5 rshd[30073]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == test_27a: write to one stripe file =========================");echo XXRETCODE:$?' May 17 01:21:29 client-5 kernel: Lustre: DEBUG MARKER: == test_27a: write to one stripe file ========================= May 17 01:21:29 client-5 xinetd[3391]: EXIT: shell status=0 pid=30072 duration=1(sec) May 17 01:21:29 client-5 xinetd[3391]: START: shell pid=30163 from=192.168.4.5 May 17 01:21:29 client-5 rshd[30165]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:21:29 client-5 xinetd[3391]: EXIT: shell status=0 pid=30163 duration=0(sec) May 17 01:21:29 client-5 xinetd[3391]: START: shell pid=30336 from=192.168.4.5 May 17 01:21:29 client-5 rshd[30337]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 27c: create two stripe file f01 ======================================================= 01:21:29 \(1305620489\)");echo XXRETCODE:$?' May 17 01:21:29 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 27c: create two stripe file f01 ======================================================= 01:21:29 (1305620489) May 17 01:21:29 client-5 xinetd[3391]: EXIT: shell status=0 pid=30336 duration=0(sec) May 17 01:21:29 client-5 xinetd[3391]: START: shell pid=30437 from=192.168.4.5 May 17 01:21:29 client-5 rshd[30438]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == test_27c: write to two stripe file file f01 ================");echo XXRETCODE:$?' May 17 01:21:29 client-5 kernel: Lustre: DEBUG MARKER: == test_27c: write to two stripe file file f01 ================ May 17 01:21:29 client-5 xinetd[3391]: EXIT: shell status=0 pid=30437 duration=0(sec) May 17 01:21:29 client-5 xinetd[3391]: START: shell pid=30528 from=192.168.4.5 May 17 01:21:29 client-5 rshd[30530]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:21:29 client-5 xinetd[3391]: EXIT: shell status=0 pid=30528 duration=0(sec) May 17 01:21:30 client-5 xinetd[3391]: START: shell pid=30701 from=192.168.4.5 May 17 01:21:30 client-5 rshd[30702]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 27d: create file with default settings ================================================ 01:21:30 \(1305620490\)");echo XXRETCODE:$?' May 17 01:21:30 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 27d: create file with default settings ================================================ 01:21:30 (1305620490) May 17 01:21:30 client-5 xinetd[3391]: EXIT: shell status=0 pid=30701 duration=0(sec) May 17 01:21:30 client-5 xinetd[3391]: START: shell pid=30796 from=192.168.4.5 May 17 01:21:30 client-5 rshd[30797]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:21:30 client-5 xinetd[3391]: EXIT: shell status=0 pid=30796 duration=0(sec) May 17 01:21:30 client-5 xinetd[3391]: START: shell pid=30968 from=192.168.4.5 May 17 01:21:30 client-5 rshd[30969]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 27e: setstripe existing file \(should return error\) ==================================== 01:21:30 \(1305620490\)");echo XXRETCODE:$?' May 17 01:21:30 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 27e: setstripe existing file (should return error) ==================================== 01:21:30 (1305620490) May 17 01:21:30 client-5 xinetd[3391]: EXIT: shell status=0 pid=30968 duration=0(sec) May 17 01:21:30 client-5 xinetd[3391]: START: shell pid=31061 from=192.168.4.5 May 17 01:21:30 client-5 rshd[31064]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:21:30 client-5 xinetd[3391]: EXIT: shell status=0 pid=31061 duration=0(sec) May 17 01:21:31 client-5 xinetd[3391]: START: shell pid=31235 from=192.168.4.5 May 17 01:21:31 client-5 rshd[31236]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 27f: setstripe with bad stripe size \(should return error\) ============================= 01:21:31 \(1305620491\)");echo XXRETCODE:$?' May 17 01:21:31 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 27f: setstripe with bad stripe size (should return error) ============================= 01:21:31 (1305620491) May 17 01:21:31 client-5 xinetd[3391]: EXIT: shell status=0 pid=31235 duration=0(sec) May 17 01:21:31 client-5 xinetd[3391]: START: shell pid=31328 from=192.168.4.5 May 17 01:21:31 client-5 rshd[31331]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:21:31 client-5 xinetd[3391]: EXIT: shell status=0 pid=31328 duration=0(sec) May 17 01:21:31 client-5 xinetd[3391]: START: shell pid=31502 from=192.168.4.5 May 17 01:21:31 client-5 rshd[31503]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 27g: test lfs getstripe =============================================================== 01:21:31 \(1305620491\)");echo XXRETCODE:$?' May 17 01:21:31 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 27g: test lfs getstripe =============================================================== 01:21:31 (1305620491) May 17 01:21:31 client-5 xinetd[3391]: EXIT: shell status=0 pid=31502 duration=0(sec) May 17 01:21:31 client-5 xinetd[3391]: START: shell pid=31599 from=192.168.4.5 May 17 01:21:31 client-5 rshd[31600]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == test 27h: lfs getstripe with no objects ====================");echo XXRETCODE:$?' May 17 01:21:31 client-5 kernel: Lustre: DEBUG MARKER: == test 27h: lfs getstripe with no objects ==================== May 17 01:21:31 client-5 xinetd[3391]: EXIT: shell status=0 pid=31599 duration=0(sec) May 17 01:21:31 client-5 xinetd[3391]: START: shell pid=31696 from=192.168.4.5 May 17 01:21:32 client-5 rshd[31697]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == test 27i: lfs getstripe with some objects ==================");echo XXRETCODE:$?' May 17 01:21:32 client-5 kernel: Lustre: DEBUG MARKER: == test 27i: lfs getstripe with some objects ================== May 17 01:21:32 client-5 xinetd[3391]: EXIT: shell status=0 pid=31696 duration=1(sec) May 17 01:21:32 client-5 xinetd[3391]: START: shell pid=31788 from=192.168.4.5 May 17 01:21:32 client-5 rshd[31791]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:21:32 client-5 xinetd[3391]: EXIT: shell status=0 pid=31788 duration=0(sec) May 17 01:21:32 client-5 xinetd[3391]: START: shell pid=31962 from=192.168.4.5 May 17 01:21:32 client-5 rshd[31963]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 27j: setstripe with bad stripe offset \(should return error\) =========================== 01:21:32 \(1305620492\)");echo XXRETCODE:$?' May 17 01:21:32 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 27j: setstripe with bad stripe offset (should return error) =========================== 01:21:32 (1305620492) May 17 01:21:32 client-5 xinetd[3391]: EXIT: shell status=0 pid=31962 duration=0(sec) May 17 01:21:32 client-5 xinetd[3391]: START: shell pid=32055 from=192.168.4.5 May 17 01:21:32 client-5 rshd[32056]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:21:32 client-5 xinetd[3391]: EXIT: shell status=0 pid=32055 duration=0(sec) May 17 01:21:32 client-5 xinetd[3391]: START: shell pid=32227 from=192.168.4.5 May 17 01:21:33 client-5 rshd[32228]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 27k: limit i_blksize for broken user apps ============================================= 01:21:32 \(1305620492\)");echo XXRETCODE:$?' May 17 01:21:33 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 27k: limit i_blksize for broken user apps ============================================= 01:21:32 (1305620492) May 17 01:21:33 client-5 xinetd[3391]: EXIT: shell status=0 pid=32227 duration=1(sec) May 17 01:21:33 client-5 xinetd[3391]: START: shell pid=32327 from=192.168.4.5 May 17 01:21:33 client-5 rshd[32328]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:21:33 client-5 xinetd[3391]: EXIT: shell status=0 pid=32327 duration=0(sec) May 17 01:21:33 client-5 xinetd[3391]: START: shell pid=32498 from=192.168.4.5 May 17 01:21:33 client-5 rshd[32500]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 27l: check setstripe permissions \(should return error\) ================================ 01:21:33 \(1305620493\)");echo XXRETCODE:$?' May 17 01:21:33 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 27l: check setstripe permissions (should return error) ================================ 01:21:33 (1305620493) May 17 01:21:33 client-5 xinetd[3391]: EXIT: shell status=0 pid=32498 duration=0(sec) May 17 01:21:33 client-5 xinetd[3391]: START: shell pid=32593 from=192.168.4.5 May 17 01:21:33 client-5 rshd[32594]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:21:33 client-5 xinetd[3391]: EXIT: shell status=0 pid=32593 duration=0(sec) May 17 01:21:34 client-5 xinetd[3391]: START: shell pid=32765 from=192.168.4.5 May 17 01:21:34 client-5 rshd[32766]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 27m: create file while OST0 was full ================================================== 01:21:33 \(1305620493\)");echo XXRETCODE:$?' May 17 01:21:34 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 27m: create file while OST0 was full ================================================== 01:21:33 (1305620493) May 17 01:21:34 client-5 xinetd[3391]: EXIT: shell status=0 pid=32765 duration=0(sec) May 17 01:21:34 client-5 xinetd[3391]: START: shell pid=392 from=192.168.4.5 May 17 01:21:34 client-5 rshd[393]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark SKIP: sanity test_27m 53539848 \> 1200000 skipping out-of-space test on OST0");echo XXRETCODE:$?' May 17 01:21:34 client-5 kernel: Lustre: DEBUG MARKER: SKIP: sanity test_27m 53539848 > 1200000 skipping out-of-space test on OST0 May 17 01:21:34 client-5 xinetd[3391]: EXIT: shell status=0 pid=392 duration=0(sec) May 17 01:21:34 client-5 xinetd[3391]: START: shell pid=491 from=192.168.4.5 May 17 01:21:34 client-5 rshd[492]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:21:34 client-5 xinetd[3391]: EXIT: shell status=0 pid=491 duration=0(sec) May 17 01:21:34 client-5 xinetd[3391]: START: shell pid=678 from=192.168.4.5 May 17 01:21:34 client-5 rshd[679]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 27n: create file with some full OSTs ================================================== 01:21:34 \(1305620494\)");echo XXRETCODE:$?' May 17 01:21:34 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 27n: create file with some full OSTs ================================================== 01:21:34 (1305620494) May 17 01:21:34 client-5 xinetd[3391]: EXIT: shell status=0 pid=678 duration=0(sec) May 17 01:21:54 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.129@o2ib. The ost_write operation failed with -28 May 17 01:22:09 client-5 xinetd[3391]: START: shell pid=1143 from=192.168.4.5 May 17 01:22:09 client-5 rshd[1144]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:22:09 client-5 xinetd[3391]: EXIT: shell status=0 pid=1143 duration=0(sec) May 17 01:22:09 client-5 xinetd[3391]: START: shell pid=1313 from=192.168.4.5 May 17 01:22:09 client-5 rshd[1316]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 27o: create file with all full OSTs \(should error\) ==================================== 01:22:09 \(1305620529\)");echo XXRETCODE:$?' May 17 01:22:09 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 27o: create file with all full OSTs (should error) ==================================== 01:22:09 (1305620529) May 17 01:22:09 client-5 xinetd[3391]: EXIT: shell status=0 pid=1313 duration=0(sec) May 17 01:23:54 client-5 xinetd[3391]: START: shell pid=3022 from=192.168.4.5 May 17 01:23:54 client-5 rshd[3024]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:23:54 client-5 xinetd[3391]: EXIT: shell status=0 pid=3022 duration=0(sec) May 17 01:23:54 client-5 xinetd[3391]: START: shell pid=3210 from=192.168.4.5 May 17 01:23:54 client-5 rshd[3211]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 27p: append to a truncated file with some full OSTs =================================== 01:23:54 \(1305620634\)");echo XXRETCODE:$?' May 17 01:23:55 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 27p: append to a truncated file with some full OSTs =================================== 01:23:54 (1305620634) May 17 01:23:55 client-5 xinetd[3391]: EXIT: shell status=0 pid=3210 duration=1(sec) May 17 01:24:29 client-5 xinetd[3391]: START: shell pid=3720 from=192.168.4.5 May 17 01:24:29 client-5 rshd[3722]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:24:29 client-5 xinetd[3391]: EXIT: shell status=0 pid=3720 duration=0(sec) May 17 01:24:29 client-5 xinetd[3391]: START: shell pid=3893 from=192.168.4.5 May 17 01:24:29 client-5 rshd[3894]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 27q: append to truncated file with all OSTs full \(should error\) ======================= 01:24:29 \(1305620669\)");echo XXRETCODE:$?' May 17 01:24:30 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 27q: append to truncated file with all OSTs full (should error) ======================= 01:24:29 (1305620669) May 17 01:24:30 client-5 xinetd[3391]: EXIT: shell status=0 pid=3893 duration=1(sec) May 17 01:24:49 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.129@o2ib. The ost_write operation failed with -28 May 17 01:24:57 client-5 ntpd[3402]: synchronized to 10.10.0.1, stratum 2 May 17 01:26:14 client-5 xinetd[3391]: START: shell pid=5598 from=192.168.4.5 May 17 01:26:14 client-5 rshd[5599]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:26:14 client-5 xinetd[3391]: EXIT: shell status=0 pid=5598 duration=0(sec) May 17 01:26:14 client-5 xinetd[3391]: START: shell pid=5770 from=192.168.4.5 May 17 01:26:15 client-5 rshd[5771]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 27r: stripe file with some full OSTs \(shouldn\'t LBUG\) ================================= 01:26:14 \(1305620774\)");echo XXRETCODE:$?' May 17 01:26:15 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 27r: stripe file with some full OSTs (shouldn't LBUG) ================================= 01:26:14 (1305620774) May 17 01:26:15 client-5 xinetd[3391]: EXIT: shell status=0 pid=5770 duration=1(sec) May 17 01:26:49 client-5 xinetd[3391]: START: shell pid=6230 from=192.168.4.5 May 17 01:26:49 client-5 rshd[6233]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:26:49 client-5 xinetd[3391]: EXIT: shell status=0 pid=6230 duration=0(sec) May 17 01:26:49 client-5 xinetd[3391]: START: shell pid=6405 from=192.168.4.5 May 17 01:26:49 client-5 rshd[6406]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 27s: lsm_xfersize overflow \(should error\) \(bug 10725\) ================================= 01:26:49 \(1305620809\)");echo XXRETCODE:$?' May 17 01:26:50 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 27s: lsm_xfersize overflow (should error) (bug 10725) ================================= 01:26:49 (1305620809) May 17 01:26:50 client-5 xinetd[3391]: EXIT: shell status=0 pid=6405 duration=1(sec) May 17 01:26:50 client-5 xinetd[3391]: START: shell pid=6497 from=192.168.4.5 May 17 01:26:50 client-5 rshd[6499]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:26:50 client-5 xinetd[3391]: EXIT: shell status=0 pid=6497 duration=0(sec) May 17 01:26:50 client-5 xinetd[3391]: START: shell pid=6670 from=192.168.4.5 May 17 01:26:50 client-5 rshd[6671]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 27t: check that utils parse path correctly ============================================ 01:26:50 \(1305620810\)");echo XXRETCODE:$?' May 17 01:26:50 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 27t: check that utils parse path correctly ============================================ 01:26:50 (1305620810) May 17 01:26:50 client-5 xinetd[3391]: EXIT: shell status=0 pid=6670 duration=0(sec) May 17 01:26:50 client-5 xinetd[3391]: START: shell pid=6763 from=192.168.4.5 May 17 01:26:50 client-5 rshd[6766]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:26:50 client-5 xinetd[3391]: EXIT: shell status=0 pid=6763 duration=0(sec) May 17 01:26:50 client-5 xinetd[3391]: START: shell pid=6927 from=192.168.4.5 May 17 01:26:50 client-5 rshd[6929]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark SKIP: sanity test_27u skipping ALWAYS excluded test 27u");echo XXRETCODE:$?' May 17 01:26:51 client-5 kernel: Lustre: DEBUG MARKER: SKIP: sanity test_27u skipping ALWAYS excluded test 27u May 17 01:26:51 client-5 xinetd[3391]: EXIT: shell status=0 pid=6927 duration=1(sec) May 17 01:26:51 client-5 xinetd[3391]: START: shell pid=7033 from=192.168.4.5 May 17 01:26:51 client-5 rshd[7034]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 27v: skip object creation on slow OST ================================================= 01:26:51 \(1305620811\)");echo XXRETCODE:$?' May 17 01:26:51 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 27v: skip object creation on slow OST ================================================= 01:26:51 (1305620811) May 17 01:26:51 client-5 xinetd[3391]: EXIT: shell status=0 pid=7033 duration=0(sec) May 17 01:28:36 client-5 xinetd[3391]: START: shell pid=8745 from=192.168.4.5 May 17 01:28:36 client-5 rshd[8748]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:28:36 client-5 xinetd[3391]: EXIT: shell status=0 pid=8745 duration=0(sec) May 17 01:28:36 client-5 xinetd[3391]: START: shell pid=8919 from=192.168.4.5 May 17 01:28:36 client-5 rshd[8920]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 27w: check lfs setstripe -c -s -i options ============================================= 01:28:36 \(1305620916\)");echo XXRETCODE:$?' May 17 01:28:36 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 27w: check lfs setstripe -c -s -i options ============================================= 01:28:36 (1305620916) May 17 01:28:36 client-5 xinetd[3391]: EXIT: shell status=0 pid=8919 duration=0(sec) May 17 01:28:36 client-5 xinetd[3391]: START: shell pid=9036 from=192.168.4.5 May 17 01:28:37 client-5 rshd[9037]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:28:37 client-5 xinetd[3391]: EXIT: shell status=0 pid=9036 duration=1(sec) May 17 01:28:37 client-5 xinetd[3391]: START: shell pid=9209 from=192.168.4.5 May 17 01:28:37 client-5 rshd[9210]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 27x: create files while OST0 is degraded ============================================== 01:28:37 \(1305620917\)");echo XXRETCODE:$?' May 17 01:28:37 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 27x: create files while OST0 is degraded ============================================== 01:28:37 (1305620917) May 17 01:28:37 client-5 xinetd[3391]: EXIT: shell status=0 pid=9209 duration=0(sec) May 17 01:28:47 client-5 xinetd[3391]: START: shell pid=9367 from=192.168.4.5 May 17 01:28:47 client-5 rshd[9368]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:28:47 client-5 xinetd[3391]: EXIT: shell status=0 pid=9367 duration=0(sec) May 17 01:28:48 client-5 xinetd[3391]: START: shell pid=9538 from=192.168.4.5 May 17 01:28:48 client-5 rshd[9540]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 27y: create files while OST0 is degraded and the rest inactive ======================== 01:28:48 \(1305620928\)");echo XXRETCODE:$?' May 17 01:28:48 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 27y: create files while OST0 is degraded and the rest inactive ======================== 01:28:48 (1305620928) May 17 01:28:48 client-5 xinetd[3391]: EXIT: shell status=0 pid=9538 duration=0(sec) May 17 01:30:00 client-5 xinetd[3391]: START: shell pid=9937 from=192.168.4.5 May 17 01:30:00 client-5 rshd[9938]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:30:00 client-5 xinetd[3391]: EXIT: shell status=0 pid=9937 duration=0(sec) May 17 01:30:00 client-5 xinetd[3391]: START: shell pid=10107 from=192.168.4.5 May 17 01:30:00 client-5 rshd[10110]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 27z: check SEQ\/OID on the MDT and OST filesystems ===================================== 01:30:00 \(1305621000\)");echo XXRETCODE:$?' May 17 01:30:00 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 27z: check SEQ/OID on the MDT and OST filesystems ===================================== 01:30:00 (1305621000) May 17 01:30:00 client-5 xinetd[3391]: EXIT: shell status=0 pid=10107 duration=0(sec) May 17 01:30:17 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.129@o2ib. The obd_ping operation failed with -107 May 17 01:30:17 client-5 kernel: Lustre: lustre-OST0000-osc-ffff81033c45d000: Connection to service lustre-OST0000 via nid 192.168.4.129@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 01:30:17 client-5 kernel: Lustre: lustre-OST0005-osc-ffff81033c45d000: Connection to service lustre-OST0005 via nid 192.168.4.129@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 01:30:19 client-5 kernel: Lustre: lustre-OST0005-osc-ffff81033c45d000: Connection restored to service lustre-OST0005 using nid 192.168.4.129@o2ib. May 17 01:30:19 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.129@o2ib. The ldlm_enqueue operation failed with -19 May 17 01:30:19 client-5 kernel: LustreError: Skipped 3 previous similar messages May 17 01:30:19 client-5 kernel: LustreError: 4124:0:(import.c:617:ptlrpc_connect_import()) already connecting May 17 01:30:19 client-5 kernel: LustreError: 4124:0:(import.c:617:ptlrpc_connect_import()) already connecting May 17 01:30:22 client-5 kernel: Lustre: lustre-OST0002-osc-ffff81033c45d000: Connection to service lustre-OST0002 via nid 192.168.4.129@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 01:30:22 client-5 kernel: Lustre: Skipped 1 previous similar message May 17 01:30:27 client-5 kernel: Lustre: 4125:0:(import.c:529:import_select_connection()) lustre-OST0000-osc-ffff81033c45d000: tried all connections, increasing latency to 6s May 17 01:30:27 client-5 kernel: Lustre: 4125:0:(import.c:529:import_select_connection()) lustre-OST0001-osc-ffff81033c45d000: tried all connections, increasing latency to 6s May 17 01:30:28 client-5 kernel: Lustre: lustre-OST0000-osc-ffff81033c45d000: Connection restored to service lustre-OST0000 using nid 192.168.4.129@o2ib. May 17 01:30:32 client-5 kernel: Lustre: 4125:0:(import.c:529:import_select_connection()) lustre-OST0002-osc-ffff81033c45d000: tried all connections, increasing latency to 6s May 17 01:30:34 client-5 kernel: Lustre: lustre-OST0002-osc-ffff81033c45d000: Connection restored to service lustre-OST0002 using nid 192.168.4.129@o2ib. May 17 01:30:34 client-5 xinetd[3391]: START: shell pid=10924 from=192.168.4.5 May 17 01:30:34 client-5 rshd[10925]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:30:34 client-5 xinetd[3391]: EXIT: shell status=0 pid=10924 duration=0(sec) May 17 01:30:34 client-5 xinetd[3391]: START: shell pid=11096 from=192.168.4.5 May 17 01:30:34 client-5 rshd[11097]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 27A: check filesystem-wide default LOV EA values ====================================== 01:30:34 \(1305621034\)");echo XXRETCODE:$?' May 17 01:30:34 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 27A: check filesystem-wide default LOV EA values ====================================== 01:30:34 (1305621034) May 17 01:30:34 client-5 xinetd[3391]: EXIT: shell status=0 pid=11096 duration=0(sec) May 17 01:30:34 client-5 xinetd[3391]: START: shell pid=11196 from=192.168.4.5 May 17 01:30:34 client-5 rshd[11197]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:30:34 client-5 xinetd[3391]: EXIT: shell status=0 pid=11196 duration=0(sec) May 17 01:30:35 client-5 xinetd[3391]: START: shell pid=11368 from=192.168.4.5 May 17 01:30:35 client-5 rshd[11369]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 28: create\/mknod\/mkdir with bad file types ============================================ 01:30:35 \(1305621035\)");echo XXRETCODE:$?' May 17 01:30:35 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 28: create/mknod/mkdir with bad file types ============================================ 01:30:35 (1305621035) May 17 01:30:35 client-5 xinetd[3391]: EXIT: shell status=0 pid=11368 duration=0(sec) May 17 01:30:35 client-5 xinetd[3391]: START: shell pid=11461 from=192.168.4.5 May 17 01:30:35 client-5 rshd[11462]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:30:35 client-5 xinetd[3391]: EXIT: shell status=0 pid=11461 duration=0(sec) May 17 01:30:35 client-5 xinetd[3391]: START: shell pid=11631 from=192.168.4.5 May 17 01:30:35 client-5 rshd[11634]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 29: IT_GETATTR regression ============================================================ 01:30:35 \(1305621035\)");echo XXRETCODE:$?' May 17 01:30:35 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 29: IT_GETATTR regression ============================================================ 01:30:35 (1305621035) May 17 01:30:35 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc start May 17 01:30:35 client-5 xinetd[3391]: EXIT: shell status=0 pid=11631 duration=0(sec) May 17 01:30:36 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc stop May 17 01:30:36 client-5 xinetd[3391]: START: shell pid=11737 from=192.168.4.5 May 17 01:30:36 client-5 rshd[11738]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark first d29");echo XXRETCODE:$?' May 17 01:30:36 client-5 kernel: Lustre: DEBUG MARKER: first d29 May 17 01:30:36 client-5 xinetd[3391]: EXIT: shell status=0 pid=11737 duration=0(sec) May 17 01:30:36 client-5 xinetd[3391]: START: shell pid=11834 from=192.168.4.5 May 17 01:30:36 client-5 rshd[11835]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark second d29");echo XXRETCODE:$?' May 17 01:30:36 client-5 kernel: Lustre: DEBUG MARKER: second d29 May 17 01:30:36 client-5 xinetd[3391]: EXIT: shell status=0 pid=11834 duration=0(sec) May 17 01:30:36 client-5 xinetd[3391]: START: shell pid=11928 from=192.168.4.5 May 17 01:30:36 client-5 rshd[11930]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark done");echo XXRETCODE:$?' May 17 01:30:36 client-5 kernel: Lustre: DEBUG MARKER: done May 17 01:30:36 client-5 xinetd[3391]: EXIT: shell status=0 pid=11928 duration=0(sec) May 17 01:30:36 client-5 xinetd[3391]: START: shell pid=12022 from=192.168.4.5 May 17 01:30:36 client-5 rshd[12023]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:30:36 client-5 xinetd[3391]: EXIT: shell status=0 pid=12022 duration=0(sec) May 17 01:30:36 client-5 xinetd[3391]: START: shell pid=12194 from=192.168.4.5 May 17 01:30:36 client-5 rshd[12195]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 30a: execute binary from Lustre \(execve\) ============================================== 01:30:36 \(1305621036\)");echo XXRETCODE:$?' May 17 01:30:37 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 30a: execute binary from Lustre (execve) ============================================== 01:30:36 (1305621036) May 17 01:30:37 client-5 xinetd[3391]: EXIT: shell status=0 pid=12194 duration=1(sec) May 17 01:30:37 client-5 xinetd[3391]: START: shell pid=12289 from=192.168.4.5 May 17 01:30:37 client-5 rshd[12290]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:30:37 client-5 xinetd[3391]: EXIT: shell status=0 pid=12289 duration=0(sec) May 17 01:30:37 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.129@o2ib. The obd_ping operation failed with -107 May 17 01:30:37 client-5 kernel: LustreError: Skipped 9 previous similar messages May 17 01:30:37 client-5 kernel: Lustre: lustre-OST0003-osc-ffff81033c45d000: Connection to service lustre-OST0003 via nid 192.168.4.129@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 01:30:37 client-5 xinetd[3391]: START: shell pid=12460 from=192.168.4.5 May 17 01:30:37 client-5 rshd[12462]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 30b: execute binary from Lustre as non-root =========================================== 01:30:37 \(1305621037\)");echo XXRETCODE:$?' May 17 01:30:37 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 30b: execute binary from Lustre as non-root =========================================== 01:30:37 (1305621037) May 17 01:30:37 client-5 xinetd[3391]: EXIT: shell status=0 pid=12460 duration=0(sec) May 17 01:30:37 client-5 xinetd[3391]: START: shell pid=12555 from=192.168.4.5 May 17 01:30:37 client-5 rshd[12558]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:30:37 client-5 xinetd[3391]: EXIT: shell status=0 pid=12555 duration=0(sec) May 17 01:30:38 client-5 xinetd[3391]: START: shell pid=12729 from=192.168.4.5 May 17 01:30:38 client-5 rshd[12730]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 30c: execute binary from Lustre without read perms ==================================== 01:30:37 \(1305621037\)");echo XXRETCODE:$?' May 17 01:30:38 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 30c: execute binary from Lustre without read perms ==================================== 01:30:37 (1305621037) May 17 01:30:38 client-5 xinetd[3391]: EXIT: shell status=0 pid=12729 duration=0(sec) May 17 01:30:38 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc start May 17 01:30:38 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc stop May 17 01:30:38 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 01:30:38 client-5 kernel: LustreError: 27338:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -11 from cancel RPC: canceling anyway May 17 01:30:38 client-5 kernel: LustreError: 27338:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -11 May 17 01:30:38 client-5 kernel: LustreError: 5172:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -11 from cancel RPC: canceling anyway May 17 01:30:38 client-5 kernel: LustreError: 5172:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -11 May 17 01:30:38 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 01:30:38 client-5 xinetd[3391]: START: shell pid=12848 from=192.168.4.5 May 17 01:30:38 client-5 rshd[12849]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:30:38 client-5 xinetd[3391]: EXIT: shell status=0 pid=12848 duration=0(sec) May 17 01:30:39 client-5 xinetd[3391]: START: shell pid=13020 from=192.168.4.5 May 17 01:30:39 client-5 rshd[13021]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 31a: open-unlink file ================================================================= 01:30:39 \(1305621039\)");echo XXRETCODE:$?' May 17 01:30:39 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 31a: open-unlink file ================================================================= 01:30:39 (1305621039) May 17 01:30:39 client-5 xinetd[3391]: EXIT: shell status=0 pid=13020 duration=0(sec) May 17 01:30:39 client-5 kernel: LustreError: 4124:0:(ldlm_request.c:2039:replay_lock_interpret()) received replay ack for unknown local cookie 0xa3083b56b6f1ce91 remote cookie 0x31e2b4d587da9899 from server id 0-0@<0:0> May 17 01:30:39 client-5 kernel: LustreError: 4124:0:(ldlm_request.c:2039:replay_lock_interpret()) received replay ack for unknown local cookie 0xa3083b56b6f1c90b remote cookie 0x31e2b4d587da98a0 from server id 0-0@<0:0> May 17 01:30:39 client-5 kernel: LustreError: 4124:0:(import.c:617:ptlrpc_connect_import()) already connecting May 17 01:30:39 client-5 xinetd[3391]: START: shell pid=13112 from=192.168.4.5 May 17 01:30:39 client-5 kernel: LustreError: 4124:0:(import.c:617:ptlrpc_connect_import()) Skipped 2 previous similar messages May 17 01:30:39 client-5 kernel: Lustre: lustre-OST0003-osc-ffff81033c45d000: Connection restored to service lustre-OST0003 using nid 192.168.4.129@o2ib. May 17 01:30:39 client-5 rshd[13114]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:30:39 client-5 xinetd[3391]: EXIT: shell status=0 pid=13112 duration=0(sec) May 17 01:30:39 client-5 xinetd[3391]: START: shell pid=13286 from=192.168.4.5 May 17 01:30:39 client-5 rshd[13288]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 31b: unlink file with multiple links while open ======================================= 01:30:39 \(1305621039\)");echo XXRETCODE:$?' May 17 01:30:39 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 31b: unlink file with multiple links while open ======================================= 01:30:39 (1305621039) May 17 01:30:39 client-5 xinetd[3391]: EXIT: shell status=0 pid=13286 duration=0(sec) May 17 01:30:39 client-5 xinetd[3391]: START: shell pid=13381 from=192.168.4.5 May 17 01:30:39 client-5 rshd[13383]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:30:39 client-5 xinetd[3391]: EXIT: shell status=0 pid=13381 duration=0(sec) May 17 01:30:40 client-5 xinetd[3391]: START: shell pid=13554 from=192.168.4.5 May 17 01:30:40 client-5 rshd[13555]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 31c: open-unlink file with multiple links ============================================= 01:30:40 \(1305621040\)");echo XXRETCODE:$?' May 17 01:30:40 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 31c: open-unlink file with multiple links ============================================= 01:30:40 (1305621040) May 17 01:30:40 client-5 xinetd[3391]: EXIT: shell status=0 pid=13554 duration=0(sec) May 17 01:30:40 client-5 xinetd[3391]: START: shell pid=13651 from=192.168.4.5 May 17 01:30:40 client-5 rshd[13652]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:30:40 client-5 xinetd[3391]: EXIT: shell status=0 pid=13651 duration=0(sec) May 17 01:30:40 client-5 xinetd[3391]: START: shell pid=13823 from=192.168.4.5 May 17 01:30:40 client-5 rshd[13824]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 31d: remove of open directory ========================================================= 01:30:40 \(1305621040\)");echo XXRETCODE:$?' May 17 01:30:40 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 31d: remove of open directory ========================================================= 01:30:40 (1305621040) May 17 01:30:40 client-5 xinetd[3391]: EXIT: shell status=0 pid=13823 duration=0(sec) May 17 01:30:40 client-5 xinetd[3391]: START: shell pid=13915 from=192.168.4.5 May 17 01:30:40 client-5 rshd[13917]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:30:41 client-5 xinetd[3391]: EXIT: shell status=0 pid=13915 duration=1(sec) May 17 01:30:41 client-5 xinetd[3391]: START: shell pid=14088 from=192.168.4.5 May 17 01:30:41 client-5 rshd[14089]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 31e: remove of open non-empty directory =============================================== 01:30:41 \(1305621041\)");echo XXRETCODE:$?' May 17 01:30:41 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 31e: remove of open non-empty directory =============================================== 01:30:41 (1305621041) May 17 01:30:41 client-5 xinetd[3391]: EXIT: shell status=0 pid=14088 duration=0(sec) May 17 01:30:41 client-5 xinetd[3391]: START: shell pid=14183 from=192.168.4.5 May 17 01:30:41 client-5 rshd[14184]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:30:41 client-5 xinetd[3391]: EXIT: shell status=0 pid=14183 duration=0(sec) May 17 01:30:41 client-5 xinetd[3391]: START: shell pid=14354 from=192.168.4.5 May 17 01:30:41 client-5 rshd[14356]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 31f: remove of open directory with open-unlink file =================================== 01:30:41 \(1305621041\)");echo XXRETCODE:$?' May 17 01:30:41 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 31f: remove of open directory with open-unlink file =================================== 01:30:41 (1305621041) May 17 01:30:41 client-5 xinetd[3391]: EXIT: shell status=0 pid=14354 duration=0(sec) May 17 01:30:43 client-5 kernel: LustreError: 4124:0:(ldlm_request.c:2039:replay_lock_interpret()) received replay ack for unknown local cookie 0xa3083b56b6f1ce98 remote cookie 0x31e2b4d587da99fe from server id 0-0@<0:0> May 17 01:30:43 client-5 kernel: LustreError: 4124:0:(ldlm_request.c:2039:replay_lock_interpret()) Skipped 2 previous similar messages May 17 01:30:43 client-5 kernel: LustreError: 4124:0:(import.c:617:ptlrpc_connect_import()) already connecting May 17 01:30:43 client-5 kernel: LustreError: 4124:0:(import.c:617:ptlrpc_connect_import()) Skipped 2 previous similar messages May 17 01:30:43 client-5 kernel: Lustre: lustre-OST0004-osc-ffff81033c45d000: Connection restored to service lustre-OST0004 using nid 192.168.4.129@o2ib. May 17 01:30:45 client-5 kernel: LustreError: 4124:0:(ldlm_request.c:2039:replay_lock_interpret()) received replay ack for unknown local cookie 0xa3083b56b6f1c8b7 remote cookie 0x31e2b4d587da9b01 from server id 0-0@<0:0> May 17 01:30:45 client-5 kernel: LustreError: 4124:0:(ldlm_request.c:2039:replay_lock_interpret()) Skipped 3 previous similar messages May 17 01:30:48 client-5 xinetd[3391]: START: shell pid=14465 from=192.168.4.5 May 17 01:30:48 client-5 rshd[14467]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:30:48 client-5 xinetd[3391]: EXIT: shell status=0 pid=14465 duration=0(sec) May 17 01:30:48 client-5 xinetd[3391]: START: shell pid=14637 from=192.168.4.5 May 17 01:30:48 client-5 rshd[14639]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 31g: cross directory link============================================================== 01:30:48 \(1305621048\)");echo XXRETCODE:$?' May 17 01:30:48 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 31g: cross directory link============================================================== 01:30:48 (1305621048) May 17 01:30:48 client-5 xinetd[3391]: EXIT: shell status=0 pid=14637 duration=0(sec) May 17 01:30:48 client-5 xinetd[3391]: START: shell pid=14735 from=192.168.4.5 May 17 01:30:48 client-5 rshd[14737]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:30:48 client-5 xinetd[3391]: EXIT: shell status=0 pid=14735 duration=0(sec) May 17 01:30:48 client-5 xinetd[3391]: START: shell pid=14908 from=192.168.4.5 May 17 01:30:48 client-5 rshd[14909]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 31h: cross directory link under child================================================== 01:30:48 \(1305621048\)");echo XXRETCODE:$?' May 17 01:30:49 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 31h: cross directory link under child================================================== 01:30:48 (1305621048) May 17 01:30:49 client-5 xinetd[3391]: EXIT: shell status=0 pid=14908 duration=1(sec) May 17 01:30:49 client-5 xinetd[3391]: START: shell pid=15007 from=192.168.4.5 May 17 01:30:49 client-5 rshd[15008]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:30:49 client-5 xinetd[3391]: EXIT: shell status=0 pid=15007 duration=0(sec) May 17 01:30:49 client-5 xinetd[3391]: START: shell pid=15179 from=192.168.4.5 May 17 01:30:49 client-5 rshd[15180]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 31i: cross directory link under parent================================================= 01:30:49 \(1305621049\)");echo XXRETCODE:$?' May 17 01:30:49 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 31i: cross directory link under parent================================================= 01:30:49 (1305621049) May 17 01:30:49 client-5 xinetd[3391]: EXIT: shell status=0 pid=15179 duration=0(sec) May 17 01:30:49 client-5 xinetd[3391]: START: shell pid=15276 from=192.168.4.5 May 17 01:30:49 client-5 rshd[15279]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:30:49 client-5 xinetd[3391]: EXIT: shell status=0 pid=15276 duration=0(sec) May 17 01:30:49 client-5 xinetd[3391]: START: shell pid=15450 from=192.168.4.5 May 17 01:30:50 client-5 rshd[15451]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 31j: link for directory================================================================ 01:30:49 \(1305621049\)");echo XXRETCODE:$?' May 17 01:30:50 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 31j: link for directory================================================================ 01:30:49 (1305621049) May 17 01:30:50 client-5 xinetd[3391]: EXIT: shell status=0 pid=15450 duration=1(sec) May 17 01:30:50 client-5 xinetd[3391]: START: shell pid=15547 from=192.168.4.5 May 17 01:30:50 client-5 rshd[15548]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:30:50 client-5 xinetd[3391]: EXIT: shell status=0 pid=15547 duration=0(sec) May 17 01:30:50 client-5 xinetd[3391]: START: shell pid=15719 from=192.168.4.5 May 17 01:30:50 client-5 rshd[15720]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 31k: link to file: the same, non-existing, dir========================================= 01:30:50 \(1305621050\)");echo XXRETCODE:$?' May 17 01:30:50 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 31k: link to file: the same, non-existing, dir========================================= 01:30:50 (1305621050) May 17 01:30:50 client-5 xinetd[3391]: EXIT: shell status=0 pid=15719 duration=0(sec) May 17 01:30:50 client-5 xinetd[3391]: START: shell pid=15820 from=192.168.4.5 May 17 01:30:50 client-5 rshd[15821]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:30:50 client-5 xinetd[3391]: EXIT: shell status=0 pid=15820 duration=0(sec) May 17 01:30:51 client-5 xinetd[3391]: START: shell pid=15990 from=192.168.4.5 May 17 01:30:51 client-5 rshd[15993]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 31m: link to file: the same, non-existing, dir========================================= 01:30:51 \(1305621051\)");echo XXRETCODE:$?' May 17 01:30:51 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 31m: link to file: the same, non-existing, dir========================================= 01:30:51 (1305621051) May 17 01:30:51 client-5 xinetd[3391]: EXIT: shell status=0 pid=15990 duration=0(sec) May 17 01:30:51 client-5 xinetd[3391]: START: shell pid=16093 from=192.168.4.5 May 17 01:30:51 client-5 rshd[16094]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:30:51 client-5 xinetd[3391]: EXIT: shell status=0 pid=16093 duration=0(sec) May 17 01:30:51 client-5 xinetd[3391]: START: shell pid=16265 from=192.168.4.5 May 17 01:30:51 client-5 rshd[16266]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 32a: stat d32a\/ext2-mountpoint\/.. ===================================================== 01:30:51 \(1305621051\)");echo XXRETCODE:$?' May 17 01:30:51 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 32a: stat d32a/ext2-mountpoint/.. ===================================================== 01:30:51 (1305621051) May 17 01:30:51 client-5 xinetd[3391]: EXIT: shell status=0 pid=16265 duration=0(sec) May 17 01:30:51 client-5 kernel: EXT2-fs warning (device loop0): ext2_fill_super: mounting ext3 filesystem as ext2 May 17 01:30:51 client-5 xinetd[3391]: START: shell pid=16363 from=192.168.4.5 May 17 01:30:51 client-5 rshd[16364]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:30:51 client-5 xinetd[3391]: EXIT: shell status=0 pid=16363 duration=0(sec) May 17 01:30:52 client-5 xinetd[3391]: START: shell pid=16535 from=192.168.4.5 May 17 01:30:52 client-5 rshd[16536]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 32b: open d32b\/ext2-mountpoint\/.. ===================================================== 01:30:52 \(1305621052\)");echo XXRETCODE:$?' May 17 01:30:52 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 32b: open d32b/ext2-mountpoint/.. ===================================================== 01:30:52 (1305621052) May 17 01:30:52 client-5 xinetd[3391]: EXIT: shell status=0 pid=16535 duration=0(sec) May 17 01:30:52 client-5 kernel: EXT2-fs warning (device loop0): ext2_fill_super: mounting ext3 filesystem as ext2 May 17 01:30:52 client-5 xinetd[3391]: START: shell pid=16634 from=192.168.4.5 May 17 01:30:52 client-5 rshd[16635]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:30:52 client-5 xinetd[3391]: EXIT: shell status=0 pid=16634 duration=0(sec) May 17 01:30:52 client-5 xinetd[3391]: START: shell pid=16806 from=192.168.4.5 May 17 01:30:52 client-5 rshd[16807]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 32c: stat d32c\/ext2-mountpoint\/..\/d2\/test_dir ========================================= 01:30:52 \(1305621052\)");echo XXRETCODE:$?' May 17 01:30:52 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 32c: stat d32c/ext2-mountpoint/../d2/test_dir ========================================= 01:30:52 (1305621052) May 17 01:30:52 client-5 xinetd[3391]: EXIT: shell status=0 pid=16806 duration=0(sec) May 17 01:30:52 client-5 kernel: EXT2-fs warning (device loop0): ext2_fill_super: mounting ext3 filesystem as ext2 May 17 01:30:52 client-5 xinetd[3391]: START: shell pid=16905 from=192.168.4.5 May 17 01:30:52 client-5 rshd[16906]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:30:53 client-5 xinetd[3391]: EXIT: shell status=0 pid=16905 duration=1(sec) May 17 01:30:53 client-5 xinetd[3391]: START: shell pid=17077 from=192.168.4.5 May 17 01:30:53 client-5 rshd[17078]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 32d: open d32d\/ext2-mountpoint\/..\/d2\/test_dir ========================================= 01:30:53 \(1305621053\)");echo XXRETCODE:$?' May 17 01:30:53 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 32d: open d32d/ext2-mountpoint/../d2/test_dir ========================================= 01:30:53 (1305621053) May 17 01:30:53 client-5 xinetd[3391]: EXIT: shell status=0 pid=17077 duration=0(sec) May 17 01:30:53 client-5 kernel: EXT2-fs warning (device loop0): ext2_fill_super: mounting ext3 filesystem as ext2 May 17 01:30:53 client-5 xinetd[3391]: START: shell pid=17176 from=192.168.4.5 May 17 01:30:53 client-5 rshd[17177]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:30:53 client-5 xinetd[3391]: EXIT: shell status=0 pid=17176 duration=0(sec) May 17 01:30:53 client-5 xinetd[3391]: START: shell pid=17348 from=192.168.4.5 May 17 01:30:53 client-5 rshd[17349]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 32e: stat d32e\/symlink-\>tmp\/symlink-\>lustre-subdir ==================================== 01:30:53 \(1305621053\)");echo XXRETCODE:$?' May 17 01:30:53 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 32e: stat d32e/symlink->tmp/symlink->lustre-subdir ==================================== 01:30:53 (1305621053) May 17 01:30:53 client-5 xinetd[3391]: EXIT: shell status=0 pid=17348 duration=0(sec) May 17 01:30:54 client-5 xinetd[3391]: START: shell pid=17442 from=192.168.4.5 May 17 01:30:54 client-5 rshd[17445]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:30:54 client-5 xinetd[3391]: EXIT: shell status=0 pid=17442 duration=0(sec) May 17 01:30:54 client-5 xinetd[3391]: START: shell pid=17614 from=192.168.4.5 May 17 01:30:54 client-5 rshd[17617]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 32f: open d32f\/symlink-\>tmp\/symlink-\>lustre-subdir ==================================== 01:30:54 \(1305621054\)");echo XXRETCODE:$?' May 17 01:30:54 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 32f: open d32f/symlink->tmp/symlink->lustre-subdir ==================================== 01:30:54 (1305621054) May 17 01:30:54 client-5 xinetd[3391]: EXIT: shell status=0 pid=17614 duration=0(sec) May 17 01:30:54 client-5 xinetd[3391]: START: shell pid=17712 from=192.168.4.5 May 17 01:30:54 client-5 rshd[17713]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:30:54 client-5 xinetd[3391]: EXIT: shell status=0 pid=17712 duration=0(sec) May 17 01:30:54 client-5 xinetd[3391]: START: shell pid=17884 from=192.168.4.5 May 17 01:30:54 client-5 rshd[17885]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 32g: stat d32g\/symlink-\>tmp\/symlink-\>lustre-subdir\/2 ================================== 01:30:54 \(1305621054\)");echo XXRETCODE:$?' May 17 01:30:55 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 32g: stat d32g/symlink->tmp/symlink->lustre-subdir/2 ================================== 01:30:54 (1305621054) May 17 01:30:55 client-5 xinetd[3391]: EXIT: shell status=0 pid=17884 duration=1(sec) May 17 01:30:55 client-5 xinetd[3391]: START: shell pid=17982 from=192.168.4.5 May 17 01:30:55 client-5 rshd[17983]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:30:55 client-5 xinetd[3391]: EXIT: shell status=0 pid=17982 duration=0(sec) May 17 01:30:55 client-5 xinetd[3391]: START: shell pid=18154 from=192.168.4.5 May 17 01:30:55 client-5 rshd[18155]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 32h: open d32h\/symlink-\>tmp\/symlink-\>lustre-subdir\/2 ================================== 01:30:55 \(1305621055\)");echo XXRETCODE:$?' May 17 01:30:55 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 32h: open d32h/symlink->tmp/symlink->lustre-subdir/2 ================================== 01:30:55 (1305621055) May 17 01:30:55 client-5 xinetd[3391]: EXIT: shell status=0 pid=18154 duration=0(sec) May 17 01:30:55 client-5 xinetd[3391]: START: shell pid=18252 from=192.168.4.5 May 17 01:30:55 client-5 rshd[18253]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:30:55 client-5 xinetd[3391]: EXIT: shell status=0 pid=18252 duration=0(sec) May 17 01:30:56 client-5 xinetd[3391]: START: shell pid=18423 from=192.168.4.5 May 17 01:30:56 client-5 rshd[18425]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 32i: stat d32i\/ext2-mountpoint\/..\/test_file =========================================== 01:30:55 \(1305621055\)");echo XXRETCODE:$?' May 17 01:30:56 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 32i: stat d32i/ext2-mountpoint/../test_file =========================================== 01:30:55 (1305621055) May 17 01:30:56 client-5 xinetd[3391]: EXIT: shell status=0 pid=18423 duration=0(sec) May 17 01:30:56 client-5 kernel: EXT2-fs warning (device loop0): ext2_fill_super: mounting ext3 filesystem as ext2 May 17 01:30:56 client-5 xinetd[3391]: START: shell pid=18523 from=192.168.4.5 May 17 01:30:56 client-5 rshd[18524]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:30:56 client-5 xinetd[3391]: EXIT: shell status=0 pid=18523 duration=0(sec) May 17 01:30:56 client-5 xinetd[3391]: START: shell pid=18695 from=192.168.4.5 May 17 01:30:56 client-5 rshd[18696]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 32j: open d32j\/ext2-mountpoint\/..\/test_file =========================================== 01:30:56 \(1305621056\)");echo XXRETCODE:$?' May 17 01:30:56 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 32j: open d32j/ext2-mountpoint/../test_file =========================================== 01:30:56 (1305621056) May 17 01:30:56 client-5 xinetd[3391]: EXIT: shell status=0 pid=18695 duration=0(sec) May 17 01:30:56 client-5 kernel: EXT2-fs warning (device loop0): ext2_fill_super: mounting ext3 filesystem as ext2 May 17 01:30:56 client-5 xinetd[3391]: START: shell pid=18792 from=192.168.4.5 May 17 01:30:56 client-5 rshd[18795]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:30:56 client-5 xinetd[3391]: EXIT: shell status=0 pid=18792 duration=0(sec) May 17 01:30:57 client-5 xinetd[3391]: START: shell pid=18966 from=192.168.4.5 May 17 01:30:57 client-5 rshd[18967]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 32k: stat d32k\/ext2-mountpoint\/..\/d2\/test_file ======================================== 01:30:57 \(1305621057\)");echo XXRETCODE:$?' May 17 01:30:57 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 32k: stat d32k/ext2-mountpoint/../d2/test_file ======================================== 01:30:57 (1305621057) May 17 01:30:57 client-5 xinetd[3391]: EXIT: shell status=0 pid=18966 duration=0(sec) May 17 01:30:57 client-5 kernel: EXT2-fs warning (device loop0): ext2_fill_super: mounting ext3 filesystem as ext2 May 17 01:30:57 client-5 xinetd[3391]: START: shell pid=19067 from=192.168.4.5 May 17 01:30:57 client-5 rshd[19068]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:30:57 client-5 xinetd[3391]: EXIT: shell status=0 pid=19067 duration=0(sec) May 17 01:30:57 client-5 xinetd[3391]: START: shell pid=19239 from=192.168.4.5 May 17 01:30:57 client-5 rshd[19240]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 32l: open d32l\/ext2-mountpoint\/..\/d2\/test_file ======================================== 01:30:57 \(1305621057\)");echo XXRETCODE:$?' May 17 01:30:57 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 32l: open d32l/ext2-mountpoint/../d2/test_file ======================================== 01:30:57 (1305621057) May 17 01:30:57 client-5 xinetd[3391]: EXIT: shell status=0 pid=19239 duration=0(sec) May 17 01:30:57 client-5 kernel: EXT2-fs warning (device loop0): ext2_fill_super: mounting ext3 filesystem as ext2 May 17 01:30:57 client-5 xinetd[3391]: START: shell pid=19339 from=192.168.4.5 May 17 01:30:57 client-5 rshd[19341]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:30:58 client-5 xinetd[3391]: EXIT: shell status=0 pid=19339 duration=1(sec) May 17 01:30:58 client-5 xinetd[3391]: START: shell pid=19512 from=192.168.4.5 May 17 01:30:58 client-5 rshd[19513]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 32m: stat d32m\/symlink-\>tmp\/symlink-\>lustre-root ====================================== 01:30:58 \(1305621058\)");echo XXRETCODE:$?' May 17 01:30:58 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 32m: stat d32m/symlink->tmp/symlink->lustre-root ====================================== 01:30:58 (1305621058) May 17 01:30:58 client-5 xinetd[3391]: EXIT: shell status=0 pid=19512 duration=0(sec) May 17 01:30:58 client-5 xinetd[3391]: START: shell pid=19609 from=192.168.4.5 May 17 01:30:58 client-5 rshd[19610]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:30:58 client-5 xinetd[3391]: EXIT: shell status=0 pid=19609 duration=0(sec) May 17 01:30:58 client-5 xinetd[3391]: START: shell pid=19781 from=192.168.4.5 May 17 01:30:58 client-5 rshd[19782]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 32n: open d32n\/symlink-\>tmp\/symlink-\>lustre-root ====================================== 01:30:58 \(1305621058\)");echo XXRETCODE:$?' May 17 01:30:58 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 32n: open d32n/symlink->tmp/symlink->lustre-root ====================================== 01:30:58 (1305621058) May 17 01:30:58 client-5 xinetd[3391]: EXIT: shell status=0 pid=19781 duration=0(sec) May 17 01:30:58 client-5 xinetd[3391]: START: shell pid=19876 from=192.168.4.5 May 17 01:30:59 client-5 rshd[19879]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:30:59 client-5 xinetd[3391]: EXIT: shell status=0 pid=19876 duration=1(sec) May 17 01:30:59 client-5 xinetd[3391]: START: shell pid=20050 from=192.168.4.5 May 17 01:30:59 client-5 rshd[20051]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 32o: stat d32o\/symlink-\>tmp\/symlink-\>lustre-root\/ ===================================== 01:30:59 \(1305621059\)");echo XXRETCODE:$?' May 17 01:30:59 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 32o: stat d32o/symlink->tmp/symlink->lustre-root/ ===================================== 01:30:59 (1305621059) May 17 01:30:59 client-5 xinetd[3391]: EXIT: shell status=0 pid=20050 duration=0(sec) May 17 01:30:59 client-5 xinetd[3391]: START: shell pid=20148 from=192.168.4.5 May 17 01:30:59 client-5 rshd[20151]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:30:59 client-5 xinetd[3391]: EXIT: shell status=0 pid=20148 duration=0(sec) May 17 01:30:59 client-5 xinetd[3391]: START: shell pid=20326 from=192.168.4.5 May 17 01:30:59 client-5 rshd[20327]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 32p: open d32p\/symlink-\>tmp\/symlink-\>lustre-root\/ ===================================== 01:30:59 \(1305621059\)");echo XXRETCODE:$?' May 17 01:31:00 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 32p: open d32p/symlink->tmp/symlink->lustre-root/ ===================================== 01:30:59 (1305621059) May 17 01:31:00 client-5 xinetd[3391]: EXIT: shell status=0 pid=20326 duration=1(sec) May 17 01:31:00 client-5 xinetd[3391]: START: shell pid=20418 from=192.168.4.5 May 17 01:31:00 client-5 rshd[20420]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark 32p_1");echo XXRETCODE:$?' May 17 01:31:00 client-5 kernel: Lustre: DEBUG MARKER: 32p_1 May 17 01:31:00 client-5 xinetd[3391]: EXIT: shell status=0 pid=20418 duration=0(sec) May 17 01:31:00 client-5 xinetd[3391]: START: shell pid=20513 from=192.168.4.5 May 17 01:31:00 client-5 rshd[20514]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark 32p_2");echo XXRETCODE:$?' May 17 01:31:00 client-5 kernel: Lustre: DEBUG MARKER: 32p_2 May 17 01:31:00 client-5 xinetd[3391]: EXIT: shell status=0 pid=20513 duration=0(sec) May 17 01:31:00 client-5 xinetd[3391]: START: shell pid=20607 from=192.168.4.5 May 17 01:31:00 client-5 rshd[20608]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark 32p_3");echo XXRETCODE:$?' May 17 01:31:00 client-5 kernel: Lustre: DEBUG MARKER: 32p_3 May 17 01:31:00 client-5 xinetd[3391]: EXIT: shell status=0 pid=20607 duration=0(sec) May 17 01:31:00 client-5 xinetd[3391]: START: shell pid=20701 from=192.168.4.5 May 17 01:31:00 client-5 rshd[20702]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark 32p_4");echo XXRETCODE:$?' May 17 01:31:00 client-5 kernel: Lustre: DEBUG MARKER: 32p_4 May 17 01:31:00 client-5 xinetd[3391]: EXIT: shell status=0 pid=20701 duration=0(sec) May 17 01:31:00 client-5 xinetd[3391]: START: shell pid=20795 from=192.168.4.5 May 17 01:31:00 client-5 rshd[20796]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark 32p_5");echo XXRETCODE:$?' May 17 01:31:00 client-5 kernel: Lustre: DEBUG MARKER: 32p_5 May 17 01:31:00 client-5 xinetd[3391]: EXIT: shell status=0 pid=20795 duration=0(sec) May 17 01:31:00 client-5 xinetd[3391]: START: shell pid=20888 from=192.168.4.5 May 17 01:31:00 client-5 rshd[20889]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark 32p_6");echo XXRETCODE:$?' May 17 01:31:00 client-5 kernel: Lustre: DEBUG MARKER: 32p_6 May 17 01:31:00 client-5 xinetd[3391]: EXIT: shell status=0 pid=20888 duration=0(sec) May 17 01:31:01 client-5 xinetd[3391]: START: shell pid=20981 from=192.168.4.5 May 17 01:31:01 client-5 rshd[20983]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark 32p_7");echo XXRETCODE:$?' May 17 01:31:01 client-5 kernel: Lustre: DEBUG MARKER: 32p_7 May 17 01:31:01 client-5 xinetd[3391]: EXIT: shell status=0 pid=20981 duration=0(sec) May 17 01:31:01 client-5 xinetd[3391]: START: shell pid=21076 from=192.168.4.5 May 17 01:31:01 client-5 rshd[21077]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark 32p_8");echo XXRETCODE:$?' May 17 01:31:01 client-5 kernel: Lustre: DEBUG MARKER: 32p_8 May 17 01:31:01 client-5 xinetd[3391]: EXIT: shell status=0 pid=21076 duration=0(sec) May 17 01:31:01 client-5 xinetd[3391]: START: shell pid=21170 from=192.168.4.5 May 17 01:31:01 client-5 rshd[21171]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark 32p_9");echo XXRETCODE:$?' May 17 01:31:01 client-5 kernel: Lustre: DEBUG MARKER: 32p_9 May 17 01:31:01 client-5 xinetd[3391]: EXIT: shell status=0 pid=21170 duration=0(sec) May 17 01:31:01 client-5 xinetd[3391]: START: shell pid=21263 from=192.168.4.5 May 17 01:31:01 client-5 rshd[21265]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark 32p_10");echo XXRETCODE:$?' May 17 01:31:01 client-5 kernel: Lustre: DEBUG MARKER: 32p_10 May 17 01:31:01 client-5 xinetd[3391]: EXIT: shell status=0 pid=21263 duration=0(sec) May 17 01:31:01 client-5 xinetd[3391]: START: shell pid=21355 from=192.168.4.5 May 17 01:31:01 client-5 rshd[21356]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:31:01 client-5 xinetd[3391]: EXIT: shell status=0 pid=21355 duration=0(sec) May 17 01:31:02 client-5 xinetd[3391]: START: shell pid=21525 from=192.168.4.5 May 17 01:31:02 client-5 rshd[21528]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 32q: stat follows mountpoints in Lustre \(should return error\) ========================= 01:31:01 \(1305621061\)");echo XXRETCODE:$?' May 17 01:31:02 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 32q: stat follows mountpoints in Lustre (should return error) ========================= 01:31:01 (1305621061) May 17 01:31:02 client-5 xinetd[3391]: EXIT: shell status=0 pid=21525 duration=0(sec) May 17 01:31:02 client-5 kernel: EXT2-fs warning (device loop0): ext2_fill_super: mounting ext3 filesystem as ext2 May 17 01:31:02 client-5 xinetd[3391]: START: shell pid=21626 from=192.168.4.5 May 17 01:31:02 client-5 rshd[21627]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:31:02 client-5 xinetd[3391]: EXIT: shell status=0 pid=21626 duration=0(sec) May 17 01:31:02 client-5 xinetd[3391]: START: shell pid=21798 from=192.168.4.5 May 17 01:31:02 client-5 rshd[21799]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 32r: opendir follows mountpoints in Lustre \(should return error\) ====================== 01:31:02 \(1305621062\)");echo XXRETCODE:$?' May 17 01:31:02 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 32r: opendir follows mountpoints in Lustre (should return error) ====================== 01:31:02 (1305621062) May 17 01:31:02 client-5 xinetd[3391]: EXIT: shell status=0 pid=21798 duration=0(sec) May 17 01:31:02 client-5 kernel: EXT2-fs warning (device loop0): ext2_fill_super: mounting ext3 filesystem as ext2 May 17 01:31:02 client-5 xinetd[3391]: START: shell pid=21898 from=192.168.4.5 May 17 01:31:02 client-5 rshd[21899]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:31:02 client-5 xinetd[3391]: EXIT: shell status=0 pid=21898 duration=0(sec) May 17 01:31:03 client-5 xinetd[3391]: START: shell pid=22070 from=192.168.4.5 May 17 01:31:03 client-5 rshd[22071]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 33: write file with mode 444 \(should return error\) ==================================== 01:31:03 \(1305621063\)");echo XXRETCODE:$?' May 17 01:31:03 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 33: write file with mode 444 (should return error) ==================================== 01:31:03 (1305621063) May 17 01:31:03 client-5 xinetd[3391]: EXIT: shell status=0 pid=22070 duration=0(sec) May 17 01:31:03 client-5 xinetd[3391]: START: shell pid=22167 from=192.168.4.5 May 17 01:31:03 client-5 rshd[22168]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark 33_1");echo XXRETCODE:$?' May 17 01:31:03 client-5 kernel: Lustre: DEBUG MARKER: 33_1 May 17 01:31:03 client-5 xinetd[3391]: EXIT: shell status=0 pid=22167 duration=0(sec) May 17 01:31:03 client-5 xinetd[3391]: START: shell pid=22261 from=192.168.4.5 May 17 01:31:03 client-5 rshd[22262]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark 33_2");echo XXRETCODE:$?' May 17 01:31:03 client-5 kernel: Lustre: DEBUG MARKER: 33_2 May 17 01:31:03 client-5 xinetd[3391]: EXIT: shell status=0 pid=22261 duration=0(sec) May 17 01:31:03 client-5 xinetd[3391]: START: shell pid=22352 from=192.168.4.5 May 17 01:31:03 client-5 rshd[22353]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:31:03 client-5 xinetd[3391]: EXIT: shell status=0 pid=22352 duration=0(sec) May 17 01:31:03 client-5 xinetd[3391]: START: shell pid=22524 from=192.168.4.5 May 17 01:31:03 client-5 rshd[22525]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 33a: test open file\(mode=0444\) with O_RDWR \(should return error\) ====================== 01:31:03 \(1305621063\)");echo XXRETCODE:$?' May 17 01:31:04 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 33a: test open file(mode=0444) with O_RDWR (should return error) ====================== 01:31:03 (1305621063) May 17 01:31:04 client-5 xinetd[3391]: EXIT: shell status=0 pid=22524 duration=1(sec) May 17 01:31:04 client-5 xinetd[3391]: START: shell pid=22620 from=192.168.4.5 May 17 01:31:04 client-5 rshd[22621]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:31:04 client-5 xinetd[3391]: EXIT: shell status=0 pid=22620 duration=0(sec) May 17 01:31:04 client-5 xinetd[3391]: START: shell pid=22790 from=192.168.4.5 May 17 01:31:04 client-5 rshd[22793]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 33b: test open file with malformed flags \(No panic and return error\) ================== 01:31:04 \(1305621064\)");echo XXRETCODE:$?' May 17 01:31:04 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 33b: test open file with malformed flags (No panic and return error) ================== 01:31:04 (1305621064) May 17 01:31:04 client-5 xinetd[3391]: EXIT: shell status=0 pid=22790 duration=0(sec) May 17 01:31:04 client-5 xinetd[3391]: START: shell pid=22887 from=192.168.4.5 May 17 01:31:04 client-5 rshd[22888]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:31:04 client-5 xinetd[3391]: EXIT: shell status=0 pid=22887 duration=0(sec) May 17 01:31:05 client-5 xinetd[3391]: START: shell pid=23059 from=192.168.4.5 May 17 01:31:05 client-5 rshd[23060]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 33c: test llobdstat and write_bytes =================================================== 01:31:04 \(1305621064\)");echo XXRETCODE:$?' May 17 01:31:05 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 33c: test llobdstat and write_bytes =================================================== 01:31:04 (1305621064) May 17 01:31:05 client-5 xinetd[3391]: EXIT: shell status=0 pid=23059 duration=0(sec) May 17 01:31:05 client-5 xinetd[3391]: START: shell pid=23190 from=192.168.4.5 May 17 01:31:05 client-5 rshd[23191]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:31:05 client-5 xinetd[3391]: EXIT: shell status=0 pid=23190 duration=0(sec) May 17 01:31:05 client-5 xinetd[3391]: START: shell pid=23360 from=192.168.4.5 May 17 01:31:05 client-5 rshd[23363]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 34a: truncate file that has not been opened =========================================== 01:31:05 \(1305621065\)");echo XXRETCODE:$?' May 17 01:31:05 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 34a: truncate file that has not been opened =========================================== 01:31:05 (1305621065) May 17 01:31:05 client-5 xinetd[3391]: EXIT: shell status=0 pid=23360 duration=0(sec) May 17 01:31:06 client-5 xinetd[3391]: START: shell pid=23461 from=192.168.4.5 May 17 01:31:06 client-5 rshd[23462]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:31:06 client-5 xinetd[3391]: EXIT: shell status=0 pid=23461 duration=0(sec) May 17 01:31:06 client-5 xinetd[3391]: START: shell pid=23633 from=192.168.4.5 May 17 01:31:06 client-5 rshd[23634]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 34b: O_RDONLY opening file doesn\'t create objects ===================================== 01:31:06 \(1305621066\)");echo XXRETCODE:$?' May 17 01:31:06 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 34b: O_RDONLY opening file doesn't create objects ===================================== 01:31:06 (1305621066) May 17 01:31:06 client-5 xinetd[3391]: EXIT: shell status=0 pid=23633 duration=0(sec) May 17 01:31:06 client-5 xinetd[3391]: START: shell pid=23729 from=192.168.4.5 May 17 01:31:06 client-5 rshd[23730]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:31:06 client-5 xinetd[3391]: EXIT: shell status=0 pid=23729 duration=0(sec) May 17 01:31:06 client-5 xinetd[3391]: START: shell pid=23901 from=192.168.4.5 May 17 01:31:06 client-5 rshd[23902]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 34c: O_RDWR opening file-with-size works ============================================== 01:31:06 \(1305621066\)");echo XXRETCODE:$?' May 17 01:31:07 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 34c: O_RDWR opening file-with-size works ============================================== 01:31:06 (1305621066) May 17 01:31:07 client-5 xinetd[3391]: EXIT: shell status=0 pid=23901 duration=1(sec) May 17 01:31:07 client-5 xinetd[3391]: START: shell pid=23997 from=192.168.4.5 May 17 01:31:07 client-5 rshd[23998]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:31:07 client-5 xinetd[3391]: EXIT: shell status=0 pid=23997 duration=0(sec) May 17 01:31:07 client-5 xinetd[3391]: START: shell pid=24169 from=192.168.4.5 May 17 01:31:07 client-5 rshd[24170]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 34d: write to sparse file ============================================================= 01:31:07 \(1305621067\)");echo XXRETCODE:$?' May 17 01:31:07 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 34d: write to sparse file ============================================================= 01:31:07 (1305621067) May 17 01:31:07 client-5 xinetd[3391]: EXIT: shell status=0 pid=24169 duration=0(sec) May 17 01:31:07 client-5 xinetd[3391]: START: shell pid=24263 from=192.168.4.5 May 17 01:31:07 client-5 rshd[24264]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:31:07 client-5 xinetd[3391]: EXIT: shell status=0 pid=24263 duration=0(sec) May 17 01:31:07 client-5 xinetd[3391]: START: shell pid=24435 from=192.168.4.5 May 17 01:31:08 client-5 rshd[24436]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 34e: create objects, some with size and some without ================================== 01:31:07 \(1305621067\)");echo XXRETCODE:$?' May 17 01:31:08 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 34e: create objects, some with size and some without ================================== 01:31:07 (1305621067) May 17 01:31:08 client-5 xinetd[3391]: EXIT: shell status=0 pid=24435 duration=1(sec) May 17 01:31:08 client-5 xinetd[3391]: START: shell pid=24532 from=192.168.4.5 May 17 01:31:08 client-5 rshd[24533]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:31:08 client-5 xinetd[3391]: EXIT: shell status=0 pid=24532 duration=0(sec) May 17 01:31:08 client-5 xinetd[3391]: START: shell pid=24704 from=192.168.4.5 May 17 01:31:08 client-5 rshd[24705]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 34f: read from a file with no objects until EOF ======================================= 01:31:08 \(1305621068\)");echo XXRETCODE:$?' May 17 01:31:08 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 34f: read from a file with no objects until EOF ======================================= 01:31:08 (1305621068) May 17 01:31:08 client-5 xinetd[3391]: EXIT: shell status=0 pid=24704 duration=0(sec) May 17 01:31:08 client-5 xinetd[3391]: START: shell pid=24802 from=192.168.4.5 May 17 01:31:08 client-5 rshd[24805]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:31:08 client-5 xinetd[3391]: EXIT: shell status=0 pid=24802 duration=0(sec) May 17 01:31:09 client-5 xinetd[3391]: START: shell pid=24976 from=192.168.4.5 May 17 01:31:09 client-5 rshd[24977]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 34g: truncate long file =============================================================== 01:31:09 \(1305621069\)");echo XXRETCODE:$?' May 17 01:31:09 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 34g: truncate long file =============================================================== 01:31:09 (1305621069) May 17 01:31:09 client-5 xinetd[3391]: EXIT: shell status=0 pid=24976 duration=0(sec) May 17 01:31:09 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 01:31:09 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 01:31:09 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 01:31:09 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 01:31:09 client-5 xinetd[3391]: START: shell pid=25102 from=192.168.4.5 May 17 01:31:09 client-5 rshd[25103]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:31:09 client-5 xinetd[3391]: EXIT: shell status=0 pid=25102 duration=0(sec) May 17 01:31:09 client-5 xinetd[3391]: START: shell pid=25274 from=192.168.4.5 May 17 01:31:09 client-5 rshd[25275]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 35a: exec file with mode 444 \(should return and not leak\) ============================= 01:31:09 \(1305621069\)");echo XXRETCODE:$?' May 17 01:31:09 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 35a: exec file with mode 444 (should return and not leak) ============================= 01:31:09 (1305621069) May 17 01:31:09 client-5 xinetd[3391]: EXIT: shell status=0 pid=25274 duration=0(sec) May 17 01:31:10 client-5 xinetd[3391]: START: shell pid=25370 from=192.168.4.5 May 17 01:31:10 client-5 rshd[25371]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:31:10 client-5 xinetd[3391]: EXIT: shell status=0 pid=25370 duration=0(sec) May 17 01:31:10 client-5 xinetd[3391]: START: shell pid=25542 from=192.168.4.5 May 17 01:31:10 client-5 rshd[25543]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 36a: MDS utime check \(mknod, utime\) =================================================== 01:31:10 \(1305621070\)");echo XXRETCODE:$?' May 17 01:31:10 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 36a: MDS utime check (mknod, utime) =================================================== 01:31:10 (1305621070) May 17 01:31:10 client-5 xinetd[3391]: EXIT: shell status=0 pid=25542 duration=0(sec) May 17 01:31:10 client-5 xinetd[3391]: START: shell pid=25635 from=192.168.4.5 May 17 01:31:10 client-5 rshd[25636]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:31:10 client-5 xinetd[3391]: EXIT: shell status=0 pid=25635 duration=0(sec) May 17 01:31:10 client-5 xinetd[3391]: START: shell pid=25807 from=192.168.4.5 May 17 01:31:10 client-5 rshd[25808]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 36b: OST utime check \(open, utime\) ==================================================== 01:31:10 \(1305621070\)");echo XXRETCODE:$?' May 17 01:31:11 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 36b: OST utime check (open, utime) ==================================================== 01:31:10 (1305621070) May 17 01:31:11 client-5 xinetd[3391]: EXIT: shell status=0 pid=25807 duration=1(sec) May 17 01:31:11 client-5 xinetd[3391]: START: shell pid=25899 from=192.168.4.5 May 17 01:31:11 client-5 rshd[25900]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:31:11 client-5 xinetd[3391]: EXIT: shell status=0 pid=25899 duration=0(sec) May 17 01:31:11 client-5 xinetd[3391]: START: shell pid=26071 from=192.168.4.5 May 17 01:31:11 client-5 rshd[26072]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 36c: non-root MDS utime check \(mknod, utime\) ========================================== 01:31:11 \(1305621071\)");echo XXRETCODE:$?' May 17 01:31:11 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 36c: non-root MDS utime check (mknod, utime) ========================================== 01:31:11 (1305621071) May 17 01:31:11 client-5 xinetd[3391]: EXIT: shell status=0 pid=26071 duration=0(sec) May 17 01:31:11 client-5 xinetd[3391]: START: shell pid=26166 from=192.168.4.5 May 17 01:31:11 client-5 rshd[26167]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:31:11 client-5 xinetd[3391]: EXIT: shell status=0 pid=26166 duration=0(sec) May 17 01:31:12 client-5 xinetd[3391]: START: shell pid=26336 from=192.168.4.5 May 17 01:31:12 client-5 rshd[26339]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 36d: non-root OST utime check \(open, utime\) =========================================== 01:31:11 \(1305621071\)");echo XXRETCODE:$?' May 17 01:31:12 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 36d: non-root OST utime check (open, utime) =========================================== 01:31:11 (1305621071) May 17 01:31:12 client-5 xinetd[3391]: EXIT: shell status=0 pid=26336 duration=0(sec) May 17 01:31:12 client-5 xinetd[3391]: START: shell pid=26430 from=192.168.4.5 May 17 01:31:12 client-5 rshd[26431]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:31:12 client-5 xinetd[3391]: EXIT: shell status=0 pid=26430 duration=0(sec) May 17 01:31:12 client-5 xinetd[3391]: START: shell pid=26602 from=192.168.4.5 May 17 01:31:12 client-5 rshd[26603]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 36e: utime on non-owned file \(should return error\) ==================================== 01:31:12 \(1305621072\)");echo XXRETCODE:$?' May 17 01:31:12 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 36e: utime on non-owned file (should return error) ==================================== 01:31:12 (1305621072) May 17 01:31:12 client-5 xinetd[3391]: EXIT: shell status=0 pid=26602 duration=0(sec) May 17 01:31:12 client-5 xinetd[3391]: START: shell pid=26695 from=192.168.4.5 May 17 01:31:12 client-5 rshd[26697]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:31:12 client-5 xinetd[3391]: EXIT: shell status=0 pid=26695 duration=0(sec) May 17 01:31:13 client-5 xinetd[3391]: START: shell pid=26868 from=192.168.4.5 May 17 01:31:13 client-5 rshd[26869]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 36f: utime on file racing with OST BRW write ========================================== 01:31:12 \(1305621072\)");echo XXRETCODE:$?' May 17 01:31:13 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 36f: utime on file racing with OST BRW write ========================================== 01:31:12 (1305621072) May 17 01:31:13 client-5 xinetd[3391]: EXIT: shell status=0 pid=26868 duration=0(sec) May 17 01:31:14 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 01:31:14 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 01:31:14 client-5 xinetd[3391]: START: shell pid=26985 from=192.168.4.5 May 17 01:31:14 client-5 rshd[26986]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:31:14 client-5 xinetd[3391]: EXIT: shell status=0 pid=26985 duration=0(sec) May 17 01:31:14 client-5 xinetd[3391]: START: shell pid=27155 from=192.168.4.5 May 17 01:31:14 client-5 rshd[27158]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 36g: filter mod data cache expiry ===================================================== 01:31:14 \(1305621074\)");echo XXRETCODE:$?' May 17 01:31:14 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 36g: filter mod data cache expiry ===================================================== 01:31:14 (1305621074) May 17 01:31:14 client-5 xinetd[3391]: EXIT: shell status=0 pid=27155 duration=0(sec) May 17 01:31:57 client-5 xinetd[3391]: START: shell pid=27301 from=192.168.4.5 May 17 01:31:57 client-5 rshd[27304]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:31:57 client-5 xinetd[3391]: EXIT: shell status=0 pid=27301 duration=0(sec) May 17 01:31:57 client-5 xinetd[3391]: START: shell pid=27476 from=192.168.4.5 May 17 01:31:57 client-5 rshd[27477]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 36h: utime on file racing with OST BRW write ========================================== 01:31:57 \(1305621117\)");echo XXRETCODE:$?' May 17 01:31:57 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 36h: utime on file racing with OST BRW write ========================================== 01:31:57 (1305621117) May 17 01:31:57 client-5 xinetd[3391]: EXIT: shell status=0 pid=27476 duration=0(sec) May 17 01:31:58 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 01:31:58 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 01:31:58 client-5 xinetd[3391]: START: shell pid=27593 from=192.168.4.5 May 17 01:31:58 client-5 rshd[27594]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:31:58 client-5 xinetd[3391]: EXIT: shell status=0 pid=27593 duration=0(sec) May 17 01:31:59 client-5 xinetd[3391]: START: shell pid=27765 from=192.168.4.5 May 17 01:31:59 client-5 rshd[27766]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 37: ls a mounted file system to check old content ===================================== 01:31:59 \(1305621119\)");echo XXRETCODE:$?' May 17 01:31:59 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 37: ls a mounted file system to check old content ===================================== 01:31:59 (1305621119) May 17 01:31:59 client-5 xinetd[3391]: EXIT: shell status=0 pid=27765 duration=0(sec) May 17 01:31:59 client-5 kernel: EXT2-fs warning (device loop0): ext2_fill_super: mounting ext3 filesystem as ext2 May 17 01:31:59 client-5 xinetd[3391]: START: shell pid=27865 from=192.168.4.5 May 17 01:31:59 client-5 rshd[27866]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:31:59 client-5 xinetd[3391]: EXIT: shell status=0 pid=27865 duration=0(sec) May 17 01:31:59 client-5 xinetd[3391]: START: shell pid=28035 from=192.168.4.5 May 17 01:31:59 client-5 rshd[28038]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 38: open a regular file with O_DIRECTORY should return -ENOTDIR ======================= 01:31:59 \(1305621119\)");echo XXRETCODE:$?' May 17 01:31:59 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 38: open a regular file with O_DIRECTORY should return -ENOTDIR ======================= 01:31:59 (1305621119) May 17 01:31:59 client-5 xinetd[3391]: EXIT: shell status=0 pid=28035 duration=0(sec) May 17 01:31:59 client-5 xinetd[3391]: START: shell pid=28130 from=192.168.4.5 May 17 01:31:59 client-5 rshd[28131]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:31:59 client-5 xinetd[3391]: EXIT: shell status=0 pid=28130 duration=0(sec) May 17 01:32:00 client-5 xinetd[3391]: START: shell pid=28302 from=192.168.4.5 May 17 01:32:00 client-5 rshd[28303]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 39: mtime changed on create =========================================================== 01:32:00 \(1305621120\)");echo XXRETCODE:$?' May 17 01:32:00 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 39: mtime changed on create =========================================================== 01:32:00 (1305621120) May 17 01:32:00 client-5 xinetd[3391]: EXIT: shell status=0 pid=28302 duration=0(sec) May 17 01:32:02 client-5 xinetd[3391]: START: shell pid=28396 from=192.168.4.5 May 17 01:32:02 client-5 rshd[28398]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:32:02 client-5 xinetd[3391]: EXIT: shell status=0 pid=28396 duration=0(sec) May 17 01:32:02 client-5 xinetd[3391]: START: shell pid=28569 from=192.168.4.5 May 17 01:32:02 client-5 rshd[28570]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 39b: mtime change on open, link, unlink, rename ====================================== 01:32:02 \(1305621122\)");echo XXRETCODE:$?' May 17 01:32:02 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 39b: mtime change on open, link, unlink, rename ====================================== 01:32:02 (1305621122) May 17 01:32:02 client-5 xinetd[3391]: EXIT: shell status=0 pid=28569 duration=0(sec) May 17 01:32:04 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 01:32:04 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 01:32:04 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 01:32:04 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 01:32:04 client-5 xinetd[3391]: START: shell pid=28711 from=192.168.4.5 May 17 01:32:04 client-5 rshd[28712]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:32:04 client-5 xinetd[3391]: EXIT: shell status=0 pid=28711 duration=0(sec) May 17 01:32:04 client-5 xinetd[3391]: START: shell pid=28884 from=192.168.4.5 May 17 01:32:04 client-5 rshd[28885]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 39c: mtime change on rename =========================================================== 01:32:04 \(1305621124\)");echo XXRETCODE:$?' May 17 01:32:04 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 39c: mtime change on rename =========================================================== 01:32:04 (1305621124) May 17 01:32:04 client-5 xinetd[3391]: EXIT: shell status=0 pid=28884 duration=0(sec) May 17 01:32:06 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 01:32:06 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 01:32:06 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 01:32:06 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 01:32:06 client-5 xinetd[3391]: START: shell pid=29014 from=192.168.4.5 May 17 01:32:06 client-5 rshd[29015]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:32:07 client-5 xinetd[3391]: EXIT: shell status=0 pid=29014 duration=1(sec) May 17 01:32:07 client-5 xinetd[3391]: START: shell pid=29184 from=192.168.4.5 May 17 01:32:07 client-5 rshd[29187]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 39d: create, utime, stat ============================================================== 01:32:07 \(1305621127\)");echo XXRETCODE:$?' May 17 01:32:07 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 39d: create, utime, stat ============================================================== 01:32:07 (1305621127) May 17 01:32:07 client-5 xinetd[3391]: EXIT: shell status=0 pid=29184 duration=0(sec) May 17 01:32:07 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 01:32:07 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 01:32:07 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 01:32:07 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 01:32:07 client-5 xinetd[3391]: START: shell pid=29309 from=192.168.4.5 May 17 01:32:07 client-5 rshd[29310]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:32:07 client-5 xinetd[3391]: EXIT: shell status=0 pid=29309 duration=0(sec) May 17 01:32:07 client-5 xinetd[3391]: START: shell pid=29480 from=192.168.4.5 May 17 01:32:07 client-5 rshd[29482]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 39e: create, stat, utime, stat ======================================================== 01:32:07 \(1305621127\)");echo XXRETCODE:$?' May 17 01:32:08 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 39e: create, stat, utime, stat ======================================================== 01:32:07 (1305621127) May 17 01:32:08 client-5 xinetd[3391]: EXIT: shell status=0 pid=29480 duration=1(sec) May 17 01:32:08 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 01:32:08 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 01:32:08 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 01:32:08 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 01:32:08 client-5 xinetd[3391]: START: shell pid=29605 from=192.168.4.5 May 17 01:32:08 client-5 rshd[29606]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:32:08 client-5 xinetd[3391]: EXIT: shell status=0 pid=29605 duration=0(sec) May 17 01:32:08 client-5 xinetd[3391]: START: shell pid=29777 from=192.168.4.5 May 17 01:32:08 client-5 rshd[29778]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 39f: create, stat, sleep, utime, stat ================================================= 01:32:08 \(1305621128\)");echo XXRETCODE:$?' May 17 01:32:08 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 39f: create, stat, sleep, utime, stat ================================================= 01:32:08 (1305621128) May 17 01:32:08 client-5 xinetd[3391]: EXIT: shell status=0 pid=29777 duration=0(sec) May 17 01:32:10 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 01:32:10 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 01:32:10 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 01:32:10 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 01:32:10 client-5 xinetd[3391]: START: shell pid=29902 from=192.168.4.5 May 17 01:32:10 client-5 rshd[29903]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:32:10 client-5 xinetd[3391]: EXIT: shell status=0 pid=29902 duration=0(sec) May 17 01:32:11 client-5 xinetd[3391]: START: shell pid=30072 from=192.168.4.5 May 17 01:32:11 client-5 rshd[30075]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 39g: write, chmod, stat =============================================================== 01:32:11 \(1305621131\)");echo XXRETCODE:$?' May 17 01:32:11 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 39g: write, chmod, stat =============================================================== 01:32:11 (1305621131) May 17 01:32:11 client-5 xinetd[3391]: EXIT: shell status=0 pid=30072 duration=0(sec) May 17 01:32:13 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 01:32:13 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 01:32:13 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 01:32:13 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 01:32:13 client-5 xinetd[3391]: START: shell pid=30196 from=192.168.4.5 May 17 01:32:13 client-5 rshd[30199]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:32:13 client-5 xinetd[3391]: EXIT: shell status=0 pid=30196 duration=0(sec) May 17 01:32:13 client-5 xinetd[3391]: START: shell pid=30368 from=192.168.4.5 May 17 01:32:13 client-5 rshd[30371]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 39h: write, utime within one second, stat ============================================= 01:32:13 \(1305621133\)");echo XXRETCODE:$?' May 17 01:32:13 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 39h: write, utime within one second, stat ============================================= 01:32:13 (1305621133) May 17 01:32:13 client-5 xinetd[3391]: EXIT: shell status=0 pid=30368 duration=0(sec) May 17 01:32:14 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 01:32:14 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 01:32:14 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 01:32:14 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 01:32:14 client-5 xinetd[3391]: START: shell pid=30501 from=192.168.4.5 May 17 01:32:15 client-5 rshd[30502]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:32:15 client-5 xinetd[3391]: EXIT: shell status=0 pid=30501 duration=1(sec) May 17 01:32:15 client-5 xinetd[3391]: START: shell pid=30673 from=192.168.4.5 May 17 01:32:15 client-5 rshd[30674]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 39i: write, rename, stat ============================================================== 01:32:15 \(1305621135\)");echo XXRETCODE:$?' May 17 01:32:15 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 39i: write, rename, stat ============================================================== 01:32:15 (1305621135) May 17 01:32:15 client-5 xinetd[3391]: EXIT: shell status=0 pid=30673 duration=0(sec) May 17 01:32:16 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 01:32:16 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 01:32:16 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 01:32:16 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 01:32:16 client-5 xinetd[3391]: START: shell pid=30798 from=192.168.4.5 May 17 01:32:16 client-5 rshd[30799]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:32:16 client-5 xinetd[3391]: EXIT: shell status=0 pid=30798 duration=0(sec) May 17 01:32:16 client-5 xinetd[3391]: START: shell pid=30970 from=192.168.4.5 May 17 01:32:17 client-5 rshd[30971]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 39j: write, rename, close, stat ======================================================= 01:32:16 \(1305621136\)");echo XXRETCODE:$?' May 17 01:32:17 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 39j: write, rename, close, stat ======================================================= 01:32:16 (1305621136) May 17 01:32:17 client-5 xinetd[3391]: EXIT: shell status=0 pid=30970 duration=1(sec) May 17 01:32:18 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 01:32:18 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 01:32:18 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 01:32:18 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 01:32:18 client-5 xinetd[3391]: START: shell pid=31098 from=192.168.4.5 May 17 01:32:18 client-5 rshd[31099]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:32:18 client-5 xinetd[3391]: EXIT: shell status=0 pid=31098 duration=0(sec) May 17 01:32:18 client-5 xinetd[3391]: START: shell pid=31270 from=192.168.4.5 May 17 01:32:18 client-5 rshd[31271]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 39k: write, utime, close, stat ======================================================== 01:32:18 \(1305621138\)");echo XXRETCODE:$?' May 17 01:32:18 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 39k: write, utime, close, stat ======================================================== 01:32:18 (1305621138) May 17 01:32:18 client-5 xinetd[3391]: EXIT: shell status=0 pid=31270 duration=0(sec) May 17 01:32:20 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 01:32:20 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 01:32:20 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 01:32:20 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 01:32:20 client-5 xinetd[3391]: START: shell pid=31397 from=192.168.4.5 May 17 01:32:20 client-5 rshd[31399]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:32:20 client-5 xinetd[3391]: EXIT: shell status=0 pid=31397 duration=0(sec) May 17 01:32:20 client-5 xinetd[3391]: START: shell pid=31571 from=192.168.4.5 May 17 01:32:20 client-5 rshd[31572]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 39l: directory atime update =========================================================== 01:32:20 \(1305621140\)");echo XXRETCODE:$?' May 17 01:32:20 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 39l: directory atime update =========================================================== 01:32:20 (1305621140) May 17 01:32:20 client-5 xinetd[3391]: EXIT: shell status=0 pid=31571 duration=0(sec) May 17 01:32:20 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc start May 17 01:32:21 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc stop May 17 01:32:24 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc start May 17 01:32:24 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc stop May 17 01:32:27 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc start May 17 01:32:27 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc stop May 17 01:32:27 client-5 xinetd[3391]: START: shell pid=31737 from=192.168.4.5 May 17 01:32:27 client-5 rshd[31738]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:32:27 client-5 xinetd[3391]: EXIT: shell status=0 pid=31737 duration=0(sec) May 17 01:32:27 client-5 xinetd[3391]: START: shell pid=31909 from=192.168.4.5 May 17 01:32:27 client-5 rshd[31910]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 39m: test atime and mtime before 1970 ================================================= 01:32:27 \(1305621147\)");echo XXRETCODE:$?' May 17 01:32:27 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 39m: test atime and mtime before 1970 ================================================= 01:32:27 (1305621147) May 17 01:32:27 client-5 xinetd[3391]: EXIT: shell status=0 pid=31909 duration=0(sec) May 17 01:32:29 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 01:32:30 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 01:32:30 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 01:32:30 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 01:32:30 client-5 xinetd[3391]: START: shell pid=32036 from=192.168.4.5 May 17 01:32:30 client-5 rshd[32037]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:32:30 client-5 xinetd[3391]: EXIT: shell status=0 pid=32036 duration=0(sec) May 17 01:32:30 client-5 xinetd[3391]: START: shell pid=32206 from=192.168.4.5 May 17 01:32:30 client-5 rshd[32209]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 40: failed open\(O_TRUNC\) doesn\'t truncate ============================================= 01:32:30 \(1305621150\)");echo XXRETCODE:$?' May 17 01:32:30 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 40: failed open(O_TRUNC) doesn't truncate ============================================= 01:32:30 (1305621150) May 17 01:32:30 client-5 xinetd[3391]: EXIT: shell status=0 pid=32206 duration=0(sec) May 17 01:32:30 client-5 xinetd[3391]: START: shell pid=32302 from=192.168.4.5 May 17 01:32:30 client-5 rshd[32303]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:32:30 client-5 xinetd[3391]: EXIT: shell status=0 pid=32302 duration=0(sec) May 17 01:32:30 client-5 xinetd[3391]: START: shell pid=32474 from=192.168.4.5 May 17 01:32:31 client-5 rshd[32475]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 41: test small file write + fstat ===================================================== 01:32:30 \(1305621150\)");echo XXRETCODE:$?' May 17 01:32:31 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 41: test small file write + fstat ===================================================== 01:32:30 (1305621150) May 17 01:32:31 client-5 xinetd[3391]: EXIT: shell status=0 pid=32474 duration=1(sec) May 17 01:32:32 client-5 xinetd[3391]: START: shell pid=32566 from=192.168.4.5 May 17 01:32:32 client-5 rshd[32567]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:32:32 client-5 xinetd[3391]: EXIT: shell status=0 pid=32566 duration=0(sec) May 17 01:32:32 client-5 xinetd[3391]: START: shell pid=32729 from=192.168.4.5 May 17 01:32:32 client-5 rshd[32730]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark SKIP: sanity test_42a skipping ALWAYS excluded test 42a");echo XXRETCODE:$?' May 17 01:32:32 client-5 kernel: Lustre: DEBUG MARKER: SKIP: sanity test_42a skipping ALWAYS excluded test 42a May 17 01:32:32 client-5 xinetd[3391]: EXIT: shell status=0 pid=32729 duration=0(sec) May 17 01:32:32 client-5 xinetd[3391]: START: shell pid=358 from=192.168.4.5 May 17 01:32:32 client-5 rshd[360]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark SKIP: sanity test_42b skipping ALWAYS excluded test 42b");echo XXRETCODE:$?' May 17 01:32:32 client-5 kernel: Lustre: DEBUG MARKER: SKIP: sanity test_42b skipping ALWAYS excluded test 42b May 17 01:32:32 client-5 xinetd[3391]: EXIT: shell status=0 pid=358 duration=0(sec) May 17 01:32:32 client-5 xinetd[3391]: START: shell pid=463 from=192.168.4.5 May 17 01:32:32 client-5 rshd[464]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark SKIP: sanity test_42c skipping ALWAYS excluded test 42c");echo XXRETCODE:$?' May 17 01:32:32 client-5 kernel: Lustre: DEBUG MARKER: SKIP: sanity test_42c skipping ALWAYS excluded test 42c May 17 01:32:32 client-5 xinetd[3391]: EXIT: shell status=0 pid=463 duration=0(sec) May 17 01:32:33 client-5 xinetd[3391]: START: shell pid=560 from=192.168.4.5 May 17 01:32:33 client-5 rshd[561]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark SKIP: sanity test_42d skipping ALWAYS excluded test 42d");echo XXRETCODE:$?' May 17 01:32:33 client-5 kernel: Lustre: DEBUG MARKER: SKIP: sanity test_42d skipping ALWAYS excluded test 42d May 17 01:32:33 client-5 xinetd[3391]: EXIT: shell status=0 pid=560 duration=0(sec) May 17 01:32:33 client-5 xinetd[3391]: START: shell pid=679 from=192.168.4.5 May 17 01:32:33 client-5 rshd[680]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 42e: verify sub-RPC writes are not done synchronously ================================= 01:32:33 \(1305621153\)");echo XXRETCODE:$?' May 17 01:32:33 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 42e: verify sub-RPC writes are not done synchronously ================================= 01:32:33 (1305621153) May 17 01:32:33 client-5 xinetd[3391]: EXIT: shell status=0 pid=679 duration=0(sec) May 17 01:33:20 client-5 xinetd[3391]: START: shell pid=4431 from=192.168.4.5 May 17 01:33:20 client-5 rshd[4432]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:33:20 client-5 xinetd[3391]: EXIT: shell status=0 pid=4431 duration=0(sec) May 17 01:33:20 client-5 xinetd[3391]: START: shell pid=4603 from=192.168.4.5 May 17 01:33:20 client-5 rshd[4604]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 43: execution of file opened for write should return -ETXTBSY ========================= 01:33:20 \(1305621200\)");echo XXRETCODE:$?' May 17 01:33:20 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 43: execution of file opened for write should return -ETXTBSY ========================= 01:33:20 (1305621200) May 17 01:33:20 client-5 xinetd[3391]: EXIT: shell status=0 pid=4603 duration=0(sec) May 17 01:33:21 client-5 xinetd[3391]: START: shell pid=4697 from=192.168.4.5 May 17 01:33:21 client-5 rshd[4700]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:33:21 client-5 xinetd[3391]: EXIT: shell status=0 pid=4697 duration=0(sec) May 17 01:33:21 client-5 xinetd[3391]: START: shell pid=4869 from=192.168.4.5 May 17 01:33:22 client-5 rshd[4872]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 43a: open\(RDWR\) of file being executed should return -ETXTBSY ========================= 01:33:21 \(1305621201\)");echo XXRETCODE:$?' May 17 01:33:22 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 43a: open(RDWR) of file being executed should return -ETXTBSY ========================= 01:33:21 (1305621201) May 17 01:33:22 client-5 xinetd[3391]: EXIT: shell status=0 pid=4869 duration=1(sec) May 17 01:33:22 client-5 xinetd[3391]: START: shell pid=4970 from=192.168.4.5 May 17 01:33:22 client-5 rshd[4971]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:33:22 client-5 xinetd[3391]: EXIT: shell status=0 pid=4970 duration=0(sec) May 17 01:33:22 client-5 xinetd[3391]: START: shell pid=5142 from=192.168.4.5 May 17 01:33:22 client-5 rshd[5143]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 43b: truncate of file being executed should return -ETXTBSY =========================== 01:33:22 \(1305621202\)");echo XXRETCODE:$?' May 17 01:33:22 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 43b: truncate of file being executed should return -ETXTBSY =========================== 01:33:22 (1305621202) May 17 01:33:22 client-5 xinetd[3391]: EXIT: shell status=0 pid=5142 duration=0(sec) May 17 01:33:22 client-5 xinetd[3391]: START: shell pid=5249 from=192.168.4.5 May 17 01:33:22 client-5 rshd[5252]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:33:22 client-5 xinetd[3391]: EXIT: shell status=0 pid=5249 duration=0(sec) May 17 01:33:23 client-5 xinetd[3391]: START: shell pid=5422 from=192.168.4.5 May 17 01:33:23 client-5 rshd[5424]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 43c: md5sum of copy into lustre======================================================== 01:33:23 \(1305621203\)");echo XXRETCODE:$?' May 17 01:33:23 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 43c: md5sum of copy into lustre======================================================== 01:33:23 (1305621203) May 17 01:33:23 client-5 xinetd[3391]: EXIT: shell status=0 pid=5422 duration=0(sec) May 17 01:33:23 client-5 xinetd[3391]: START: shell pid=5521 from=192.168.4.5 May 17 01:33:23 client-5 rshd[5523]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:33:23 client-5 xinetd[3391]: EXIT: shell status=0 pid=5521 duration=0(sec) May 17 01:33:23 client-5 xinetd[3391]: START: shell pid=5694 from=192.168.4.5 May 17 01:33:23 client-5 rshd[5695]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 44: zero length read from a sparse stripe ============================================= 01:33:23 \(1305621203\)");echo XXRETCODE:$?' May 17 01:33:23 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 44: zero length read from a sparse stripe ============================================= 01:33:23 (1305621203) May 17 01:33:23 client-5 xinetd[3391]: EXIT: shell status=0 pid=5694 duration=0(sec) May 17 01:33:23 client-5 xinetd[3391]: START: shell pid=5787 from=192.168.4.5 May 17 01:33:24 client-5 rshd[5788]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:33:24 client-5 xinetd[3391]: EXIT: shell status=0 pid=5787 duration=1(sec) May 17 01:33:24 client-5 xinetd[3391]: START: shell pid=5959 from=192.168.4.5 May 17 01:33:24 client-5 rshd[5960]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 44a: test sparse pwrite =============================================================== 01:33:24 \(1305621204\)");echo XXRETCODE:$?' May 17 01:33:24 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 44a: test sparse pwrite =============================================================== 01:33:24 (1305621204) May 17 01:33:24 client-5 xinetd[3391]: EXIT: shell status=0 pid=5959 duration=0(sec) May 17 01:33:25 client-5 xinetd[3391]: START: shell pid=6079 from=192.168.4.5 May 17 01:33:25 client-5 rshd[6080]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:33:25 client-5 xinetd[3391]: EXIT: shell status=0 pid=6079 duration=0(sec) May 17 01:33:25 client-5 xinetd[3391]: START: shell pid=6242 from=192.168.4.5 May 17 01:33:25 client-5 rshd[6243]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark SKIP: sanity test_45 skipping ALWAYS excluded test 45");echo XXRETCODE:$?' May 17 01:33:25 client-5 kernel: Lustre: DEBUG MARKER: SKIP: sanity test_45 skipping ALWAYS excluded test 45 May 17 01:33:25 client-5 xinetd[3391]: EXIT: shell status=0 pid=6242 duration=0(sec) May 17 01:33:25 client-5 xinetd[3391]: START: shell pid=6348 from=192.168.4.5 May 17 01:33:25 client-5 rshd[6349]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 46: dirtying a previously written page ================================================ 01:33:25 \(1305621205\)");echo XXRETCODE:$?' May 17 01:33:26 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 46: dirtying a previously written page ================================================ 01:33:25 (1305621205) May 17 01:33:26 client-5 xinetd[3391]: EXIT: shell status=0 pid=6348 duration=1(sec) May 17 01:33:26 client-5 xinetd[3391]: START: shell pid=6460 from=192.168.4.5 May 17 01:33:26 client-5 rshd[6462]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:33:26 client-5 xinetd[3391]: EXIT: shell status=0 pid=6460 duration=0(sec) May 17 01:33:26 client-5 xinetd[3391]: START: shell pid=6633 from=192.168.4.5 May 17 01:33:26 client-5 rshd[6634]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 48a: Access renamed working dir \(should return errors\)================================= 01:33:26 \(1305621206\)");echo XXRETCODE:$?' May 17 01:33:26 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 48a: Access renamed working dir (should return errors)================================= 01:33:26 (1305621206) May 17 01:33:26 client-5 xinetd[3391]: EXIT: shell status=0 pid=6633 duration=0(sec) May 17 01:33:26 client-5 xinetd[3391]: START: shell pid=6741 from=192.168.4.5 May 17 01:33:26 client-5 rshd[6743]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:33:26 client-5 xinetd[3391]: EXIT: shell status=0 pid=6741 duration=0(sec) May 17 01:33:27 client-5 xinetd[3391]: START: shell pid=6914 from=192.168.4.5 May 17 01:33:27 client-5 rshd[6915]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 48b: Access removed working dir \(should return errors\)================================= 01:33:27 \(1305621207\)");echo XXRETCODE:$?' May 17 01:33:27 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 48b: Access removed working dir (should return errors)================================= 01:33:27 (1305621207) May 17 01:33:27 client-5 xinetd[3391]: EXIT: shell status=0 pid=6914 duration=0(sec) May 17 01:33:27 client-5 kernel: LustreError: 6958:0:(file.c:2161:ll_inode_revalidate_fini()) failure -2 inode 144115205255839374 May 17 01:33:27 client-5 kernel: LustreError: 6851:0:(file.c:2161:ll_inode_revalidate_fini()) failure -2 inode 144115205255839374 May 17 01:33:27 client-5 xinetd[3391]: START: shell pid=7024 from=192.168.4.5 May 17 01:33:27 client-5 rshd[7025]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:33:27 client-5 xinetd[3391]: EXIT: shell status=0 pid=7024 duration=0(sec) May 17 01:33:28 client-5 xinetd[3391]: START: shell pid=7196 from=192.168.4.5 May 17 01:33:28 client-5 rshd[7197]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 48c: Access removed working subdir \(should return errors\) ============================= 01:33:28 \(1305621208\)");echo XXRETCODE:$?' May 17 01:33:28 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 48c: Access removed working subdir (should return errors) ============================= 01:33:28 (1305621208) May 17 01:33:28 client-5 xinetd[3391]: EXIT: shell status=0 pid=7196 duration=0(sec) May 17 01:33:28 client-5 kernel: LustreError: 7240:0:(file.c:2161:ll_inode_revalidate_fini()) failure -2 inode 144115205255839376 May 17 01:33:28 client-5 kernel: LustreError: 7240:0:(file.c:2161:ll_inode_revalidate_fini()) Skipped 4 previous similar messages May 17 01:33:28 client-5 xinetd[3391]: START: shell pid=7306 from=192.168.4.5 May 17 01:33:28 client-5 rshd[7307]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:33:28 client-5 xinetd[3391]: EXIT: shell status=0 pid=7306 duration=0(sec) May 17 01:33:29 client-5 xinetd[3391]: START: shell pid=7478 from=192.168.4.5 May 17 01:33:29 client-5 rshd[7479]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 48d: Access removed parent subdir \(should return errors\) ============================== 01:33:29 \(1305621209\)");echo XXRETCODE:$?' May 17 01:33:29 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 48d: Access removed parent subdir (should return errors) ============================== 01:33:29 (1305621209) May 17 01:33:29 client-5 xinetd[3391]: EXIT: shell status=0 pid=7478 duration=0(sec) May 17 01:33:29 client-5 kernel: LustreError: 7523:0:(file.c:2161:ll_inode_revalidate_fini()) failure -116 inode 144115205255839378 May 17 01:33:29 client-5 kernel: LustreError: 7523:0:(file.c:2161:ll_inode_revalidate_fini()) Skipped 5 previous similar messages May 17 01:33:29 client-5 xinetd[3391]: START: shell pid=7595 from=192.168.4.5 May 17 01:33:30 client-5 rshd[7596]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:33:30 client-5 xinetd[3391]: EXIT: shell status=0 pid=7595 duration=1(sec) May 17 01:33:30 client-5 xinetd[3391]: START: shell pid=7767 from=192.168.4.5 May 17 01:33:30 client-5 rshd[7768]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 48e: Access to recreated parent subdir \(should return errors\) ========================= 01:33:30 \(1305621210\)");echo XXRETCODE:$?' May 17 01:33:30 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 48e: Access to recreated parent subdir (should return errors) ========================= 01:33:30 (1305621210) May 17 01:33:30 client-5 xinetd[3391]: EXIT: shell status=0 pid=7767 duration=0(sec) May 17 01:33:30 client-5 xinetd[3391]: START: shell pid=7867 from=192.168.4.5 May 17 01:33:30 client-5 rshd[7869]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:33:30 client-5 xinetd[3391]: EXIT: shell status=0 pid=7867 duration=0(sec) May 17 01:33:30 client-5 xinetd[3391]: START: shell pid=8040 from=192.168.4.5 May 17 01:33:30 client-5 rshd[8041]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 50: special situations: \/proc symlinks =============================================== 01:33:30 \(1305621210\)");echo XXRETCODE:$?' May 17 01:33:30 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 50: special situations: /proc symlinks =============================================== 01:33:30 (1305621210) May 17 01:33:31 client-5 xinetd[3391]: EXIT: shell status=0 pid=8040 duration=1(sec) May 17 01:33:31 client-5 xinetd[3391]: START: shell pid=8132 from=192.168.4.5 May 17 01:33:31 client-5 rshd[8134]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:33:31 client-5 xinetd[3391]: EXIT: shell status=0 pid=8132 duration=0(sec) May 17 01:33:31 client-5 xinetd[3391]: START: shell pid=8305 from=192.168.4.5 May 17 01:33:31 client-5 rshd[8306]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 51a: special situations: split htree with empty entry ================================= 01:33:31 \(1305621211\)");echo XXRETCODE:$?' May 17 01:33:31 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 51a: special situations: split htree with empty entry ================================= 01:33:31 (1305621211) May 17 01:33:31 client-5 xinetd[3391]: EXIT: shell status=0 pid=8305 duration=0(sec) May 17 01:33:31 client-5 xinetd[3391]: START: shell pid=8404 from=192.168.4.5 May 17 01:33:31 client-5 rshd[8407]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:33:32 client-5 xinetd[3391]: EXIT: shell status=0 pid=8404 duration=1(sec) May 17 01:33:32 client-5 xinetd[3391]: START: shell pid=8579 from=192.168.4.5 May 17 01:33:32 client-5 rshd[8580]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 51b: mkdir ...\/t-0 --- ...\/t-70 ======================================================= 01:33:32 \(1305621212\)");echo XXRETCODE:$?' May 17 01:33:32 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 51b: mkdir .../t-0 --- .../t-70 ======================================================= 01:33:32 (1305621212) May 17 01:33:32 client-5 xinetd[3391]: EXIT: shell status=0 pid=8579 duration=0(sec) May 17 01:33:32 client-5 xinetd[3391]: START: shell pid=8679 from=192.168.4.5 May 17 01:33:32 client-5 rshd[8680]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:33:32 client-5 xinetd[3391]: EXIT: shell status=0 pid=8679 duration=0(sec) May 17 01:33:32 client-5 xinetd[3391]: START: shell pid=8851 from=192.168.4.5 May 17 01:33:32 client-5 rshd[8852]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 51bb: mkdir createmany CMD 1 ========================================================= 01:33:32 \(1305621212\)");echo XXRETCODE:$?' May 17 01:33:33 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 51bb: mkdir createmany CMD 1 ========================================================= 01:33:32 (1305621212) May 17 01:33:33 client-5 xinetd[3391]: EXIT: shell status=0 pid=8851 duration=1(sec) May 17 01:33:33 client-5 xinetd[3391]: START: shell pid=8942 from=192.168.4.5 May 17 01:33:33 client-5 rshd[8945]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark SKIP: sanity test_51bb needs \>= 2 MDTs");echo XXRETCODE:$?' May 17 01:33:33 client-5 kernel: Lustre: DEBUG MARKER: SKIP: sanity test_51bb needs >= 2 MDTs May 17 01:33:33 client-5 xinetd[3391]: EXIT: shell status=0 pid=8942 duration=0(sec) May 17 01:33:33 client-5 xinetd[3391]: START: shell pid=9035 from=192.168.4.5 May 17 01:33:33 client-5 rshd[9036]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:33:33 client-5 xinetd[3391]: EXIT: shell status=0 pid=9035 duration=0(sec) May 17 01:33:33 client-5 xinetd[3391]: START: shell pid=9208 from=192.168.4.5 May 17 01:33:33 client-5 rshd[9209]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 51c: rmdir ...\/t-0 --- ...\/t-70 ======================================================= 01:33:33 \(1305621213\)");echo XXRETCODE:$?' May 17 01:33:33 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 51c: rmdir .../t-0 --- .../t-70 ======================================================= 01:33:33 (1305621213) May 17 01:33:33 client-5 xinetd[3391]: EXIT: shell status=0 pid=9208 duration=0(sec) May 17 01:33:33 client-5 xinetd[3391]: START: shell pid=9300 from=192.168.4.5 May 17 01:33:33 client-5 rshd[9301]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:33:34 client-5 xinetd[3391]: EXIT: shell status=0 pid=9300 duration=1(sec) May 17 01:33:34 client-5 xinetd[3391]: START: shell pid=9463 from=192.168.4.5 May 17 01:33:34 client-5 rshd[9464]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark SKIP: sanity test_51d skipping ALWAYS excluded test 51d");echo XXRETCODE:$?' May 17 01:33:34 client-5 kernel: Lustre: DEBUG MARKER: SKIP: sanity test_51d skipping ALWAYS excluded test 51d May 17 01:33:34 client-5 xinetd[3391]: EXIT: shell status=0 pid=9463 duration=0(sec) May 17 01:33:34 client-5 xinetd[3391]: START: shell pid=9568 from=192.168.4.5 May 17 01:33:34 client-5 rshd[9569]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 52a: append-only flag test \(should return errors\) ===================================== 01:33:34 \(1305621214\)");echo XXRETCODE:$?' May 17 01:33:34 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 52a: append-only flag test (should return errors) ===================================== 01:33:34 (1305621214) May 17 01:33:34 client-5 xinetd[3391]: EXIT: shell status=0 pid=9568 duration=0(sec) May 17 01:33:34 client-5 xinetd[3391]: START: shell pid=9670 from=192.168.4.5 May 17 01:33:34 client-5 rshd[9672]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:33:34 client-5 xinetd[3391]: EXIT: shell status=0 pid=9670 duration=0(sec) May 17 01:33:35 client-5 xinetd[3391]: START: shell pid=9843 from=192.168.4.5 May 17 01:33:35 client-5 rshd[9844]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 52b: immutable flag test \(should return errors\) ======================================= 01:33:35 \(1305621215\)");echo XXRETCODE:$?' May 17 01:33:35 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 52b: immutable flag test (should return errors) ======================================= 01:33:35 (1305621215) May 17 01:33:35 client-5 xinetd[3391]: EXIT: shell status=0 pid=9843 duration=0(sec) May 17 01:33:35 client-5 xinetd[3391]: START: shell pid=9944 from=192.168.4.5 May 17 01:33:35 client-5 rshd[9947]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:33:35 client-5 xinetd[3391]: EXIT: shell status=0 pid=9944 duration=0(sec) May 17 01:33:35 client-5 xinetd[3391]: START: shell pid=10118 from=192.168.4.5 May 17 01:33:35 client-5 rshd[10119]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 53: verify that MDS and OSTs agree on pre-creation ==================================== 01:33:35 \(1305621215\)");echo XXRETCODE:$?' May 17 01:33:35 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 53: verify that MDS and OSTs agree on pre-creation ==================================== 01:33:35 (1305621215) May 17 01:33:35 client-5 xinetd[3391]: EXIT: shell status=0 pid=10118 duration=0(sec) May 17 01:33:37 client-5 xinetd[3391]: START: shell pid=10505 from=192.168.4.5 May 17 01:33:37 client-5 rshd[10507]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:33:37 client-5 xinetd[3391]: EXIT: shell status=0 pid=10505 duration=0(sec) May 17 01:33:38 client-5 xinetd[3391]: START: shell pid=10678 from=192.168.4.5 May 17 01:33:38 client-5 rshd[10679]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 54a: unix domain socket test ========================================================== 01:33:38 \(1305621218\)");echo XXRETCODE:$?' May 17 01:33:38 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 54a: unix domain socket test ========================================================== 01:33:38 (1305621218) May 17 01:33:38 client-5 xinetd[3391]: EXIT: shell status=0 pid=10678 duration=0(sec) May 17 01:33:38 client-5 xinetd[3391]: START: shell pid=10773 from=192.168.4.5 May 17 01:33:38 client-5 rshd[10774]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:33:38 client-5 xinetd[3391]: EXIT: shell status=0 pid=10773 duration=0(sec) May 17 01:33:38 client-5 xinetd[3391]: START: shell pid=10945 from=192.168.4.5 May 17 01:33:38 client-5 rshd[10946]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 54b: char device works in lustre ====================================================== 01:33:38 \(1305621218\)");echo XXRETCODE:$?' May 17 01:33:38 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 54b: char device works in lustre ====================================================== 01:33:38 (1305621218) May 17 01:33:38 client-5 xinetd[3391]: EXIT: shell status=0 pid=10945 duration=0(sec) May 17 01:33:39 client-5 xinetd[3391]: START: shell pid=11040 from=192.168.4.5 May 17 01:33:39 client-5 rshd[11042]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:33:39 client-5 xinetd[3391]: EXIT: shell status=0 pid=11040 duration=0(sec) May 17 01:33:39 client-5 xinetd[3391]: START: shell pid=11213 from=192.168.4.5 May 17 01:33:39 client-5 rshd[11214]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 54c: block device works in lustre ===================================================== 01:33:39 \(1305621219\)");echo XXRETCODE:$?' May 17 01:33:39 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 54c: block device works in lustre ===================================================== 01:33:39 (1305621219) May 17 01:33:39 client-5 xinetd[3391]: EXIT: shell status=0 pid=11213 duration=0(sec) May 17 01:33:40 client-5 xinetd[3391]: START: shell pid=11326 from=192.168.4.5 May 17 01:33:40 client-5 rshd[11328]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:33:40 client-5 xinetd[3391]: EXIT: shell status=0 pid=11326 duration=0(sec) May 17 01:33:40 client-5 xinetd[3391]: START: shell pid=11499 from=192.168.4.5 May 17 01:33:40 client-5 rshd[11500]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 54d: fifo device works in lustre ====================================================== 01:33:40 \(1305621220\)");echo XXRETCODE:$?' May 17 01:33:40 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 54d: fifo device works in lustre ====================================================== 01:33:40 (1305621220) May 17 01:33:40 client-5 xinetd[3391]: EXIT: shell status=0 pid=11499 duration=0(sec) May 17 01:33:40 client-5 xinetd[3391]: START: shell pid=11594 from=192.168.4.5 May 17 01:33:40 client-5 rshd[11595]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:33:40 client-5 xinetd[3391]: EXIT: shell status=0 pid=11594 duration=0(sec) May 17 01:33:40 client-5 xinetd[3391]: START: shell pid=11766 from=192.168.4.5 May 17 01:33:40 client-5 rshd[11767]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 54e: console\/tty device works in lustre =============================================== 01:33:40 \(1305621220\)");echo XXRETCODE:$?' May 17 01:33:40 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 54e: console/tty device works in lustre =============================================== 01:33:40 (1305621220) May 17 01:33:41 client-5 xinetd[3391]: EXIT: shell status=0 pid=11766 duration=1(sec) May 17 01:33:41 client-5 xinetd[3391]: START: shell pid=11859 from=192.168.4.5 May 17 01:33:41 client-5 rshd[11862]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:33:41 client-5 xinetd[3391]: EXIT: shell status=0 pid=11859 duration=0(sec) May 17 01:33:41 client-5 xinetd[3391]: START: shell pid=12031 from=192.168.4.5 May 17 01:33:41 client-5 rshd[12034]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 56a: check lfs getstripe ============================================================== 01:33:41 \(1305621221\)");echo XXRETCODE:$?' May 17 01:33:41 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 56a: check lfs getstripe ============================================================== 01:33:41 (1305621221) May 17 01:33:41 client-5 xinetd[3391]: EXIT: shell status=0 pid=12031 duration=0(sec) May 17 01:33:41 client-5 xinetd[3391]: START: shell pid=12170 from=192.168.4.5 May 17 01:33:41 client-5 rshd[12171]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:33:41 client-5 xinetd[3391]: EXIT: shell status=0 pid=12170 duration=0(sec) May 17 01:33:42 client-5 xinetd[3391]: START: shell pid=12342 from=192.168.4.5 May 17 01:33:42 client-5 rshd[12343]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 56g: check lfs find -name ============================================================= 01:33:42 \(1305621222\)");echo XXRETCODE:$?' May 17 01:33:42 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 56g: check lfs find -name ============================================================= 01:33:42 (1305621222) May 17 01:33:42 client-5 xinetd[3391]: EXIT: shell status=0 pid=12342 duration=0(sec) May 17 01:33:42 client-5 xinetd[3391]: START: shell pid=12465 from=192.168.4.5 May 17 01:33:42 client-5 rshd[12466]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:33:42 client-5 xinetd[3391]: EXIT: shell status=0 pid=12465 duration=0(sec) May 17 01:33:42 client-5 xinetd[3391]: START: shell pid=12637 from=192.168.4.5 May 17 01:33:42 client-5 rshd[12638]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 56h: check lfs find ! -name =========================================================== 01:33:42 \(1305621222\)");echo XXRETCODE:$?' May 17 01:33:42 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 56h: check lfs find ! -name =========================================================== 01:33:42 (1305621222) May 17 01:33:42 client-5 xinetd[3391]: EXIT: shell status=0 pid=12637 duration=0(sec) May 17 01:33:42 client-5 xinetd[3391]: START: shell pid=12739 from=192.168.4.5 May 17 01:33:42 client-5 rshd[12740]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:33:43 client-5 xinetd[3391]: EXIT: shell status=0 pid=12739 duration=1(sec) May 17 01:33:43 client-5 xinetd[3391]: START: shell pid=12911 from=192.168.4.5 May 17 01:33:43 client-5 rshd[12912]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 56i: check \'lfs find -ost UUID\' skips directories ===================================== 01:33:43 \(1305621223\)");echo XXRETCODE:$?' May 17 01:33:43 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 56i: check 'lfs find -ost UUID' skips directories ===================================== 01:33:43 (1305621223) May 17 01:33:43 client-5 xinetd[3391]: EXIT: shell status=0 pid=12911 duration=0(sec) May 17 01:33:43 client-5 xinetd[3391]: START: shell pid=13007 from=192.168.4.5 May 17 01:33:43 client-5 rshd[13008]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:33:43 client-5 xinetd[3391]: EXIT: shell status=0 pid=13007 duration=0(sec) May 17 01:33:43 client-5 xinetd[3391]: START: shell pid=13179 from=192.168.4.5 May 17 01:33:43 client-5 rshd[13180]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 56j: check lfs find -type d =========================================================== 01:33:43 \(1305621223\)");echo XXRETCODE:$?' May 17 01:33:43 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 56j: check lfs find -type d =========================================================== 01:33:43 (1305621223) May 17 01:33:43 client-5 xinetd[3391]: EXIT: shell status=0 pid=13179 duration=0(sec) May 17 01:33:44 client-5 xinetd[3391]: START: shell pid=13293 from=192.168.4.5 May 17 01:33:44 client-5 rshd[13294]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:33:44 client-5 xinetd[3391]: EXIT: shell status=0 pid=13293 duration=0(sec) May 17 01:33:44 client-5 xinetd[3391]: START: shell pid=13464 from=192.168.4.5 May 17 01:33:44 client-5 rshd[13466]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 56k: check lfs find -type f =========================================================== 01:33:44 \(1305621224\)");echo XXRETCODE:$?' May 17 01:33:44 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 56k: check lfs find -type f =========================================================== 01:33:44 (1305621224) May 17 01:33:44 client-5 xinetd[3391]: EXIT: shell status=0 pid=13464 duration=0(sec) May 17 01:33:44 client-5 xinetd[3391]: START: shell pid=13559 from=192.168.4.5 May 17 01:33:44 client-5 rshd[13560]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:33:44 client-5 xinetd[3391]: EXIT: shell status=0 pid=13559 duration=0(sec) May 17 01:33:44 client-5 xinetd[3391]: START: shell pid=13731 from=192.168.4.5 May 17 01:33:44 client-5 rshd[13732]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 56l: check lfs find -type b =========================================================== 01:33:44 \(1305621224\)");echo XXRETCODE:$?' May 17 01:33:45 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 56l: check lfs find -type b =========================================================== 01:33:44 (1305621224) May 17 01:33:45 client-5 xinetd[3391]: EXIT: shell status=0 pid=13731 duration=1(sec) May 17 01:33:45 client-5 xinetd[3391]: START: shell pid=13825 from=192.168.4.5 May 17 01:33:45 client-5 rshd[13826]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:33:45 client-5 xinetd[3391]: EXIT: shell status=0 pid=13825 duration=0(sec) May 17 01:33:45 client-5 xinetd[3391]: START: shell pid=13997 from=192.168.4.5 May 17 01:33:45 client-5 rshd[13998]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 56m: check lfs find -type c =========================================================== 01:33:45 \(1305621225\)");echo XXRETCODE:$?' May 17 01:33:45 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 56m: check lfs find -type c =========================================================== 01:33:45 (1305621225) May 17 01:33:45 client-5 xinetd[3391]: EXIT: shell status=0 pid=13997 duration=0(sec) May 17 01:33:45 client-5 xinetd[3391]: START: shell pid=14091 from=192.168.4.5 May 17 01:33:45 client-5 rshd[14092]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:33:45 client-5 xinetd[3391]: EXIT: shell status=0 pid=14091 duration=0(sec) May 17 01:33:46 client-5 xinetd[3391]: START: shell pid=14261 from=192.168.4.5 May 17 01:33:46 client-5 rshd[14264]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 56n: check lfs find -type l =========================================================== 01:33:45 \(1305621225\)");echo XXRETCODE:$?' May 17 01:33:46 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 56n: check lfs find -type l =========================================================== 01:33:45 (1305621225) May 17 01:33:46 client-5 xinetd[3391]: EXIT: shell status=0 pid=14261 duration=0(sec) May 17 01:33:46 client-5 kernel: Lustre: Increasing default stripe size to min 1048576 May 17 01:33:46 client-5 xinetd[3391]: START: shell pid=14358 from=192.168.4.5 May 17 01:33:46 client-5 rshd[14360]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:33:46 client-5 xinetd[3391]: EXIT: shell status=0 pid=14358 duration=0(sec) May 17 01:33:46 client-5 xinetd[3391]: START: shell pid=14529 from=192.168.4.5 May 17 01:33:46 client-5 rshd[14532]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 56o: check lfs find -mtime for old files ============================================== 01:33:46 \(1305621226\)");echo XXRETCODE:$?' May 17 01:33:46 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 56o: check lfs find -mtime for old files ============================================== 01:33:46 (1305621226) May 17 01:33:46 client-5 xinetd[3391]: EXIT: shell status=0 pid=14529 duration=0(sec) May 17 01:33:46 client-5 xinetd[3391]: START: shell pid=14630 from=192.168.4.5 May 17 01:33:46 client-5 rshd[14631]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:33:46 client-5 xinetd[3391]: EXIT: shell status=0 pid=14630 duration=0(sec) May 17 01:33:47 client-5 xinetd[3391]: START: shell pid=14802 from=192.168.4.5 May 17 01:33:47 client-5 rshd[14803]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 56p: check lfs find -uid and ! -uid =================================================== 01:33:47 \(1305621227\)");echo XXRETCODE:$?' May 17 01:33:47 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 56p: check lfs find -uid and ! -uid =================================================== 01:33:47 (1305621227) May 17 01:33:47 client-5 xinetd[3391]: EXIT: shell status=0 pid=14802 duration=0(sec) May 17 01:33:47 client-5 xinetd[3391]: START: shell pid=14926 from=192.168.4.5 May 17 01:33:47 client-5 rshd[14927]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:33:47 client-5 xinetd[3391]: EXIT: shell status=0 pid=14926 duration=0(sec) May 17 01:33:47 client-5 xinetd[3391]: START: shell pid=15097 from=192.168.4.5 May 17 01:33:47 client-5 rshd[15099]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 56q: check lfs find -gid and ! -gid =================================================== 01:33:47 \(1305621227\)");echo XXRETCODE:$?' May 17 01:33:48 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 56q: check lfs find -gid and ! -gid =================================================== 01:33:47 (1305621227) May 17 01:33:48 client-5 xinetd[3391]: EXIT: shell status=0 pid=15097 duration=1(sec) May 17 01:33:48 client-5 xinetd[3391]: START: shell pid=15221 from=192.168.4.5 May 17 01:33:48 client-5 rshd[15222]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:33:48 client-5 xinetd[3391]: EXIT: shell status=0 pid=15221 duration=0(sec) May 17 01:33:48 client-5 xinetd[3391]: START: shell pid=15393 from=192.168.4.5 May 17 01:33:48 client-5 rshd[15394]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 56r: check lfs find -size works ======================================================= 01:33:48 \(1305621228\)");echo XXRETCODE:$?' May 17 01:33:48 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 56r: check lfs find -size works ======================================================= 01:33:48 (1305621228) May 17 01:33:48 client-5 xinetd[3391]: EXIT: shell status=0 pid=15393 duration=0(sec) May 17 01:33:48 client-5 xinetd[3391]: START: shell pid=15496 from=192.168.4.5 May 17 01:33:48 client-5 rshd[15497]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark sanity test_56r: @@@@@@ FAIL: lfs find \/mnt\/lustre\/d0.sanity\/d56g -size 5 wrong: found 0, expected 1 ");echo XXRETCODE:$?' May 17 01:33:49 client-5 kernel: Lustre: DEBUG MARKER: sanity test_56r: @@@@@@ FAIL: lfs find /mnt/lustre/d0.sanity/d56g -size 5 wrong: found 0, expected 1 May 17 01:33:49 client-5 xinetd[3391]: EXIT: shell status=0 pid=15496 duration=1(sec) May 17 01:33:49 client-5 xinetd[3391]: START: shell pid=15586 from=192.168.4.5 May 17 01:33:49 client-5 rshd[15588]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl dk > /tmp/test_logs/2011-05-17/001301/sanity.test_56r.debug_log.\$(hostname).1305621229.log; dmesg > /tmp/test_logs/2011-05-17/001301/sanity.test_56r.dmesg.\$(hostname).1305621229.log");echo XXRETCODE:$?' May 17 01:33:49 client-5 xinetd[3391]: EXIT: shell status=0 pid=15586 duration=0(sec) May 17 01:33:49 client-5 xinetd[3391]: START: shell pid=15635 from=192.168.4.5 May 17 01:33:49 client-5 rshd[15638]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "rsync -az /tmp/test_logs/2011-05-17/001301/sanity.test_56r.*.1305621229.log client-5-ib:/tmp/test_logs/2011-05-17/001301");echo XXRETCODE:$?' May 17 01:33:49 client-5 xinetd[3391]: EXIT: shell status=0 pid=15635 duration=0(sec) May 17 01:33:52 client-5 xinetd[3391]: START: shell pid=15756 from=192.168.4.5 May 17 01:33:52 client-5 rshd[15757]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 57a: verify MDS filesystem created with large inodes ================================== 01:33:52 \(1305621232\)");echo XXRETCODE:$?' May 17 01:33:52 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 57a: verify MDS filesystem created with large inodes ================================== 01:33:52 (1305621232) May 17 01:33:52 client-5 xinetd[3391]: EXIT: shell status=0 pid=15756 duration=0(sec) May 17 01:33:52 client-5 xinetd[3391]: START: shell pid=15883 from=192.168.4.5 May 17 01:33:52 client-5 rshd[15884]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:33:52 client-5 xinetd[3391]: EXIT: shell status=0 pid=15883 duration=0(sec) May 17 01:33:53 client-5 xinetd[3391]: START: shell pid=16055 from=192.168.4.5 May 17 01:33:53 client-5 rshd[16056]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 57b: default LOV EAs are stored inside large inodes =================================== 01:33:53 \(1305621233\)");echo XXRETCODE:$?' May 17 01:33:53 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 57b: default LOV EAs are stored inside large inodes =================================== 01:33:53 (1305621233) May 17 01:33:53 client-5 xinetd[3391]: EXIT: shell status=0 pid=16055 duration=0(sec) May 17 01:33:58 client-5 xinetd[3391]: START: shell pid=16299 from=192.168.4.5 May 17 01:33:58 client-5 rshd[16300]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:33:58 client-5 xinetd[3391]: EXIT: shell status=0 pid=16299 duration=0(sec) May 17 01:33:58 client-5 xinetd[3391]: START: shell pid=16471 from=192.168.4.5 May 17 01:33:58 client-5 rshd[16472]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 58: verify cross-platform wire constants ============================================== 01:33:58 \(1305621238\)");echo XXRETCODE:$?' May 17 01:33:58 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 58: verify cross-platform wire constants ============================================== 01:33:58 (1305621238) May 17 01:33:58 client-5 xinetd[3391]: EXIT: shell status=0 pid=16471 duration=0(sec) May 17 01:33:58 client-5 xinetd[3391]: START: shell pid=16565 from=192.168.4.5 May 17 01:33:58 client-5 rshd[16566]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:33:58 client-5 xinetd[3391]: EXIT: shell status=0 pid=16565 duration=0(sec) May 17 01:33:59 client-5 xinetd[3391]: START: shell pid=16737 from=192.168.4.5 May 17 01:33:59 client-5 rshd[16738]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 59: verify cancellation of llog records async ========================================= 01:33:59 \(1305621239\)");echo XXRETCODE:$?' May 17 01:33:59 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 59: verify cancellation of llog records async ========================================= 01:33:59 (1305621239) May 17 01:33:59 client-5 xinetd[3391]: EXIT: shell status=0 pid=16737 duration=0(sec) May 17 01:34:01 client-5 xinetd[3391]: START: shell pid=16832 from=192.168.4.5 May 17 01:34:01 client-5 rshd[16833]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:34:02 client-5 xinetd[3391]: EXIT: shell status=0 pid=16832 duration=1(sec) May 17 01:34:02 client-5 xinetd[3391]: START: shell pid=17004 from=192.168.4.5 May 17 01:34:02 client-5 rshd[17005]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 60a: llog sanity tests run from kernel module ========================================= 01:34:02 \(1305621242\)");echo XXRETCODE:$?' May 17 01:34:02 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 60a: llog sanity tests run from kernel module ========================================= 01:34:02 (1305621242) May 17 01:34:02 client-5 xinetd[3391]: EXIT: shell status=0 pid=17004 duration=0(sec) May 17 01:34:02 client-5 xinetd[3391]: START: shell pid=17095 from=192.168.4.5 May 17 01:34:02 client-5 rshd[17098]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark test_60 run 22840 - from kernel mode");echo XXRETCODE:$?' May 17 01:34:02 client-5 kernel: Lustre: DEBUG MARKER: test_60 run 22840 - from kernel mode May 17 01:34:02 client-5 xinetd[3391]: EXIT: shell status=0 pid=17095 duration=0(sec) May 17 01:34:02 client-5 xinetd[3391]: START: shell pid=17192 from=192.168.4.5 May 17 01:34:02 client-5 rshd[17194]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:34:02 client-5 xinetd[3391]: EXIT: shell status=0 pid=17192 duration=0(sec) May 17 01:34:02 client-5 xinetd[3391]: START: shell pid=17365 from=192.168.4.5 May 17 01:34:02 client-5 rshd[17366]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 60b: limit repeated messages from CERROR\/CWARN ======================================== 01:34:02 \(1305621242\)");echo XXRETCODE:$?' May 17 01:34:03 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 60b: limit repeated messages from CERROR/CWARN ======================================== 01:34:02 (1305621242) May 17 01:34:03 client-5 xinetd[3391]: EXIT: shell status=0 pid=17365 duration=1(sec) May 17 01:34:03 client-5 xinetd[3391]: START: shell pid=17460 from=192.168.4.5 May 17 01:34:03 client-5 rshd[17461]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:34:03 client-5 xinetd[3391]: EXIT: shell status=0 pid=17460 duration=0(sec) May 17 01:34:03 client-5 xinetd[3391]: START: shell pid=17632 from=192.168.4.5 May 17 01:34:03 client-5 rshd[17633]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 60c: unlink file when mds full ======================================================== 01:34:03 \(1305621243\)");echo XXRETCODE:$?' May 17 01:34:03 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 60c: unlink file when mds full ======================================================== 01:34:03 (1305621243) May 17 01:34:03 client-5 xinetd[3391]: EXIT: shell status=0 pid=17632 duration=0(sec) May 17 01:34:28 client-5 xinetd[3391]: START: shell pid=17727 from=192.168.4.5 May 17 01:34:28 client-5 rshd[17728]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:34:28 client-5 xinetd[3391]: EXIT: shell status=0 pid=17727 duration=0(sec) May 17 01:34:28 client-5 xinetd[3391]: START: shell pid=17898 from=192.168.4.5 May 17 01:34:28 client-5 rshd[17900]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 60d: test printk console message masking ============================================== 01:34:28 \(1305621268\)");echo XXRETCODE:$?' May 17 01:34:28 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 60d: test printk console message masking ============================================== 01:34:28 (1305621268) May 17 01:34:28 client-5 xinetd[3391]: EXIT: shell status=0 pid=17898 duration=0(sec) May 17 01:34:28 client-5 kernel: Lustre: DEBUG MARKER: test message ID 8930 6248 May 17 01:34:28 client-5 kernel: Lustre: 17938:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 01:34:28 client-5 kernel: Lustre: 17938:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 2 previous similar messages May 17 01:34:28 client-5 xinetd[3391]: START: shell pid=18001 from=192.168.4.5 May 17 01:34:28 client-5 rshd[18002]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:34:28 client-5 xinetd[3391]: EXIT: shell status=0 pid=18001 duration=0(sec) May 17 01:34:29 client-5 xinetd[3391]: START: shell pid=18173 from=192.168.4.5 May 17 01:34:29 client-5 rshd[18174]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 61: mmap\(\) writes don\'t make sync hang ================================================ 01:34:29 \(1305621269\)");echo XXRETCODE:$?' May 17 01:34:29 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 61: mmap() writes don't make sync hang ================================================ 01:34:29 (1305621269) May 17 01:34:29 client-5 xinetd[3391]: EXIT: shell status=0 pid=18173 duration=0(sec) May 17 01:34:29 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 01:34:29 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 01:34:29 client-5 xinetd[3391]: START: shell pid=18281 from=192.168.4.5 May 17 01:34:29 client-5 rshd[18284]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:34:29 client-5 xinetd[3391]: EXIT: shell status=0 pid=18281 duration=0(sec) May 17 01:34:30 client-5 xinetd[3391]: START: shell pid=18455 from=192.168.4.5 May 17 01:34:30 client-5 rshd[18456]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 63a: Verify oig_wait interruption does not crash ====================================== 01:34:30 \(1305621270\)");echo XXRETCODE:$?' May 17 01:34:30 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 63a: Verify oig_wait interruption does not crash ====================================== 01:34:30 (1305621270) May 17 01:34:30 client-5 xinetd[3391]: EXIT: shell status=0 pid=18455 duration=0(sec) May 17 01:35:30 client-5 xinetd[3391]: START: shell pid=18587 from=192.168.4.5 May 17 01:35:30 client-5 rshd[18588]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:35:30 client-5 xinetd[3391]: EXIT: shell status=0 pid=18587 duration=0(sec) May 17 01:35:30 client-5 xinetd[3391]: START: shell pid=18759 from=192.168.4.5 May 17 01:35:30 client-5 rshd[18760]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 63b: async write errors should be returned to fsync =================================== 01:35:30 \(1305621330\)");echo XXRETCODE:$?' May 17 01:35:30 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 63b: async write errors should be returned to fsync =================================== 01:35:30 (1305621330) May 17 01:35:30 client-5 xinetd[3391]: EXIT: shell status=0 pid=18759 duration=0(sec) May 17 01:35:30 client-5 kernel: Lustre: 18795:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 01:35:30 client-5 kernel: LustreError: 18799:0:(obd_support.h:456:obd_fail_check_set()) *** obd_fail_loc=406 *** May 17 01:35:30 client-5 kernel: LustreError: 18799:0:(osc_request.c:2354:osc_build_req()) prep_req failed: -12 May 17 01:35:30 client-5 kernel: LustreError: 18799:0:(osc_request.c:2760:osc_check_rpcs()) Write request failed with -12 May 17 01:35:33 client-5 xinetd[3391]: START: shell pid=18861 from=192.168.4.5 May 17 01:35:33 client-5 rshd[18862]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl set_param debug=\"trace inode super ext2 malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec\";");echo XXRETCODE:$?' May 17 01:35:33 client-5 xinetd[3391]: EXIT: shell status=0 pid=18861 duration=0(sec) May 17 01:35:33 client-5 xinetd[3391]: START: shell pid=18950 from=192.168.4.5 May 17 01:35:33 client-5 rshd[18953]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:35:33 client-5 xinetd[3391]: EXIT: shell status=0 pid=18950 duration=0(sec) May 17 01:35:34 client-5 xinetd[3391]: START: shell pid=19124 from=192.168.4.5 May 17 01:35:34 client-5 rshd[19125]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 64a: verify filter grant calculations \(in kernel\) ===================================== 01:35:33 \(1305621333\)");echo XXRETCODE:$?' May 17 01:35:34 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 64a: verify filter grant calculations (in kernel) ===================================== 01:35:33 (1305621333) May 17 01:35:34 client-5 xinetd[3391]: EXIT: shell status=0 pid=19124 duration=0(sec) May 17 01:35:34 client-5 xinetd[3391]: START: shell pid=19218 from=192.168.4.5 May 17 01:35:34 client-5 rshd[19219]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:35:34 client-5 xinetd[3391]: EXIT: shell status=0 pid=19218 duration=0(sec) May 17 01:35:34 client-5 xinetd[3391]: START: shell pid=19390 from=192.168.4.5 May 17 01:35:34 client-5 rshd[19391]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 64b: check out-of-space detection on client =========================================== 01:35:34 \(1305621334\)");echo XXRETCODE:$?' May 17 01:35:34 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 64b: check out-of-space detection on client =========================================== 01:35:34 (1305621334) May 17 01:35:34 client-5 xinetd[3391]: EXIT: shell status=0 pid=19390 duration=0(sec) May 17 01:35:35 client-5 xinetd[3391]: START: shell pid=19503 from=192.168.4.5 May 17 01:35:35 client-5 rshd[19504]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark SKIP: sanity test_64b oos.sh: 53523596kB free gt MAXFREE 2400000kB, increase 2400000 \(or reduce test fs size\) to proceed");echo XXRETCODE:$?' May 17 01:35:35 client-5 kernel: Lustre: DEBUG MARKER: SKIP: sanity test_64b oos.sh: 53523596kB free gt MAXFREE 2400000kB, increase 2400000 (or reduce test fs size) to proceed May 17 01:35:35 client-5 xinetd[3391]: EXIT: shell status=0 pid=19503 duration=0(sec) May 17 01:35:35 client-5 xinetd[3391]: START: shell pid=19594 from=192.168.4.5 May 17 01:35:35 client-5 rshd[19595]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:35:36 client-5 xinetd[3391]: EXIT: shell status=0 pid=19594 duration=1(sec) May 17 01:35:36 client-5 xinetd[3391]: START: shell pid=19757 from=192.168.4.5 May 17 01:35:36 client-5 rshd[19758]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark SKIP: sanity test_65a skipping ALWAYS excluded test 65a");echo XXRETCODE:$?' May 17 01:35:36 client-5 kernel: Lustre: DEBUG MARKER: SKIP: sanity test_65a skipping ALWAYS excluded test 65a May 17 01:35:36 client-5 xinetd[3391]: EXIT: shell status=0 pid=19757 duration=0(sec) May 17 01:35:36 client-5 xinetd[3391]: START: shell pid=19862 from=192.168.4.5 May 17 01:35:36 client-5 rshd[19863]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 65b: directory setstripe 2097152 0 1 ================================================== 01:35:36 \(1305621336\)");echo XXRETCODE:$?' May 17 01:35:36 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 65b: directory setstripe 2097152 0 1 ================================================== 01:35:36 (1305621336) May 17 01:35:36 client-5 xinetd[3391]: EXIT: shell status=0 pid=19862 duration=0(sec) May 17 01:35:36 client-5 xinetd[3391]: START: shell pid=19959 from=192.168.4.5 May 17 01:35:36 client-5 rshd[19961]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:35:36 client-5 xinetd[3391]: EXIT: shell status=0 pid=19959 duration=0(sec) May 17 01:35:37 client-5 xinetd[3391]: START: shell pid=20132 from=192.168.4.5 May 17 01:35:37 client-5 rshd[20133]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 65c: directory setstripe 4194304 1 5 ================================================== 01:35:37 \(1305621337\)");echo XXRETCODE:$?' May 17 01:35:37 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 65c: directory setstripe 4194304 1 5 ================================================== 01:35:37 (1305621337) May 17 01:35:37 client-5 xinetd[3391]: EXIT: shell status=0 pid=20132 duration=0(sec) May 17 01:35:37 client-5 xinetd[3391]: START: shell pid=20230 from=192.168.4.5 May 17 01:35:37 client-5 rshd[20231]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:35:37 client-5 xinetd[3391]: EXIT: shell status=0 pid=20230 duration=0(sec) May 17 01:35:37 client-5 xinetd[3391]: START: shell pid=20402 from=192.168.4.5 May 17 01:35:37 client-5 rshd[20403]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 65d: directory setstripe 1048576 -1 stripe_count ====================================== 01:35:37 \(1305621337\)");echo XXRETCODE:$?' May 17 01:35:37 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 65d: directory setstripe 1048576 -1 stripe_count ====================================== 01:35:37 (1305621337) May 17 01:35:37 client-5 xinetd[3391]: EXIT: shell status=0 pid=20402 duration=0(sec) May 17 01:35:37 client-5 xinetd[3391]: START: shell pid=20500 from=192.168.4.5 May 17 01:35:37 client-5 rshd[20501]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:35:37 client-5 xinetd[3391]: EXIT: shell status=0 pid=20500 duration=0(sec) May 17 01:35:38 client-5 xinetd[3391]: START: shell pid=20661 from=192.168.4.5 May 17 01:35:38 client-5 rshd[20664]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark SKIP: sanity test_65e skipping ALWAYS excluded test 65e");echo XXRETCODE:$?' May 17 01:35:38 client-5 kernel: Lustre: DEBUG MARKER: SKIP: sanity test_65e skipping ALWAYS excluded test 65e May 17 01:35:38 client-5 xinetd[3391]: EXIT: shell status=0 pid=20661 duration=0(sec) May 17 01:35:38 client-5 xinetd[3391]: START: shell pid=20767 from=192.168.4.5 May 17 01:35:38 client-5 rshd[20769]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 65f: dir setstripe permission \(should return error\) =================================== 01:35:38 \(1305621338\)");echo XXRETCODE:$?' May 17 01:35:38 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 65f: dir setstripe permission (should return error) =================================== 01:35:38 (1305621338) May 17 01:35:38 client-5 xinetd[3391]: EXIT: shell status=0 pid=20767 duration=0(sec) May 17 01:35:38 client-5 xinetd[3391]: START: shell pid=20861 from=192.168.4.5 May 17 01:35:38 client-5 rshd[20862]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:35:38 client-5 xinetd[3391]: EXIT: shell status=0 pid=20861 duration=0(sec) May 17 01:35:38 client-5 xinetd[3391]: START: shell pid=21033 from=192.168.4.5 May 17 01:35:38 client-5 rshd[21034]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 65g: directory setstripe -d =========================================================== 01:35:38 \(1305621338\)");echo XXRETCODE:$?' May 17 01:35:38 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 65g: directory setstripe -d =========================================================== 01:35:38 (1305621338) May 17 01:35:39 client-5 xinetd[3391]: EXIT: shell status=0 pid=21033 duration=1(sec) May 17 01:35:39 client-5 xinetd[3391]: START: shell pid=21128 from=192.168.4.5 May 17 01:35:39 client-5 rshd[21130]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:35:39 client-5 xinetd[3391]: EXIT: shell status=0 pid=21128 duration=0(sec) May 17 01:35:39 client-5 xinetd[3391]: START: shell pid=21301 from=192.168.4.5 May 17 01:35:39 client-5 rshd[21302]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 65h: directory stripe info inherit ==================================================== 01:35:39 \(1305621339\)");echo XXRETCODE:$?' May 17 01:35:39 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 65h: directory stripe info inherit ==================================================== 01:35:39 (1305621339) May 17 01:35:39 client-5 xinetd[3391]: EXIT: shell status=0 pid=21301 duration=0(sec) May 17 01:35:39 client-5 xinetd[3391]: START: shell pid=21401 from=192.168.4.5 May 17 01:35:39 client-5 rshd[21402]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:35:39 client-5 xinetd[3391]: EXIT: shell status=0 pid=21401 duration=0(sec) May 17 01:35:40 client-5 xinetd[3391]: START: shell pid=21573 from=192.168.4.5 May 17 01:35:40 client-5 rshd[21574]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 65i: set non-default striping on root directory \(bug 6367\)============================= 01:35:39 \(1305621339\)");echo XXRETCODE:$?' May 17 01:35:40 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 65i: set non-default striping on root directory (bug 6367)============================= 01:35:39 (1305621339) May 17 01:35:40 client-5 xinetd[3391]: EXIT: shell status=0 pid=21573 duration=0(sec) May 17 01:35:40 client-5 xinetd[3391]: START: shell pid=21666 from=192.168.4.5 May 17 01:35:40 client-5 rshd[21667]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:35:40 client-5 xinetd[3391]: EXIT: shell status=0 pid=21666 duration=0(sec) May 17 01:35:40 client-5 xinetd[3391]: START: shell pid=21838 from=192.168.4.5 May 17 01:35:40 client-5 rshd[21839]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 65ia: getstripe on -1 default directory striping ====================================== 01:35:40 \(1305621340\)");echo XXRETCODE:$?' May 17 01:35:40 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 65ia: getstripe on -1 default directory striping ====================================== 01:35:40 (1305621340) May 17 01:35:40 client-5 xinetd[3391]: EXIT: shell status=0 pid=21838 duration=0(sec) May 17 01:35:41 client-5 xinetd[3391]: START: shell pid=21928 from=192.168.4.5 May 17 01:35:41 client-5 rshd[21931]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:35:41 client-5 xinetd[3391]: EXIT: shell status=0 pid=21928 duration=0(sec) May 17 01:35:41 client-5 xinetd[3391]: START: shell pid=22102 from=192.168.4.5 May 17 01:35:41 client-5 rshd[22103]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 65ib: getstripe -v on -1 default directory striping =================================== 01:35:41 \(1305621341\)");echo XXRETCODE:$?' May 17 01:35:41 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 65ib: getstripe -v on -1 default directory striping =================================== 01:35:41 (1305621341) May 17 01:35:41 client-5 xinetd[3391]: EXIT: shell status=0 pid=22102 duration=0(sec) May 17 01:35:41 client-5 xinetd[3391]: START: shell pid=22194 from=192.168.4.5 May 17 01:35:41 client-5 rshd[22195]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:35:42 client-5 xinetd[3391]: EXIT: shell status=0 pid=22194 duration=1(sec) May 17 01:35:42 client-5 xinetd[3391]: START: shell pid=22366 from=192.168.4.5 May 17 01:35:42 client-5 rshd[22367]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 65ic: new find on -1 default directory striping ======================================= 01:35:42 \(1305621342\)");echo XXRETCODE:$?' May 17 01:35:42 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 65ic: new find on -1 default directory striping ======================================= 01:35:42 (1305621342) May 17 01:35:42 client-5 xinetd[3391]: EXIT: shell status=0 pid=22366 duration=0(sec) May 17 01:35:43 client-5 xinetd[3391]: START: shell pid=22457 from=192.168.4.5 May 17 01:35:44 client-5 rshd[22459]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:35:44 client-5 xinetd[3391]: EXIT: shell status=0 pid=22457 duration=1(sec) May 17 01:35:44 client-5 xinetd[3391]: START: shell pid=22628 from=192.168.4.5 May 17 01:35:44 client-5 rshd[22631]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 65j: set default striping on root directory \(bug 6367\)================================= 01:35:44 \(1305621344\)");echo XXRETCODE:$?' May 17 01:35:44 client-5 kernel: Lustre: Increasing default stripe size to min 1048576 May 17 01:35:44 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 65j: set default striping on root directory (bug 6367)================================= 01:35:44 (1305621344) May 17 01:35:44 client-5 xinetd[3391]: EXIT: shell status=0 pid=22628 duration=0(sec) May 17 01:35:45 client-5 xinetd[3391]: START: shell pid=22726 from=192.168.4.5 May 17 01:35:45 client-5 rshd[22727]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:35:45 client-5 xinetd[3391]: EXIT: shell status=0 pid=22726 duration=0(sec) May 17 01:35:45 client-5 xinetd[3391]: START: shell pid=22898 from=192.168.4.5 May 17 01:35:45 client-5 rshd[22899]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 65k: validate manual striping works properly with deactivated OSCs ==================== 01:35:45 \(1305621345\)");echo XXRETCODE:$?' May 17 01:35:45 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 65k: validate manual striping works properly with deactivated OSCs ==================== 01:35:45 (1305621345) May 17 01:35:45 client-5 xinetd[3391]: EXIT: shell status=0 pid=22898 duration=0(sec) May 17 01:35:51 client-5 kernel: Lustre: Increasing default stripe size to min 1048576 May 17 01:35:53 client-5 xinetd[3391]: START: shell pid=24132 from=192.168.4.5 May 17 01:35:53 client-5 rshd[24133]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:35:53 client-5 xinetd[3391]: EXIT: shell status=0 pid=24132 duration=0(sec) May 17 01:35:53 client-5 xinetd[3391]: START: shell pid=24304 from=192.168.4.5 May 17 01:35:53 client-5 rshd[24305]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 65l: lfs find on -1 stripe dir ======================================================== 01:35:53 \(1305621353\)");echo XXRETCODE:$?' May 17 01:35:53 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 65l: lfs find on -1 stripe dir ======================================================== 01:35:53 (1305621353) May 17 01:35:53 client-5 xinetd[3391]: EXIT: shell status=0 pid=24304 duration=0(sec) May 17 01:35:53 client-5 xinetd[3391]: START: shell pid=24396 from=192.168.4.5 May 17 01:35:53 client-5 rshd[24399]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:35:53 client-5 xinetd[3391]: EXIT: shell status=0 pid=24396 duration=0(sec) May 17 01:35:54 client-5 xinetd[3391]: START: shell pid=24570 from=192.168.4.5 May 17 01:35:54 client-5 rshd[24571]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 66: update inode blocks count on client =============================================== 01:35:54 \(1305621354\)");echo XXRETCODE:$?' May 17 01:35:54 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 66: update inode blocks count on client =============================================== 01:35:54 (1305621354) May 17 01:35:54 client-5 xinetd[3391]: EXIT: shell status=0 pid=24570 duration=0(sec) May 17 01:35:55 client-5 xinetd[3391]: START: shell pid=24668 from=192.168.4.5 May 17 01:35:55 client-5 rshd[24669]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:35:55 client-5 xinetd[3391]: EXIT: shell status=0 pid=24668 duration=0(sec) May 17 01:35:55 client-5 xinetd[3391]: START: shell pid=24838 from=192.168.4.5 May 17 01:35:55 client-5 rshd[24841]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 68a: lloop driver - basic test ======================================================== 01:35:55 \(1305621355\)");echo XXRETCODE:$?' May 17 01:35:55 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 68a: lloop driver - basic test ======================================================== 01:35:55 (1305621355) May 17 01:35:55 client-5 xinetd[3391]: EXIT: shell status=0 pid=24838 duration=0(sec) May 17 01:35:56 client-5 kernel: Lustre: 24880:0:(lloop.c:711:lloop_ioctl()) Enter llop_ioctl May 17 01:35:57 client-5 xinetd[3391]: START: shell pid=24945 from=192.168.4.5 May 17 01:35:57 client-5 rshd[24946]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:35:57 client-5 xinetd[3391]: EXIT: shell status=0 pid=24945 duration=0(sec) May 17 01:35:58 client-5 xinetd[3391]: START: shell pid=25108 from=192.168.4.5 May 17 01:35:58 client-5 rshd[25109]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark SKIP: sanity test_68b skipping ALWAYS excluded test 68b");echo XXRETCODE:$?' May 17 01:35:58 client-5 kernel: Lustre: DEBUG MARKER: SKIP: sanity test_68b skipping ALWAYS excluded test 68b May 17 01:35:58 client-5 xinetd[3391]: EXIT: shell status=0 pid=25108 duration=0(sec) May 17 01:35:58 client-5 xinetd[3391]: START: shell pid=25213 from=192.168.4.5 May 17 01:35:58 client-5 rshd[25214]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 69: verify oa2dentry return -ENOENT doesn\'t LBUG ====================================== 01:35:58 \(1305621358\)");echo XXRETCODE:$?' May 17 01:35:58 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 69: verify oa2dentry return -ENOENT doesn't LBUG ====================================== 01:35:58 (1305621358) May 17 01:35:58 client-5 xinetd[3391]: EXIT: shell status=0 pid=25213 duration=0(sec) May 17 01:35:58 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.129@o2ib. The ost_write operation failed with -2 May 17 01:35:58 client-5 kernel: LustreError: Skipped 1 previous similar message May 17 01:35:59 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 01:35:59 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 01:35:59 client-5 xinetd[3391]: START: shell pid=25387 from=192.168.4.5 May 17 01:35:59 client-5 rshd[25388]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:36:00 client-5 xinetd[3391]: EXIT: shell status=0 pid=25387 duration=1(sec) May 17 01:36:00 client-5 xinetd[3391]: START: shell pid=25559 from=192.168.4.5 May 17 01:36:00 client-5 rshd[25560]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 71: Running dbench on lustre \(don\'t segment fault\) ==================================== 01:36:00 \(1305621360\)");echo XXRETCODE:$?' May 17 01:36:00 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 71: Running dbench on lustre (don't segment fault) ==================================== 01:36:00 (1305621360) May 17 01:36:00 client-5 xinetd[3391]: EXIT: shell status=0 pid=25559 duration=0(sec) May 17 01:48:05 client-5 xinetd[3391]: START: shell pid=25733 from=192.168.4.5 May 17 01:48:05 client-5 rshd[25734]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:48:05 client-5 xinetd[3391]: EXIT: shell status=0 pid=25733 duration=0(sec) May 17 01:48:05 client-5 xinetd[3391]: START: shell pid=25905 from=192.168.4.5 May 17 01:48:05 client-5 rshd[25906]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 72a: Test that remove suid works properly \(bug5695\) =================================== 01:48:05 \(1305622085\)");echo XXRETCODE:$?' May 17 01:48:06 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 72a: Test that remove suid works properly (bug5695) =================================== 01:48:05 (1305622085) May 17 01:48:06 client-5 xinetd[3391]: EXIT: shell status=0 pid=25905 duration=1(sec) May 17 01:48:09 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc start May 17 01:48:09 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc stop May 17 01:48:09 client-5 xinetd[3391]: START: shell pid=26123 from=192.168.4.5 May 17 01:48:09 client-5 rshd[26124]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:48:09 client-5 xinetd[3391]: EXIT: shell status=0 pid=26123 duration=0(sec) May 17 01:48:09 client-5 xinetd[3391]: START: shell pid=26295 from=192.168.4.5 May 17 01:48:09 client-5 rshd[26296]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 72b: Test that we keep mode setting if without file data changed \(bug 24226\) ========== 01:48:09 \(1305622089\)");echo XXRETCODE:$?' May 17 01:48:09 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 72b: Test that we keep mode setting if without file data changed (bug 24226) ========== 01:48:09 (1305622089) May 17 01:48:09 client-5 xinetd[3391]: EXIT: shell status=0 pid=26295 duration=0(sec) May 17 01:48:09 client-5 xinetd[3391]: START: shell pid=26407 from=192.168.4.5 May 17 01:48:09 client-5 rshd[26409]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:48:10 client-5 xinetd[3391]: EXIT: shell status=0 pid=26407 duration=1(sec) May 17 01:48:10 client-5 xinetd[3391]: START: shell pid=26580 from=192.168.4.5 May 17 01:48:10 client-5 rshd[26581]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 73: multiple MDC requests \(should not deadlock\) ======================================= 01:48:10 \(1305622090\)");echo XXRETCODE:$?' May 17 01:48:10 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 73: multiple MDC requests (should not deadlock) ======================================= 01:48:10 (1305622090) May 17 01:48:10 client-5 xinetd[3391]: EXIT: shell status=0 pid=26580 duration=0(sec) May 17 01:48:36 client-5 xinetd[3391]: START: shell pid=26690 from=192.168.4.5 May 17 01:48:36 client-5 rshd[26691]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:48:36 client-5 xinetd[3391]: EXIT: shell status=0 pid=26690 duration=0(sec) May 17 01:48:36 client-5 xinetd[3391]: START: shell pid=26860 from=192.168.4.5 May 17 01:48:36 client-5 rshd[26863]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 74a: ldlm_enqueue freed-export error path, ls \(shouldn\'t LBUG\) ======================== 01:48:36 \(1305622116\)");echo XXRETCODE:$?' May 17 01:48:36 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 74a: ldlm_enqueue freed-export error path, ls (shouldn't LBUG) ======================== 01:48:36 (1305622116) May 17 01:48:36 client-5 xinetd[3391]: EXIT: shell status=0 pid=26860 duration=0(sec) May 17 01:48:37 client-5 xinetd[3391]: START: shell pid=26958 from=192.168.4.5 May 17 01:48:37 client-5 rshd[26959]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:48:37 client-5 xinetd[3391]: EXIT: shell status=0 pid=26958 duration=0(sec) May 17 01:48:37 client-5 xinetd[3391]: START: shell pid=27129 from=192.168.4.5 May 17 01:48:37 client-5 rshd[27131]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 74b: ldlm_enqueue freed-export error path, touch \(shouldn\'t LBUG\) ===================== 01:48:37 \(1305622117\)");echo XXRETCODE:$?' May 17 01:48:37 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 74b: ldlm_enqueue freed-export error path, touch (shouldn't LBUG) ===================== 01:48:37 (1305622117) May 17 01:48:37 client-5 xinetd[3391]: EXIT: shell status=0 pid=27129 duration=0(sec) May 17 01:48:37 client-5 xinetd[3391]: START: shell pid=27225 from=192.168.4.5 May 17 01:48:37 client-5 rshd[27226]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:48:37 client-5 xinetd[3391]: EXIT: shell status=0 pid=27225 duration=0(sec) May 17 01:48:37 client-5 xinetd[3391]: START: shell pid=27398 from=192.168.4.5 May 17 01:48:37 client-5 rshd[27399]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 74c: ldlm_lock_create error path, \(shouldn\'t LBUG\) ==================================== 01:48:37 \(1305622117\)");echo XXRETCODE:$?' May 17 01:48:38 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 74c: ldlm_lock_create error path, (shouldn't LBUG) ==================================== 01:48:37 (1305622117) May 17 01:48:38 client-5 xinetd[3391]: EXIT: shell status=0 pid=27398 duration=1(sec) May 17 01:48:38 client-5 kernel: LustreError: 27434:0:(obd_support.h:456:obd_fail_check_set()) *** obd_fail_loc=319 *** May 17 01:48:38 client-5 kernel: LustreError: 27434:0:(mdc_locks.c:722:mdc_enqueue()) ldlm_cli_enqueue: -12 May 17 01:48:38 client-5 xinetd[3391]: START: shell pid=27491 from=192.168.4.5 May 17 01:48:38 client-5 rshd[27492]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:48:38 client-5 xinetd[3391]: EXIT: shell status=0 pid=27491 duration=0(sec) May 17 01:48:38 client-5 xinetd[3391]: START: shell pid=27663 from=192.168.4.5 May 17 01:48:38 client-5 rshd[27664]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 76: confirm clients recycle inodes properly =========================================== 01:48:38 \(1305622118\)");echo XXRETCODE:$?' May 17 01:48:38 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 76: confirm clients recycle inodes properly =========================================== 01:48:38 (1305622118) May 17 01:48:38 client-5 xinetd[3391]: EXIT: shell status=0 pid=27663 duration=0(sec) May 17 01:48:38 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 01:48:38 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 01:48:49 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 01:48:49 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 01:48:50 client-5 xinetd[3391]: START: shell pid=29790 from=192.168.4.5 May 17 01:48:50 client-5 rshd[29791]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:48:50 client-5 xinetd[3391]: EXIT: shell status=0 pid=29790 duration=0(sec) May 17 01:48:50 client-5 xinetd[3391]: START: shell pid=29962 from=192.168.4.5 May 17 01:48:50 client-5 rshd[29963]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 77a: normal checksum read\/write operation ============================================= 01:48:50 \(1305622130\)");echo XXRETCODE:$?' May 17 01:48:50 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 77a: normal checksum read/write operation ============================================= 01:48:50 (1305622130) May 17 01:48:50 client-5 xinetd[3391]: EXIT: shell status=0 pid=29962 duration=0(sec) May 17 01:48:52 client-5 xinetd[3391]: START: shell pid=30061 from=192.168.4.5 May 17 01:48:52 client-5 rshd[30062]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:48:52 client-5 xinetd[3391]: EXIT: shell status=0 pid=30061 duration=0(sec) May 17 01:48:52 client-5 xinetd[3391]: START: shell pid=30233 from=192.168.4.5 May 17 01:48:52 client-5 rshd[30234]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 77b: checksum error on client write =================================================== 01:48:52 \(1305622132\)");echo XXRETCODE:$?' May 17 01:48:53 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 77b: checksum error on client write =================================================== 01:48:52 (1305622132) May 17 01:48:53 client-5 xinetd[3391]: EXIT: shell status=0 pid=30233 duration=1(sec) May 17 01:48:53 client-5 kernel: LustreError: 30273:0:(obd_support.h:456:obd_fail_check_set()) *** obd_fail_loc=409 *** May 17 01:48:53 client-5 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed on the client after we checksummed it - likely false positive due to mmap IO (bug 11742): from 192.168.4.129@o2ib inode [0x200000401:0x2144:0x0] object 4860/0 extent [0-1048575] May 17 01:48:53 client-5 kernel: LustreError: 4121:0:(osc_request.c:1510:check_write_checksum()) original client csum bfae93d4 (type 2), server csum bfae93d3 (type 2), client csum now bfae93d3 May 17 01:48:53 client-5 kernel: LustreError: 4121:0:(osc_request.c:1741:osc_brw_redo_request()) @@@ redo for recoverable error req@ffff81031e462400 x1369038007675019/t7507(7507) o-1->lustre-OST0005_UUID@192.168.4.129@o2ib:6/4 lens 456/416 e 0 to 0 dl 1305622140 ref 3 fl Interpret:R/ffffffff/ffffffff rc 0/-1 May 17 01:48:53 client-5 xinetd[3391]: START: shell pid=30332 from=192.168.4.5 May 17 01:48:53 client-5 rshd[30333]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:48:53 client-5 xinetd[3391]: EXIT: shell status=0 pid=30332 duration=0(sec) May 17 01:48:54 client-5 xinetd[3391]: START: shell pid=30504 from=192.168.4.5 May 17 01:48:54 client-5 rshd[30505]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 77c: checksum error on client read ==================================================== 01:48:54 \(1305622134\)");echo XXRETCODE:$?' May 17 01:48:54 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 77c: checksum error on client read ==================================================== 01:48:54 (1305622134) May 17 01:48:54 client-5 xinetd[3391]: EXIT: shell status=0 pid=30504 duration=0(sec) May 17 01:48:54 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 01:48:54 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 01:48:54 client-5 xinetd[3391]: START: shell pid=30620 from=192.168.4.5 May 17 01:48:54 client-5 rshd[30621]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark set checksum type to crc32");echo XXRETCODE:$?' May 17 01:48:54 client-5 kernel: Lustre: DEBUG MARKER: set checksum type to crc32 May 17 01:48:54 client-5 xinetd[3391]: EXIT: shell status=0 pid=30620 duration=0(sec) May 17 01:48:54 client-5 kernel: LustreError: 4121:0:(obd_support.h:456:obd_fail_check_set()) *** obd_fail_loc=408 *** May 17 01:48:54 client-5 kernel: LustreError: 133-1: lustre-OST0005-osc-ffff81033c45d000: BAD READ CHECKSUM: from 192.168.4.129@o2ib inode [0x0:0x0:0x0] object 4860/0 extent [0-1048575] May 17 01:48:54 client-5 kernel: LustreError: 4121:0:(osc_request.c:1652:osc_brw_fini_request()) client c5e3c536, server b8cfaf76, cksum_type 1 May 17 01:48:54 client-5 kernel: LustreError: 4121:0:(osc_request.c:1741:osc_brw_redo_request()) @@@ redo for recoverable error req@ffff810316ad7800 x1369038007675033/t0(0) o-1->lustre-OST0005_UUID@192.168.4.129@o2ib:6/4 lens 456/400 e 0 to 0 dl 1305622141 ref 2 fl Interpret:R/ffffffff/ffffffff rc 1048576/-1 May 17 01:49:01 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 01:49:01 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 01:49:01 client-5 xinetd[3391]: START: shell pid=30731 from=192.168.4.5 May 17 01:49:01 client-5 rshd[30732]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark set checksum type to adler");echo XXRETCODE:$?' May 17 01:49:01 client-5 kernel: Lustre: DEBUG MARKER: set checksum type to adler May 17 01:49:01 client-5 xinetd[3391]: EXIT: shell status=0 pid=30731 duration=0(sec) May 17 01:49:01 client-5 kernel: LustreError: 4121:0:(obd_support.h:456:obd_fail_check_set()) *** obd_fail_loc=408 *** May 17 01:49:01 client-5 kernel: LustreError: 133-1: lustre-OST0005-osc-ffff81033c45d000: BAD READ CHECKSUM: from 192.168.4.129@o2ib inode [0x0:0x0:0x0] object 4860/0 extent [0-1048575] May 17 01:49:01 client-5 kernel: LustreError: 4121:0:(osc_request.c:1652:osc_brw_fini_request()) client e10193f6, server bfae93d3, cksum_type 2 May 17 01:49:01 client-5 kernel: LustreError: 4121:0:(osc_request.c:1741:osc_brw_redo_request()) @@@ redo for recoverable error req@ffff81031257d000 x1369038007675051/t0(0) o-1->lustre-OST0005_UUID@192.168.4.129@o2ib:6/4 lens 456/400 e 0 to 0 dl 1305622148 ref 2 fl Interpret:R/ffffffff/ffffffff rc 1048576/-1 May 17 01:49:08 client-5 xinetd[3391]: START: shell pid=30829 from=192.168.4.5 May 17 01:49:08 client-5 rshd[30830]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark set checksum type to adler");echo XXRETCODE:$?' May 17 01:49:08 client-5 kernel: Lustre: DEBUG MARKER: set checksum type to adler May 17 01:49:08 client-5 xinetd[3391]: EXIT: shell status=0 pid=30829 duration=0(sec) May 17 01:49:08 client-5 xinetd[3391]: START: shell pid=30921 from=192.168.4.5 May 17 01:49:08 client-5 rshd[30922]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:49:09 client-5 xinetd[3391]: EXIT: shell status=0 pid=30921 duration=1(sec) May 17 01:49:09 client-5 xinetd[3391]: START: shell pid=31093 from=192.168.4.5 May 17 01:49:09 client-5 rshd[31094]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 77d: checksum error on OST direct write =============================================== 01:49:09 \(1305622149\)");echo XXRETCODE:$?' May 17 01:49:09 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 77d: checksum error on OST direct write =============================================== 01:49:09 (1305622149) May 17 01:49:09 client-5 xinetd[3391]: EXIT: shell status=0 pid=31093 duration=0(sec) May 17 01:49:09 client-5 kernel: LustreError: 31133:0:(obd_support.h:456:obd_fail_check_set()) *** obd_fail_loc=409 *** May 17 01:49:09 client-5 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed on the client after we checksummed it - likely false positive due to mmap IO (bug 11742): from 192.168.4.129@o2ib inode [0x200000401:0x2145:0x0] object 4835/0 extent [0-1048575] May 17 01:49:09 client-5 kernel: LustreError: 4121:0:(osc_request.c:1510:check_write_checksum()) original client csum 19eeae62 (type 2), server csum 19eeae61 (type 2), client csum now 19eeae61 May 17 01:49:09 client-5 kernel: LustreError: 4121:0:(osc_request.c:1741:osc_brw_redo_request()) @@@ redo for recoverable error req@ffff81031fa7a400 x1369038007675079/t8331(8331) o-1->lustre-OST0000_UUID@192.168.4.129@o2ib:6/4 lens 456/416 e 0 to 0 dl 1305622156 ref 2 fl Interpret:R/ffffffff/ffffffff rc 0/-1 May 17 01:49:16 client-5 xinetd[3391]: START: shell pid=31191 from=192.168.4.5 May 17 01:49:16 client-5 rshd[31193]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:49:16 client-5 xinetd[3391]: EXIT: shell status=0 pid=31191 duration=0(sec) May 17 01:49:16 client-5 xinetd[3391]: START: shell pid=31362 from=192.168.4.5 May 17 01:49:16 client-5 rshd[31365]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 77e: checksum error on OST direct read ================================================ 01:49:16 \(1305622156\)");echo XXRETCODE:$?' May 17 01:49:16 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 77e: checksum error on OST direct read ================================================ 01:49:16 (1305622156) May 17 01:49:16 client-5 xinetd[3391]: EXIT: shell status=0 pid=31362 duration=0(sec) May 17 01:49:16 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 01:49:16 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 01:49:16 client-5 kernel: LustreError: 4121:0:(obd_support.h:456:obd_fail_check_set()) *** obd_fail_loc=408 *** May 17 01:49:16 client-5 kernel: LustreError: 133-1: lustre-OST0000-osc-ffff81033c45d000: BAD READ CHECKSUM: from 192.168.4.129@o2ib inode [0x0:0x0:0x0] object 4835/0 extent [0-1048575] May 17 01:49:16 client-5 kernel: LustreError: 4121:0:(osc_request.c:1652:osc_brw_fini_request()) client a570acd1, server 19eeae61, cksum_type 2 May 17 01:49:16 client-5 kernel: LustreError: 4121:0:(osc_request.c:1741:osc_brw_redo_request()) @@@ redo for recoverable error req@ffff81031013bc00 x1369038007675098/t0(0) o-1->lustre-OST0000_UUID@192.168.4.129@o2ib:6/4 lens 456/400 e 0 to 0 dl 1305622163 ref 2 fl Interpret:R/ffffffff/ffffffff rc 1048576/-1 May 17 01:49:18 client-5 xinetd[3391]: START: shell pid=31477 from=192.168.4.5 May 17 01:49:18 client-5 rshd[31478]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:49:18 client-5 xinetd[3391]: EXIT: shell status=0 pid=31477 duration=0(sec) May 17 01:49:18 client-5 xinetd[3391]: START: shell pid=31647 from=192.168.4.5 May 17 01:49:18 client-5 rshd[31650]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 77f: repeat checksum error on write \(expect error\) ==================================== 01:49:18 \(1305622158\)");echo XXRETCODE:$?' May 17 01:49:18 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 77f: repeat checksum error on write (expect error) ==================================== 01:49:18 (1305622158) May 17 01:49:18 client-5 xinetd[3391]: EXIT: shell status=0 pid=31647 duration=0(sec) May 17 01:49:18 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 01:49:18 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 01:49:18 client-5 xinetd[3391]: START: shell pid=31765 from=192.168.4.5 May 17 01:49:18 client-5 rshd[31766]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark set checksum type to crc32");echo XXRETCODE:$?' May 17 01:49:18 client-5 kernel: Lustre: DEBUG MARKER: set checksum type to crc32 May 17 01:49:18 client-5 xinetd[3391]: EXIT: shell status=0 pid=31765 duration=0(sec) May 17 01:49:19 client-5 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 192.168.4.129@o2ib inode [0x200000401:0x2145:0x0] object 4835/0 extent [2097152-3145727] May 17 01:49:19 client-5 kernel: LustreError: 4121:0:(osc_request.c:1510:check_write_checksum()) original client csum b2f1b12 (type 1), server csum b2f1b11 (type 1), client csum now b2f1b12 May 17 01:49:25 client-5 kernel: LustreError: 4121:0:(obd_support.h:456:obd_fail_check_set()) *** obd_fail_loc=409 *** May 17 01:49:25 client-5 kernel: LustreError: 4121:0:(obd_support.h:456:obd_fail_check_set()) Skipped 24 previous similar messages May 17 01:49:25 client-5 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 192.168.4.129@o2ib inode [0x200000401:0x2145:0x0] object 4835/0 extent [3145728-4194303] May 17 01:49:25 client-5 kernel: LustreError: Skipped 7 previous similar messages May 17 01:49:25 client-5 kernel: LustreError: 4121:0:(osc_request.c:1510:check_write_checksum()) original client csum b2f1b12 (type 1), server csum b2f1b11 (type 1), client csum now b2f1b12 May 17 01:49:25 client-5 kernel: LustreError: 4121:0:(osc_request.c:1510:check_write_checksum()) Skipped 7 previous similar messages May 17 01:49:25 client-5 kernel: LustreError: 4121:0:(osc_request.c:1741:osc_brw_redo_request()) @@@ redo for recoverable error req@ffff81031fa7ac00 x1369038007675126/t8346(8346) o-1->lustre-OST0000_UUID@192.168.4.129@o2ib:6/4 lens 456/416 e 0 to 0 dl 1305622172 ref 2 fl Interpret:R/ffffffff/ffffffff rc 0/-1 May 17 01:49:25 client-5 kernel: LustreError: 4121:0:(osc_request.c:1741:osc_brw_redo_request()) Skipped 8 previous similar messages May 17 01:49:27 client-5 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 192.168.4.129@o2ib inode [0x200000401:0x2145:0x0] object 4835/0 extent [0-1048575] May 17 01:49:27 client-5 kernel: LustreError: Skipped 7 previous similar messages May 17 01:49:27 client-5 kernel: LustreError: 4121:0:(osc_request.c:1510:check_write_checksum()) original client csum b2f1b12 (type 1), server csum b2f1b11 (type 1), client csum now b2f1b12 May 17 01:49:27 client-5 kernel: LustreError: 4121:0:(osc_request.c:1510:check_write_checksum()) Skipped 7 previous similar messages May 17 01:49:34 client-5 kernel: LustreError: 4121:0:(obd_support.h:456:obd_fail_check_set()) *** obd_fail_loc=409 *** May 17 01:49:34 client-5 kernel: LustreError: 4121:0:(obd_support.h:456:obd_fail_check_set()) Skipped 47 previous similar messages May 17 01:49:34 client-5 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 192.168.4.129@o2ib inode [0x200000401:0x2145:0x0] object 4835/0 extent [5242880-6291455] May 17 01:49:34 client-5 kernel: LustreError: Skipped 15 previous similar messages May 17 01:49:34 client-5 kernel: LustreError: 4121:0:(osc_request.c:1510:check_write_checksum()) original client csum b2f1b12 (type 1), server csum b2f1b11 (type 1), client csum now b2f1b12 May 17 01:49:34 client-5 kernel: LustreError: 4121:0:(osc_request.c:1510:check_write_checksum()) Skipped 15 previous similar messages May 17 01:49:34 client-5 kernel: LustreError: 4121:0:(osc_request.c:1741:osc_brw_redo_request()) @@@ redo for recoverable error req@ffff81031fa79000 x1369038007675164/t8370(8370) o-1->lustre-OST0000_UUID@192.168.4.129@o2ib:6/4 lens 456/416 e 0 to 0 dl 1305622181 ref 2 fl Interpret:R/ffffffff/ffffffff rc 0/-1 May 17 01:49:34 client-5 kernel: LustreError: 4121:0:(osc_request.c:1741:osc_brw_redo_request()) Skipped 23 previous similar messages May 17 01:49:45 client-5 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 192.168.4.129@o2ib inode [0x200000401:0x2145:0x0] object 4835/0 extent [3145728-4194303] May 17 01:49:45 client-5 kernel: LustreError: Skipped 15 previous similar messages May 17 01:49:46 client-5 kernel: LustreError: 4121:0:(osc_request.c:1510:check_write_checksum()) original client csum b2f1b12 (type 1), server csum b2f1b11 (type 1), client csum now b2f1b12 May 17 01:49:46 client-5 kernel: LustreError: 4121:0:(osc_request.c:1510:check_write_checksum()) Skipped 15 previous similar messages May 17 01:49:53 client-5 kernel: LustreError: 4121:0:(obd_support.h:456:obd_fail_check_set()) *** obd_fail_loc=409 *** May 17 01:49:53 client-5 kernel: LustreError: 4121:0:(obd_support.h:456:obd_fail_check_set()) Skipped 47 previous similar messages May 17 01:49:53 client-5 kernel: LustreError: 4121:0:(osc_request.c:1741:osc_brw_redo_request()) @@@ redo for recoverable error req@ffff81031fa79c00 x1369038007675207/t8394(8394) o-1->lustre-OST0000_UUID@192.168.4.129@o2ib:6/4 lens 456/416 e 0 to 0 dl 1305622200 ref 2 fl Interpret:R/ffffffff/ffffffff rc 0/-1 May 17 01:49:53 client-5 kernel: LustreError: 4121:0:(osc_request.c:1741:osc_brw_redo_request()) Skipped 23 previous similar messages May 17 01:50:10 client-5 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 192.168.4.129@o2ib inode [0x200000401:0x2145:0x0] object 4835/0 extent [6291456-7340031] May 17 01:50:10 client-5 kernel: LustreError: Skipped 23 previous similar messages May 17 01:50:10 client-5 kernel: LustreError: 4121:0:(osc_request.c:1510:check_write_checksum()) original client csum b2f1b12 (type 1), server csum b2f1b11 (type 1), client csum now b2f1b12 May 17 01:50:10 client-5 kernel: LustreError: 4121:0:(osc_request.c:1510:check_write_checksum()) Skipped 23 previous similar messages May 17 01:50:20 client-5 kernel: LustreError: 4121:0:(osc_request.c:1737:osc_brw_redo_request()) too many resent retries, returning error May 17 01:50:20 client-5 kernel: LustreError: 4121:0:(osc_request.c:1737:osc_brw_redo_request()) too many resent retries, returning error May 17 01:50:20 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 01:50:20 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 01:50:20 client-5 xinetd[3391]: START: shell pid=31880 from=192.168.4.5 May 17 01:50:20 client-5 rshd[31881]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark set checksum type to adler");echo XXRETCODE:$?' May 17 01:50:20 client-5 kernel: Lustre: DEBUG MARKER: set checksum type to adler May 17 01:50:20 client-5 xinetd[3391]: EXIT: shell status=0 pid=31880 duration=0(sec) May 17 01:50:27 client-5 kernel: LustreError: 4121:0:(obd_support.h:456:obd_fail_check_set()) *** obd_fail_loc=409 *** May 17 01:50:27 client-5 kernel: LustreError: 4121:0:(obd_support.h:456:obd_fail_check_set()) Skipped 111 previous similar messages May 17 01:50:27 client-5 kernel: LustreError: 4121:0:(osc_request.c:1741:osc_brw_redo_request()) @@@ redo for recoverable error req@ffff81031fa79000 x1369038007675325/t8450(8450) o-1->lustre-OST0000_UUID@192.168.4.129@o2ib:6/4 lens 456/416 e 0 to 0 dl 1305622234 ref 2 fl Interpret:R/ffffffff/ffffffff rc 0/-1 May 17 01:50:27 client-5 kernel: LustreError: 4121:0:(osc_request.c:1741:osc_brw_redo_request()) Skipped 47 previous similar messages May 17 01:50:42 client-5 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 192.168.4.129@o2ib inode [0x200000401:0x2145:0x0] object 4835/0 extent [1048576-2097151] May 17 01:50:42 client-5 kernel: LustreError: Skipped 63 previous similar messages May 17 01:50:42 client-5 kernel: LustreError: 4121:0:(osc_request.c:1510:check_write_checksum()) original client csum 19eeae62 (type 2), server csum 19eeae61 (type 2), client csum now 19eeae62 May 17 01:50:42 client-5 kernel: LustreError: 4121:0:(osc_request.c:1510:check_write_checksum()) Skipped 63 previous similar messages May 17 01:51:16 client-5 kernel: LustreError: 4121:0:(osc_request.c:1737:osc_brw_redo_request()) too many resent retries, returning error May 17 01:51:16 client-5 kernel: LustreError: 4121:0:(osc_request.c:1737:osc_brw_redo_request()) Skipped 6 previous similar messages May 17 01:51:17 client-5 xinetd[3391]: START: shell pid=31981 from=192.168.4.5 May 17 01:51:17 client-5 rshd[31982]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark set checksum type to adler");echo XXRETCODE:$?' May 17 01:51:17 client-5 kernel: Lustre: DEBUG MARKER: set checksum type to adler May 17 01:51:17 client-5 xinetd[3391]: EXIT: shell status=0 pid=31981 duration=0(sec) May 17 01:51:17 client-5 xinetd[3391]: START: shell pid=32073 from=192.168.4.5 May 17 01:51:17 client-5 rshd[32074]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:51:17 client-5 xinetd[3391]: EXIT: shell status=0 pid=32073 duration=0(sec) May 17 01:51:17 client-5 xinetd[3391]: START: shell pid=32243 from=192.168.4.5 May 17 01:51:17 client-5 rshd[32246]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 77g: checksum error on OST write ====================================================== 01:51:17 \(1305622277\)");echo XXRETCODE:$?' May 17 01:51:17 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 77g: checksum error on OST write ====================================================== 01:51:17 (1305622277) May 17 01:51:17 client-5 xinetd[3391]: EXIT: shell status=0 pid=32243 duration=0(sec) May 17 01:51:24 client-5 xinetd[3391]: START: shell pid=32382 from=192.168.4.5 May 17 01:51:24 client-5 rshd[32385]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:51:24 client-5 xinetd[3391]: EXIT: shell status=0 pid=32382 duration=0(sec) May 17 01:51:24 client-5 xinetd[3391]: START: shell pid=32556 from=192.168.4.5 May 17 01:51:24 client-5 rshd[32557]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 77h: checksum error on OST read ======================================================= 01:51:24 \(1305622284\)");echo XXRETCODE:$?' May 17 01:51:24 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 77h: checksum error on OST read ======================================================= 01:51:24 (1305622284) May 17 01:51:24 client-5 xinetd[3391]: EXIT: shell status=0 pid=32556 duration=0(sec) May 17 01:51:25 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 01:51:25 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 01:51:25 client-5 kernel: LustreError: 133-1: lustre-OST0000-osc-ffff81033c45d000: BAD READ CHECKSUM: from 192.168.4.129@o2ib inode [0x0:0x0:0x0] object 4836/0 extent [0-1048575] May 17 01:51:25 client-5 kernel: LustreError: 4121:0:(osc_request.c:1652:osc_brw_fini_request()) client e3c893f9, server bfae93d3, cksum_type 2 May 17 01:51:32 client-5 xinetd[3391]: START: shell pid=32707 from=192.168.4.5 May 17 01:51:32 client-5 rshd[32709]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:51:32 client-5 xinetd[3391]: EXIT: shell status=0 pid=32707 duration=0(sec) May 17 01:51:32 client-5 xinetd[3391]: START: shell pid=420 from=192.168.4.5 May 17 01:51:32 client-5 rshd[423]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 77i: client not supporting OSD_CONNECT_CKSUM ========================================== 01:51:32 \(1305622292\)");echo XXRETCODE:$?' May 17 01:51:32 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 77i: client not supporting OSD_CONNECT_CKSUM ========================================== 01:51:32 (1305622292) May 17 01:51:32 client-5 xinetd[3391]: EXIT: shell status=0 pid=420 duration=0(sec) May 17 01:51:33 client-5 kernel: LustreError: 476:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 01:51:33 client-5 kernel: LustreError: 476:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Skipped 1 previous similar message May 17 01:51:33 client-5 kernel: LustreError: 476:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 01:51:33 client-5 kernel: LustreError: 476:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) Skipped 1 previous similar message May 17 01:51:33 client-5 kernel: Lustre: client ffff81033c45d000 umount complete May 17 01:51:33 client-5 kernel: Lustre: 486:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 01:51:33 client-5 kernel: Lustre: 486:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 5 previous similar messages May 17 01:51:33 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 01:51:33 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 01:51:33 client-5 kernel: Lustre: Increasing default stripe size to min 1048576 May 17 01:51:33 client-5 kernel: LustreError: 486:0:(obd_support.h:456:obd_fail_check_set()) *** obd_fail_loc=40b *** May 17 01:51:33 client-5 kernel: Lustre: Client lustre-client has started May 17 01:51:33 client-5 kernel: Lustre: 503:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 01:51:33 client-5 kernel: LustreError: 563:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 01:51:33 client-5 kernel: LustreError: 563:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Skipped 3 previous similar messages May 17 01:51:33 client-5 kernel: LustreError: 563:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 01:51:33 client-5 kernel: LustreError: 563:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) Skipped 3 previous similar messages May 17 01:51:33 client-5 kernel: Lustre: client ffff81033c45d400 umount complete May 17 01:51:33 client-5 kernel: Lustre: 573:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 01:51:33 client-5 kernel: Lustre: 573:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 7 previous similar messages May 17 01:51:33 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 01:51:33 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 01:51:33 client-5 kernel: Lustre: Increasing default stripe size to min 1048576 May 17 01:51:33 client-5 kernel: Lustre: Client lustre-client has started May 17 01:51:34 client-5 kernel: Lustre: 590:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 01:51:34 client-5 kernel: Lustre: 590:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message May 17 01:51:34 client-5 xinetd[3391]: START: shell pid=662 from=192.168.4.5 May 17 01:51:34 client-5 rshd[663]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:51:34 client-5 xinetd[3391]: EXIT: shell status=0 pid=662 duration=0(sec) May 17 01:51:34 client-5 xinetd[3391]: START: shell pid=834 from=192.168.4.5 May 17 01:51:34 client-5 rshd[837]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 77j: client only supporting ADLER32 =================================================== 01:51:34 \(1305622294\)");echo XXRETCODE:$?' May 17 01:51:34 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 77j: client only supporting ADLER32 =================================================== 01:51:34 (1305622294) May 17 01:51:34 client-5 xinetd[3391]: EXIT: shell status=0 pid=834 duration=0(sec) May 17 01:51:34 client-5 kernel: LustreError: 890:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 01:51:34 client-5 kernel: LustreError: 890:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Skipped 1 previous similar message May 17 01:51:34 client-5 kernel: LustreError: 890:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 01:51:34 client-5 kernel: LustreError: 890:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) Skipped 1 previous similar message May 17 01:51:34 client-5 kernel: Lustre: client ffff81031a2c0c00 umount complete May 17 01:51:34 client-5 kernel: Lustre: 900:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 01:51:34 client-5 kernel: Lustre: 900:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 7 previous similar messages May 17 01:51:34 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 01:51:35 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 01:51:35 client-5 kernel: Lustre: Increasing default stripe size to min 1048576 May 17 01:51:35 client-5 kernel: LustreError: 900:0:(obd_support.h:456:obd_fail_check_set()) *** obd_fail_loc=40c *** May 17 01:51:35 client-5 kernel: Lustre: Client lustre-client has started May 17 01:51:35 client-5 kernel: Lustre: 917:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 01:51:35 client-5 kernel: Lustre: 917:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message May 17 01:51:37 client-5 kernel: LustreError: 977:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 01:51:37 client-5 kernel: LustreError: 977:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Skipped 1 previous similar message May 17 01:51:37 client-5 kernel: LustreError: 977:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 01:51:37 client-5 kernel: LustreError: 977:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) Skipped 1 previous similar message May 17 01:51:37 client-5 kernel: Lustre: client ffff81031f807c00 umount complete May 17 01:51:37 client-5 kernel: Lustre: 987:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 01:51:37 client-5 kernel: Lustre: 987:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 7 previous similar messages May 17 01:51:37 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 01:51:37 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 01:51:37 client-5 kernel: Lustre: Increasing default stripe size to min 1048576 May 17 01:51:37 client-5 kernel: Lustre: Client lustre-client has started May 17 01:51:37 client-5 kernel: Lustre: 1004:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 01:51:37 client-5 kernel: Lustre: 1004:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message May 17 01:51:37 client-5 xinetd[3391]: START: shell pid=1063 from=192.168.4.5 May 17 01:51:37 client-5 rshd[1064]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:51:37 client-5 xinetd[3391]: EXIT: shell status=0 pid=1063 duration=0(sec) May 17 01:51:38 client-5 xinetd[3391]: START: shell pid=1235 from=192.168.4.5 May 17 01:51:38 client-5 rshd[1237]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 78: handle large O_DIRECT writes correctly ============================================ 01:51:38 \(1305622298\)");echo XXRETCODE:$?' May 17 01:51:38 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 78: handle large O_DIRECT writes correctly ============================================ 01:51:38 (1305622298) May 17 01:51:38 client-5 xinetd[3391]: EXIT: shell status=0 pid=1235 duration=0(sec) May 17 01:52:04 client-5 xinetd[3391]: START: shell pid=1368 from=192.168.4.5 May 17 01:52:04 client-5 rshd[1369]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:52:04 client-5 xinetd[3391]: EXIT: shell status=0 pid=1368 duration=0(sec) May 17 01:52:04 client-5 xinetd[3391]: START: shell pid=1541 from=192.168.4.5 May 17 01:52:04 client-5 rshd[1542]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 79: df report consistency check ======================================================= 01:52:04 \(1305622324\)");echo XXRETCODE:$?' May 17 01:52:04 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 79: df report consistency check ======================================================= 01:52:04 (1305622324) May 17 01:52:04 client-5 xinetd[3391]: EXIT: shell status=0 pid=1541 duration=0(sec) May 17 01:52:05 client-5 xinetd[3391]: START: shell pid=1667 from=192.168.4.5 May 17 01:52:05 client-5 rshd[1668]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:52:05 client-5 xinetd[3391]: EXIT: shell status=0 pid=1667 duration=0(sec) May 17 01:52:05 client-5 xinetd[3391]: START: shell pid=1842 from=192.168.4.5 May 17 01:52:05 client-5 rshd[1843]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 80: Page eviction is equally fast at high offsets too ================================ 01:52:05 \(1305622325\)");echo XXRETCODE:$?' May 17 01:52:06 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 80: Page eviction is equally fast at high offsets too ================================ 01:52:05 (1305622325) May 17 01:52:06 client-5 xinetd[3391]: EXIT: shell status=0 pid=1842 duration=1(sec) May 17 01:52:07 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 01:52:07 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 01:52:07 client-5 xinetd[3391]: START: shell pid=1959 from=192.168.4.5 May 17 01:52:07 client-5 rshd[1960]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:52:07 client-5 xinetd[3391]: EXIT: shell status=0 pid=1959 duration=0(sec) May 17 01:52:07 client-5 xinetd[3391]: START: shell pid=2134 from=192.168.4.5 May 17 01:52:07 client-5 rshd[2135]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 99a: cvs init ========================================================================= 01:52:07 \(1305622327\)");echo XXRETCODE:$?' May 17 01:52:07 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 99a: cvs init ========================================================================= 01:52:07 (1305622327) May 17 01:52:07 client-5 xinetd[3391]: EXIT: shell status=0 pid=2134 duration=0(sec) May 17 01:52:08 client-5 xinetd[3391]: START: shell pid=2234 from=192.168.4.5 May 17 01:52:08 client-5 rshd[2235]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:52:08 client-5 xinetd[3391]: EXIT: shell status=0 pid=2234 duration=0(sec) May 17 01:52:08 client-5 xinetd[3391]: START: shell pid=2425 from=192.168.4.5 May 17 01:52:08 client-5 rshd[2426]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 99b: cvs import ======================================================================= 01:52:08 \(1305622328\)");echo XXRETCODE:$?' May 17 01:52:08 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 99b: cvs import ======================================================================= 01:52:08 (1305622328) May 17 01:52:08 client-5 xinetd[3391]: EXIT: shell status=0 pid=2425 duration=0(sec) May 17 01:52:09 client-5 xinetd[3391]: START: shell pid=2521 from=192.168.4.5 May 17 01:52:09 client-5 rshd[2522]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:52:09 client-5 xinetd[3391]: EXIT: shell status=0 pid=2521 duration=0(sec) May 17 01:52:10 client-5 xinetd[3391]: START: shell pid=2698 from=192.168.4.5 May 17 01:52:10 client-5 rshd[2699]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 99c: cvs checkout ===================================================================== 01:52:10 \(1305622330\)");echo XXRETCODE:$?' May 17 01:52:10 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 99c: cvs checkout ===================================================================== 01:52:10 (1305622330) May 17 01:52:10 client-5 xinetd[3391]: EXIT: shell status=0 pid=2698 duration=0(sec) May 17 01:52:13 client-5 xinetd[3391]: START: shell pid=2797 from=192.168.4.5 May 17 01:52:13 client-5 rshd[2798]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:52:13 client-5 xinetd[3391]: EXIT: shell status=0 pid=2797 duration=0(sec) May 17 01:52:13 client-5 xinetd[3391]: START: shell pid=2969 from=192.168.4.5 May 17 01:52:13 client-5 rshd[2970]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 99d: cvs add ========================================================================== 01:52:13 \(1305622333\)");echo XXRETCODE:$?' May 17 01:52:13 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 99d: cvs add ========================================================================== 01:52:13 (1305622333) May 17 01:52:13 client-5 xinetd[3391]: EXIT: shell status=0 pid=2969 duration=0(sec) May 17 01:52:13 client-5 xinetd[3391]: START: shell pid=3074 from=192.168.4.5 May 17 01:52:13 client-5 rshd[3075]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:52:13 client-5 xinetd[3391]: EXIT: shell status=0 pid=3074 duration=0(sec) May 17 01:52:13 client-5 xinetd[3391]: START: shell pid=3259 from=192.168.4.5 May 17 01:52:14 client-5 rshd[3260]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 99e: cvs update ======================================================================= 01:52:13 \(1305622333\)");echo XXRETCODE:$?' May 17 01:52:14 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 99e: cvs update ======================================================================= 01:52:13 (1305622333) May 17 01:52:14 client-5 xinetd[3391]: EXIT: shell status=0 pid=3259 duration=1(sec) May 17 01:52:14 client-5 xinetd[3391]: START: shell pid=3368 from=192.168.4.5 May 17 01:52:14 client-5 rshd[3369]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:52:14 client-5 xinetd[3391]: EXIT: shell status=0 pid=3368 duration=0(sec) May 17 01:52:14 client-5 xinetd[3391]: START: shell pid=3559 from=192.168.4.5 May 17 01:52:14 client-5 rshd[3562]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 99f: cvs commit ======================================================================= 01:52:14 \(1305622334\)");echo XXRETCODE:$?' May 17 01:52:15 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 99f: cvs commit ======================================================================= 01:52:14 (1305622334) May 17 01:52:15 client-5 xinetd[3391]: EXIT: shell status=0 pid=3559 duration=1(sec) May 17 01:52:16 client-5 xinetd[3391]: START: shell pid=3669 from=192.168.4.5 May 17 01:52:16 client-5 rshd[3671]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:52:16 client-5 xinetd[3391]: EXIT: shell status=0 pid=3669 duration=0(sec) May 17 01:52:16 client-5 xinetd[3391]: START: shell pid=3843 from=192.168.4.5 May 17 01:52:16 client-5 rshd[3844]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 100: check local port using privileged port =========================================== 01:52:16 \(1305622336\)");echo XXRETCODE:$?' May 17 01:52:16 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 100: check local port using privileged port =========================================== 01:52:16 (1305622336) May 17 01:52:16 client-5 xinetd[3391]: EXIT: shell status=0 pid=3843 duration=0(sec) May 17 01:52:16 client-5 xinetd[3391]: START: shell pid=3934 from=192.168.4.5 May 17 01:52:16 client-5 rshd[3937]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark SKIP: sanity test_100 TCP secure port test, not useful for NETTYPE=o2ib");echo XXRETCODE:$?' May 17 01:52:16 client-5 kernel: Lustre: DEBUG MARKER: SKIP: sanity test_100 TCP secure port test, not useful for NETTYPE=o2ib May 17 01:52:16 client-5 xinetd[3391]: EXIT: shell status=0 pid=3934 duration=0(sec) May 17 01:52:16 client-5 xinetd[3391]: START: shell pid=4032 from=192.168.4.5 May 17 01:52:16 client-5 rshd[4033]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:52:17 client-5 xinetd[3391]: EXIT: shell status=0 pid=4032 duration=1(sec) May 17 01:52:17 client-5 xinetd[3391]: START: shell pid=4233 from=192.168.4.5 May 17 01:52:17 client-5 rshd[4234]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 101: check read-ahead for random reads ================================================ 01:52:17 \(1305622337\)");echo XXRETCODE:$?' May 17 01:52:17 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 101: check read-ahead for random reads ================================================ 01:52:17 (1305622337) May 17 01:52:17 client-5 xinetd[3391]: EXIT: shell status=0 pid=4233 duration=0(sec) May 17 01:52:19 client-5 xinetd[3391]: START: shell pid=4335 from=192.168.4.5 May 17 01:52:19 client-5 rshd[4336]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:52:19 client-5 xinetd[3391]: EXIT: shell status=0 pid=4335 duration=0(sec) May 17 01:52:19 client-5 xinetd[3391]: START: shell pid=4507 from=192.168.4.5 May 17 01:52:19 client-5 rshd[4508]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 101b: check stride-io mode read-ahead ================================================= 01:52:19 \(1305622339\)");echo XXRETCODE:$?' May 17 01:52:19 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 101b: check stride-io mode read-ahead ================================================= 01:52:19 (1305622339) May 17 01:52:19 client-5 xinetd[3391]: EXIT: shell status=0 pid=4507 duration=0(sec) May 17 01:52:20 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 01:52:21 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 01:52:22 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 01:52:22 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 01:52:22 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 01:52:22 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 01:52:22 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 01:52:22 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 01:52:22 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 01:52:22 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 01:52:22 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 01:52:23 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 01:52:23 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 01:52:23 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 01:52:23 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 01:52:23 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 01:52:23 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 01:52:23 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 01:52:23 client-5 xinetd[3391]: START: shell pid=4792 from=192.168.4.5 May 17 01:52:23 client-5 rshd[4794]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:52:23 client-5 xinetd[3391]: EXIT: shell status=0 pid=4792 duration=0(sec) May 17 01:52:23 client-5 xinetd[3391]: START: shell pid=4964 from=192.168.4.5 May 17 01:52:23 client-5 rshd[4966]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 101d: file read with and without read-ahead enabled ================================== 01:52:23 \(1305622343\)");echo XXRETCODE:$?' May 17 01:52:24 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 101d: file read with and without read-ahead enabled ================================== 01:52:23 (1305622343) May 17 01:52:24 client-5 xinetd[3391]: EXIT: shell status=0 pid=4964 duration=1(sec) May 17 01:52:28 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 01:52:29 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 01:53:22 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 01:53:23 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 01:53:25 client-5 xinetd[3391]: START: shell pid=5104 from=192.168.4.5 May 17 01:53:25 client-5 rshd[5105]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:53:25 client-5 xinetd[3391]: EXIT: shell status=0 pid=5104 duration=0(sec) May 17 01:53:25 client-5 xinetd[3391]: START: shell pid=5276 from=192.168.4.5 May 17 01:53:25 client-5 rshd[5277]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 102a: user xattr test ================================================================= 01:53:25 \(1305622405\)");echo XXRETCODE:$?' May 17 01:53:25 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 102a: user xattr test ================================================================= 01:53:25 (1305622405) May 17 01:53:25 client-5 xinetd[3391]: EXIT: shell status=0 pid=5276 duration=0(sec) May 17 01:53:25 client-5 xinetd[3391]: START: shell pid=5402 from=192.168.4.5 May 17 01:53:25 client-5 rshd[5403]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:53:25 client-5 xinetd[3391]: EXIT: shell status=0 pid=5402 duration=0(sec) May 17 01:53:26 client-5 xinetd[3391]: START: shell pid=5574 from=192.168.4.5 May 17 01:53:26 client-5 rshd[5575]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 102b: getfattr\/setfattr for trusted.lov EAs =========================================== 01:53:26 \(1305622406\)");echo XXRETCODE:$?' May 17 01:53:26 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 102b: getfattr/setfattr for trusted.lov EAs =========================================== 01:53:26 (1305622406) May 17 01:53:26 client-5 xinetd[3391]: EXIT: shell status=0 pid=5574 duration=0(sec) May 17 01:53:26 client-5 xinetd[3391]: START: shell pid=5682 from=192.168.4.5 May 17 01:53:26 client-5 rshd[5683]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:53:26 client-5 xinetd[3391]: EXIT: shell status=0 pid=5682 duration=0(sec) May 17 01:53:26 client-5 xinetd[3391]: START: shell pid=5854 from=192.168.4.5 May 17 01:53:26 client-5 rshd[5855]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 102c: non-root getfattr\/setfattr for lustre.lov EAs =================================== 01:53:26 \(1305622406\)");echo XXRETCODE:$?' May 17 01:53:26 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 102c: non-root getfattr/setfattr for lustre.lov EAs =================================== 01:53:26 (1305622406) May 17 01:53:26 client-5 xinetd[3391]: EXIT: shell status=0 pid=5854 duration=0(sec) May 17 01:53:26 client-5 xinetd[3391]: START: shell pid=5963 from=192.168.4.5 May 17 01:53:26 client-5 rshd[5964]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:53:27 client-5 xinetd[3391]: EXIT: shell status=0 pid=5963 duration=1(sec) May 17 01:53:27 client-5 xinetd[3391]: START: shell pid=6135 from=192.168.4.5 May 17 01:53:27 client-5 rshd[6136]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 102d: tar restore stripe info from tarfile,not keep osts ============================== 01:53:27 \(1305622407\)");echo XXRETCODE:$?' May 17 01:53:27 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 102d: tar restore stripe info from tarfile,not keep osts ============================== 01:53:27 (1305622407) May 17 01:53:27 client-5 xinetd[3391]: EXIT: shell status=0 pid=6135 duration=0(sec) May 17 01:53:27 client-5 xinetd[3391]: START: shell pid=6234 from=192.168.4.5 May 17 01:53:27 client-5 rshd[6236]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark SKIP: sanity test_102d lustre-aware tar is not installed");echo XXRETCODE:$?' May 17 01:53:27 client-5 kernel: Lustre: DEBUG MARKER: SKIP: sanity test_102d lustre-aware tar is not installed May 17 01:53:27 client-5 xinetd[3391]: EXIT: shell status=0 pid=6234 duration=0(sec) May 17 01:53:27 client-5 xinetd[3391]: START: shell pid=6327 from=192.168.4.5 May 17 01:53:27 client-5 rshd[6328]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:53:27 client-5 xinetd[3391]: EXIT: shell status=0 pid=6327 duration=0(sec) May 17 01:53:28 client-5 xinetd[3391]: START: shell pid=6499 from=192.168.4.5 May 17 01:53:28 client-5 rshd[6500]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 102f: tar copy files, not keep osts =================================================== 01:53:27 \(1305622407\)");echo XXRETCODE:$?' May 17 01:53:28 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 102f: tar copy files, not keep osts =================================================== 01:53:27 (1305622407) May 17 01:53:28 client-5 xinetd[3391]: EXIT: shell status=0 pid=6499 duration=0(sec) May 17 01:53:28 client-5 xinetd[3391]: START: shell pid=6599 from=192.168.4.5 May 17 01:53:28 client-5 rshd[6600]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark SKIP: sanity test_102f lustre-aware tar is not installed");echo XXRETCODE:$?' May 17 01:53:28 client-5 kernel: Lustre: DEBUG MARKER: SKIP: sanity test_102f lustre-aware tar is not installed May 17 01:53:28 client-5 xinetd[3391]: EXIT: shell status=0 pid=6599 duration=0(sec) May 17 01:53:28 client-5 xinetd[3391]: START: shell pid=6690 from=192.168.4.5 May 17 01:53:28 client-5 rshd[6691]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:53:28 client-5 xinetd[3391]: EXIT: shell status=0 pid=6690 duration=0(sec) May 17 01:53:28 client-5 xinetd[3391]: START: shell pid=6860 from=192.168.4.5 May 17 01:53:28 client-5 rshd[6863]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 102h: grow xattr from inside inode to external block ================================== 01:53:28 \(1305622408\)");echo XXRETCODE:$?' May 17 01:53:28 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 102h: grow xattr from inside inode to external block ================================== 01:53:28 (1305622408) May 17 01:53:28 client-5 xinetd[3391]: EXIT: shell status=0 pid=6860 duration=0(sec) May 17 01:53:28 client-5 xinetd[3391]: START: shell pid=6979 from=192.168.4.5 May 17 01:53:28 client-5 rshd[6980]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark save trusted.big on \/mnt\/lustre\/f102h");echo XXRETCODE:$?' May 17 01:53:29 client-5 kernel: Lustre: DEBUG MARKER: save trusted.big on /mnt/lustre/f102h May 17 01:53:29 client-5 xinetd[3391]: EXIT: shell status=0 pid=6979 duration=1(sec) May 17 01:53:29 client-5 xinetd[3391]: START: shell pid=7079 from=192.168.4.5 May 17 01:53:29 client-5 rshd[7080]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark save trusted.sml on \/mnt\/lustre\/f102h");echo XXRETCODE:$?' May 17 01:53:29 client-5 kernel: Lustre: DEBUG MARKER: save trusted.sml on /mnt/lustre/f102h May 17 01:53:29 client-5 xinetd[3391]: EXIT: shell status=0 pid=7079 duration=0(sec) May 17 01:53:29 client-5 xinetd[3391]: START: shell pid=7174 from=192.168.4.5 May 17 01:53:29 client-5 rshd[7177]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark grow trusted.sml on \/mnt\/lustre\/f102h");echo XXRETCODE:$?' May 17 01:53:29 client-5 kernel: Lustre: DEBUG MARKER: grow trusted.sml on /mnt/lustre/f102h May 17 01:53:29 client-5 xinetd[3391]: EXIT: shell status=0 pid=7174 duration=0(sec) May 17 01:53:29 client-5 xinetd[3391]: START: shell pid=7273 from=192.168.4.5 May 17 01:53:29 client-5 rshd[7274]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark trusted.big still valid after growing trusted.sml");echo XXRETCODE:$?' May 17 01:53:29 client-5 kernel: Lustre: DEBUG MARKER: trusted.big still valid after growing trusted.sml May 17 01:53:29 client-5 xinetd[3391]: EXIT: shell status=0 pid=7273 duration=0(sec) May 17 01:53:29 client-5 xinetd[3391]: START: shell pid=7365 from=192.168.4.5 May 17 01:53:29 client-5 rshd[7366]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:53:29 client-5 xinetd[3391]: EXIT: shell status=0 pid=7365 duration=0(sec) May 17 01:53:30 client-5 xinetd[3391]: START: shell pid=7537 from=192.168.4.5 May 17 01:53:30 client-5 rshd[7538]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 102i: lgetxattr test on symbolic link ================================================= 01:53:29 \(1305622409\)");echo XXRETCODE:$?' May 17 01:53:30 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 102i: lgetxattr test on symbolic link ================================================= 01:53:29 (1305622409) May 17 01:53:30 client-5 xinetd[3391]: EXIT: shell status=0 pid=7537 duration=0(sec) May 17 01:53:30 client-5 xinetd[3391]: START: shell pid=7638 from=192.168.4.5 May 17 01:53:30 client-5 rshd[7639]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:53:30 client-5 xinetd[3391]: EXIT: shell status=0 pid=7638 duration=0(sec) May 17 01:53:30 client-5 xinetd[3391]: START: shell pid=7810 from=192.168.4.5 May 17 01:53:30 client-5 rshd[7811]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 102j: non-root tar restore stripe info from tarfile, not keep osts ==================== 01:53:30 \(1305622410\)");echo XXRETCODE:$?' May 17 01:53:30 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 102j: non-root tar restore stripe info from tarfile, not keep osts ==================== 01:53:30 (1305622410) May 17 01:53:30 client-5 xinetd[3391]: EXIT: shell status=0 pid=7810 duration=0(sec) May 17 01:53:30 client-5 xinetd[3391]: START: shell pid=7909 from=192.168.4.5 May 17 01:53:30 client-5 rshd[7911]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark SKIP: sanity test_102j lustre-aware tar is not installed");echo XXRETCODE:$?' May 17 01:53:30 client-5 kernel: Lustre: DEBUG MARKER: SKIP: sanity test_102j lustre-aware tar is not installed May 17 01:53:30 client-5 xinetd[3391]: EXIT: shell status=0 pid=7909 duration=0(sec) May 17 01:53:30 client-5 xinetd[3391]: START: shell pid=8001 from=192.168.4.5 May 17 01:53:30 client-5 rshd[8002]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:53:31 client-5 xinetd[3391]: EXIT: shell status=0 pid=8001 duration=1(sec) May 17 01:53:31 client-5 xinetd[3391]: START: shell pid=8173 from=192.168.4.5 May 17 01:53:31 client-5 rshd[8174]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 102k: setfattr without parameter of value shouldn\'t cause a crash ===================== 01:53:31 \(1305622411\)");echo XXRETCODE:$?' May 17 01:53:31 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 102k: setfattr without parameter of value shouldn't cause a crash ===================== 01:53:31 (1305622411) May 17 01:53:31 client-5 xinetd[3391]: EXIT: shell status=0 pid=8173 duration=0(sec) May 17 01:53:31 client-5 xinetd[3391]: START: shell pid=8276 from=192.168.4.5 May 17 01:53:31 client-5 rshd[8277]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:53:31 client-5 xinetd[3391]: EXIT: shell status=0 pid=8276 duration=0(sec) May 17 01:53:31 client-5 xinetd[3391]: START: shell pid=8450 from=192.168.4.5 May 17 01:53:31 client-5 rshd[8451]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 103: acl test ========================================================================= 01:53:31 \(1305622411\)");echo XXRETCODE:$?' May 17 01:53:31 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 103: acl test ========================================================================= 01:53:31 (1305622411) May 17 01:53:31 client-5 xinetd[3391]: EXIT: shell status=0 pid=8450 duration=0(sec) May 17 01:53:42 client-5 xinetd[3391]: START: shell pid=10155 from=192.168.4.5 May 17 01:53:42 client-5 rshd[10156]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:53:42 client-5 xinetd[3391]: EXIT: shell status=0 pid=10155 duration=0(sec) May 17 01:53:42 client-5 xinetd[3391]: START: shell pid=10327 from=192.168.4.5 May 17 01:53:42 client-5 rshd[10328]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 104a: lfs df [-ih] [path] test ======================================================== 01:53:42 \(1305622422\)");echo XXRETCODE:$?' May 17 01:53:42 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 104a: lfs df [-ih] [path] test ======================================================== 01:53:42 (1305622422) May 17 01:53:42 client-5 xinetd[3391]: EXIT: shell status=0 pid=10327 duration=0(sec) May 17 01:53:42 client-5 kernel: Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request May 17 01:53:42 client-5 kernel: Lustre: lustre-OST0000-osc-ffff8103201c4000: Connection to service lustre-OST0000 via nid 192.168.4.129@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 01:53:42 client-5 kernel: Lustre: Skipped 1 previous similar message May 17 01:53:42 client-5 kernel: LustreError: 167-0: This client was evicted by lustre-OST0000; in progress operations using this service will fail. May 17 01:53:42 client-5 kernel: Lustre: lustre-OST0000-osc-ffff8103201c4000: Connection restored to service lustre-OST0000 using nid 192.168.4.129@o2ib. May 17 01:53:42 client-5 kernel: Lustre: Skipped 1 previous similar message May 17 01:53:43 client-5 xinetd[3391]: START: shell pid=10435 from=192.168.4.5 May 17 01:53:43 client-5 rshd[10436]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:53:43 client-5 xinetd[3391]: EXIT: shell status=0 pid=10435 duration=0(sec) May 17 01:53:43 client-5 xinetd[3391]: START: shell pid=10607 from=192.168.4.5 May 17 01:53:43 client-5 rshd[10608]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 104b: runas -u 500 lfs check servers test ============================================= 01:53:43 \(1305622423\)");echo XXRETCODE:$?' May 17 01:53:43 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 104b: runas -u 500 lfs check servers test ============================================= 01:53:43 (1305622423) May 17 01:53:43 client-5 xinetd[3391]: EXIT: shell status=0 pid=10607 duration=0(sec) May 17 01:53:43 client-5 xinetd[3391]: START: shell pid=10703 from=192.168.4.5 May 17 01:53:43 client-5 rshd[10704]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:53:43 client-5 xinetd[3391]: EXIT: shell status=0 pid=10703 duration=0(sec) May 17 01:53:43 client-5 xinetd[3391]: START: shell pid=10875 from=192.168.4.5 May 17 01:53:43 client-5 rshd[10876]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 105a: flock when mounted without -o flock test ======================================== 01:53:43 \(1305622423\)");echo XXRETCODE:$?' May 17 01:53:43 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 105a: flock when mounted without -o flock test ======================================== 01:53:43 (1305622423) May 17 01:53:44 client-5 xinetd[3391]: EXIT: shell status=0 pid=10875 duration=1(sec) May 17 01:53:44 client-5 xinetd[3391]: START: shell pid=10973 from=192.168.4.5 May 17 01:53:44 client-5 rshd[10974]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:53:44 client-5 xinetd[3391]: EXIT: shell status=0 pid=10973 duration=0(sec) May 17 01:53:44 client-5 xinetd[3391]: START: shell pid=11145 from=192.168.4.5 May 17 01:53:44 client-5 rshd[11146]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 105b: fcntl when mounted without -o flock test ======================================== 01:53:44 \(1305622424\)");echo XXRETCODE:$?' May 17 01:53:44 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 105b: fcntl when mounted without -o flock test ======================================== 01:53:44 (1305622424) May 17 01:53:44 client-5 xinetd[3391]: EXIT: shell status=0 pid=11145 duration=0(sec) May 17 01:53:44 client-5 xinetd[3391]: START: shell pid=11242 from=192.168.4.5 May 17 01:53:44 client-5 rshd[11244]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:53:44 client-5 xinetd[3391]: EXIT: shell status=0 pid=11242 duration=0(sec) May 17 01:53:44 client-5 xinetd[3391]: START: shell pid=11414 from=192.168.4.5 May 17 01:53:44 client-5 rshd[11416]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 105c: lockf when mounted without -o flock test ======================================== 01:53:44 \(1305622424\)");echo XXRETCODE:$?' May 17 01:53:45 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 105c: lockf when mounted without -o flock test ======================================== 01:53:44 (1305622424) May 17 01:53:45 client-5 xinetd[3391]: EXIT: shell status=0 pid=11414 duration=1(sec) May 17 01:53:45 client-5 xinetd[3391]: START: shell pid=11513 from=192.168.4.5 May 17 01:53:45 client-5 rshd[11514]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:53:45 client-5 xinetd[3391]: EXIT: shell status=0 pid=11513 duration=0(sec) May 17 01:53:45 client-5 xinetd[3391]: START: shell pid=11685 from=192.168.4.5 May 17 01:53:45 client-5 rshd[11686]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 105d: flock race \(should not freeze\) ================================================== 01:53:45 \(1305622425\)");echo XXRETCODE:$?' May 17 01:53:45 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 105d: flock race (should not freeze) ================================================== 01:53:45 (1305622425) May 17 01:53:45 client-5 xinetd[3391]: EXIT: shell status=0 pid=11685 duration=0(sec) May 17 01:53:45 client-5 kernel: LustreError: 11727:0:(lvfs_lib.c:150:__obd_fail_timeout_set()) obd_fail_timeout id 315 sleeping for 10000ms May 17 01:53:55 client-5 kernel: LustreError: 11727:0:(lvfs_lib.c:154:__obd_fail_timeout_set()) obd_fail_timeout id 315 awake May 17 01:53:55 client-5 xinetd[3391]: START: shell pid=11785 from=192.168.4.5 May 17 01:53:55 client-5 rshd[11786]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:53:55 client-5 xinetd[3391]: EXIT: shell status=0 pid=11785 duration=0(sec) May 17 01:53:56 client-5 xinetd[3391]: START: shell pid=11955 from=192.168.4.5 May 17 01:53:56 client-5 rshd[11958]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 105e: Two conflicting flocks from same process ======================================== 01:53:56 \(1305622436\)");echo XXRETCODE:$?' May 17 01:53:56 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 105e: Two conflicting flocks from same process ======================================== 01:53:56 (1305622436) May 17 01:53:56 client-5 xinetd[3391]: EXIT: shell status=0 pid=11955 duration=0(sec) May 17 01:53:56 client-5 xinetd[3391]: START: shell pid=12055 from=192.168.4.5 May 17 01:53:56 client-5 rshd[12056]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:53:56 client-5 xinetd[3391]: EXIT: shell status=0 pid=12055 duration=0(sec) May 17 01:53:56 client-5 xinetd[3391]: START: shell pid=12227 from=192.168.4.5 May 17 01:53:56 client-5 rshd[12228]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 106: attempt exec of dir followed by chown of that dir ================================ 01:53:56 \(1305622436\)");echo XXRETCODE:$?' May 17 01:53:56 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 106: attempt exec of dir followed by chown of that dir ================================ 01:53:56 (1305622436) May 17 01:53:56 client-5 xinetd[3391]: EXIT: shell status=0 pid=12227 duration=0(sec) May 17 01:53:56 client-5 xinetd[3391]: START: shell pid=12319 from=192.168.4.5 May 17 01:53:56 client-5 rshd[12322]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:53:56 client-5 xinetd[3391]: EXIT: shell status=0 pid=12319 duration=0(sec) May 17 01:53:57 client-5 xinetd[3391]: START: shell pid=12493 from=192.168.4.5 May 17 01:53:57 client-5 rshd[12494]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 107: Coredump on SIG ================================================================== 01:53:57 \(1305622437\)");echo XXRETCODE:$?' May 17 01:53:57 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 107: Coredump on SIG ================================================================== 01:53:57 (1305622437) May 17 01:53:57 client-5 xinetd[3391]: EXIT: shell status=0 pid=12493 duration=0(sec) May 17 01:53:58 client-5 xinetd[3391]: START: shell pid=12596 from=192.168.4.5 May 17 01:53:58 client-5 rshd[12597]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:53:58 client-5 xinetd[3391]: EXIT: shell status=0 pid=12596 duration=0(sec) May 17 01:53:58 client-5 xinetd[3391]: START: shell pid=12768 from=192.168.4.5 May 17 01:53:58 client-5 rshd[12769]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 110: filename length checking ========================================================= 01:53:58 \(1305622438\)");echo XXRETCODE:$?' May 17 01:53:58 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 110: filename length checking ========================================================= 01:53:58 (1305622438) May 17 01:53:58 client-5 xinetd[3391]: EXIT: shell status=0 pid=12768 duration=0(sec) May 17 01:53:58 client-5 xinetd[3391]: START: shell pid=12866 from=192.168.4.5 May 17 01:53:58 client-5 rshd[12869]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:53:59 client-5 xinetd[3391]: EXIT: shell status=0 pid=12866 duration=1(sec) May 17 01:53:59 client-5 xinetd[3391]: START: shell pid=13040 from=192.168.4.5 May 17 01:53:59 client-5 rshd[13041]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 115: verify dynamic thread creation==================================================== 01:53:59 \(1305622439\)");echo XXRETCODE:$?' May 17 01:53:59 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 115: verify dynamic thread creation==================================================== 01:53:59 (1305622439) May 17 01:53:59 client-5 xinetd[3391]: EXIT: shell status=0 pid=13040 duration=0(sec) May 17 01:53:59 client-5 xinetd[3391]: START: shell pid=13140 from=192.168.4.5 May 17 01:53:59 client-5 rshd[13141]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark SKIP: sanity test_115 no OSS threads");echo XXRETCODE:$?' May 17 01:53:59 client-5 kernel: Lustre: DEBUG MARKER: SKIP: sanity test_115 no OSS threads May 17 01:53:59 client-5 xinetd[3391]: EXIT: shell status=0 pid=13140 duration=0(sec) May 17 01:53:59 client-5 xinetd[3391]: START: shell pid=13229 from=192.168.4.5 May 17 01:53:59 client-5 rshd[13232]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:53:59 client-5 xinetd[3391]: EXIT: shell status=0 pid=13229 duration=0(sec) May 17 01:53:59 client-5 xinetd[3391]: START: shell pid=13403 from=192.168.4.5 May 17 01:54:00 client-5 rshd[13404]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 116: stripe QOS: free space balance =================================================== 01:53:59 \(1305622439\)");echo XXRETCODE:$?' May 17 01:54:00 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 116: stripe QOS: free space balance =================================================== 01:53:59 (1305622439) May 17 01:54:00 client-5 xinetd[3391]: EXIT: shell status=0 pid=13403 duration=1(sec) May 17 01:54:01 client-5 xinetd[3391]: START: shell pid=13505 from=192.168.4.5 May 17 01:54:01 client-5 rshd[13506]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark SKIP: sanity test_116 too much free space in OST0, skip");echo XXRETCODE:$?' May 17 01:54:01 client-5 kernel: Lustre: DEBUG MARKER: SKIP: sanity test_116 too much free space in OST0, skip May 17 01:54:01 client-5 xinetd[3391]: EXIT: shell status=0 pid=13505 duration=0(sec) May 17 01:54:01 client-5 xinetd[3391]: START: shell pid=13594 from=192.168.4.5 May 17 01:54:01 client-5 rshd[13597]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:54:01 client-5 xinetd[3391]: EXIT: shell status=0 pid=13594 duration=0(sec) May 17 01:54:01 client-5 xinetd[3391]: START: shell pid=13768 from=192.168.4.5 May 17 01:54:01 client-5 rshd[13769]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 117: verify fsfilt_extend ============================================================= 01:54:01 \(1305622441\)");echo XXRETCODE:$?' May 17 01:54:01 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 117: verify fsfilt_extend ============================================================= 01:54:01 (1305622441) May 17 01:54:01 client-5 xinetd[3391]: EXIT: shell status=0 pid=13768 duration=0(sec) May 17 01:54:01 client-5 xinetd[3391]: START: shell pid=13863 from=192.168.4.5 May 17 01:54:01 client-5 rshd[13864]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:54:02 client-5 xinetd[3391]: EXIT: shell status=0 pid=13863 duration=1(sec) May 17 01:54:02 client-5 xinetd[3391]: START: shell pid=14035 from=192.168.4.5 May 17 01:54:02 client-5 rshd[14036]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 118a: verify O_SYNC works ============================================================= 01:54:02 \(1305622442\)");echo XXRETCODE:$?' May 17 01:54:02 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 118a: verify O_SYNC works ============================================================= 01:54:02 (1305622442) May 17 01:54:02 client-5 xinetd[3391]: EXIT: shell status=0 pid=14035 duration=0(sec) May 17 01:54:03 client-5 xinetd[3391]: START: shell pid=14136 from=192.168.4.5 May 17 01:54:03 client-5 rshd[14139]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:54:03 client-5 xinetd[3391]: EXIT: shell status=0 pid=14136 duration=0(sec) May 17 01:54:03 client-5 xinetd[3391]: START: shell pid=14310 from=192.168.4.5 May 17 01:54:03 client-5 rshd[14311]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 118b: Reclaim dirty pages on fatal error ============================================== 01:54:03 \(1305622443\)");echo XXRETCODE:$?' May 17 01:54:04 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 118b: Reclaim dirty pages on fatal error ============================================== 01:54:03 (1305622443) May 17 01:54:04 client-5 xinetd[3391]: EXIT: shell status=0 pid=14310 duration=1(sec) May 17 01:54:04 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.129@o2ib. The ost_write operation failed with -2 May 17 01:54:04 client-5 kernel: LustreError: Skipped 5 previous similar messages May 17 01:54:05 client-5 xinetd[3391]: START: shell pid=14510 from=192.168.4.5 May 17 01:54:05 client-5 rshd[14512]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:54:05 client-5 xinetd[3391]: EXIT: shell status=0 pid=14510 duration=0(sec) May 17 01:54:05 client-5 xinetd[3391]: START: shell pid=14683 from=192.168.4.5 May 17 01:54:05 client-5 rshd[14684]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 118c: Fsync blocks on EROFS until dirty pages are flushed ============================= 01:54:05 \(1305622445\)");echo XXRETCODE:$?' May 17 01:54:05 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 118c: Fsync blocks on EROFS until dirty pages are flushed ============================= 01:54:05 (1305622445) May 17 01:54:05 client-5 xinetd[3391]: EXIT: shell status=0 pid=14683 duration=0(sec) May 17 01:54:06 client-5 kernel: LustreError: 4121:0:(osc_request.c:1741:osc_brw_redo_request()) @@@ redo for recoverable error req@ffff81033f771800 x1369038007821713/t0(0) o-1->lustre-OST0002_UUID@192.168.4.129@o2ib:6/4 lens 456/192 e 0 to 0 dl 1305622490 ref 2 fl Interpret:R/ffffffff/ffffffff rc -30/-1 May 17 01:54:06 client-5 kernel: LustreError: 4121:0:(osc_request.c:1741:osc_brw_redo_request()) Skipped 57 previous similar messages May 17 01:54:50 client-5 xinetd[3391]: START: shell pid=14892 from=192.168.4.5 May 17 01:54:50 client-5 rshd[14893]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:54:50 client-5 xinetd[3391]: EXIT: shell status=0 pid=14892 duration=0(sec) May 17 01:54:50 client-5 xinetd[3391]: START: shell pid=15064 from=192.168.4.5 May 17 01:54:50 client-5 rshd[15065]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 118d: Fsync validation inject a delay of the bulk ===================================== 01:54:50 \(1305622490\)");echo XXRETCODE:$?' May 17 01:54:50 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 118d: Fsync validation inject a delay of the bulk ===================================== 01:54:50 (1305622490) May 17 01:54:50 client-5 xinetd[3391]: EXIT: shell status=0 pid=15064 duration=0(sec) May 17 01:54:57 client-5 xinetd[3391]: START: shell pid=15269 from=192.168.4.5 May 17 01:54:57 client-5 rshd[15270]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:54:57 client-5 xinetd[3391]: EXIT: shell status=0 pid=15269 duration=0(sec) May 17 01:54:57 client-5 xinetd[3391]: START: shell pid=15441 from=192.168.4.5 May 17 01:54:57 client-5 rshd[15442]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 118f: Simulate unrecoverable OSC side error =========================================== 01:54:57 \(1305622497\)");echo XXRETCODE:$?' May 17 01:54:57 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 118f: Simulate unrecoverable OSC side error =========================================== 01:54:57 (1305622497) May 17 01:54:57 client-5 xinetd[3391]: EXIT: shell status=0 pid=15441 duration=0(sec) May 17 01:54:57 client-5 kernel: LustreError: 15481:0:(obd_support.h:456:obd_fail_check_set()) *** obd_fail_loc=40a *** May 17 01:54:57 client-5 kernel: LustreError: 15481:0:(obd_support.h:456:obd_fail_check_set()) Skipped 119 previous similar messages May 17 01:54:57 client-5 kernel: LustreError: 15481:0:(osc_request.c:2354:osc_build_req()) prep_req failed: -22 May 17 01:54:57 client-5 kernel: LustreError: 15481:0:(osc_request.c:2760:osc_check_rpcs()) Write request failed with -22 May 17 01:54:59 client-5 xinetd[3391]: START: shell pid=15553 from=192.168.4.5 May 17 01:54:59 client-5 rshd[15554]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:54:59 client-5 xinetd[3391]: EXIT: shell status=0 pid=15553 duration=0(sec) May 17 01:54:59 client-5 xinetd[3391]: START: shell pid=15725 from=192.168.4.5 May 17 01:54:59 client-5 rshd[15726]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 118g: Don\'t stay in wait if we got local -ENOMEM ===================================== 01:54:59 \(1305622499\)");echo XXRETCODE:$?' May 17 01:54:59 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 118g: Don't stay in wait if we got local -ENOMEM ===================================== 01:54:59 (1305622499) May 17 01:54:59 client-5 xinetd[3391]: EXIT: shell status=0 pid=15725 duration=0(sec) May 17 01:54:59 client-5 kernel: LustreError: 15765:0:(osc_request.c:2354:osc_build_req()) prep_req failed: -12 May 17 01:54:59 client-5 kernel: LustreError: 15765:0:(osc_request.c:2760:osc_check_rpcs()) Write request failed with -12 May 17 01:55:01 client-5 xinetd[3391]: START: shell pid=15837 from=192.168.4.5 May 17 01:55:01 client-5 rshd[15838]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:55:01 client-5 xinetd[3391]: EXIT: shell status=0 pid=15837 duration=0(sec) May 17 01:55:01 client-5 xinetd[3391]: START: shell pid=16009 from=192.168.4.5 May 17 01:55:01 client-5 rshd[16010]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 118h: Verify timeout in handling recoverables errors ================================= 01:55:01 \(1305622501\)");echo XXRETCODE:$?' May 17 01:55:01 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 118h: Verify timeout in handling recoverables errors ================================= 01:55:01 (1305622501) May 17 01:55:01 client-5 xinetd[3391]: EXIT: shell status=0 pid=16009 duration=0(sec) May 17 01:55:02 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.129@o2ib. The ost_write operation failed with -5 May 17 01:55:02 client-5 kernel: LustreError: Skipped 1 previous similar message May 17 01:55:45 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.129@o2ib. The ost_write operation failed with -5 May 17 01:55:54 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.129@o2ib. The ost_write operation failed with -5 May 17 01:55:54 client-5 kernel: LustreError: Skipped 2 previous similar messages May 17 01:56:12 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.129@o2ib. The ost_write operation failed with -5 May 17 01:56:12 client-5 kernel: LustreError: Skipped 2 previous similar messages May 17 01:56:20 client-5 kernel: LustreError: 4121:0:(osc_request.c:1741:osc_brw_redo_request()) @@@ redo for recoverable error req@ffff81031e478c00 x1369038007822089/t0(0) o-1->lustre-OST0000_UUID@192.168.4.129@o2ib:6/4 lens 456/192 e 0 to 0 dl 1305622624 ref 2 fl Interpret:R/ffffffff/ffffffff rc -5/-1 May 17 01:56:20 client-5 kernel: LustreError: 4121:0:(osc_request.c:1741:osc_brw_redo_request()) Skipped 8 previous similar messages May 17 01:56:39 client-5 kernel: LustreError: 4121:0:(osc_request.c:1737:osc_brw_redo_request()) too many resent retries, returning error May 17 01:56:39 client-5 kernel: LustreError: 4121:0:(osc_request.c:1737:osc_brw_redo_request()) Skipped 7 previous similar messages May 17 01:56:41 client-5 xinetd[3391]: START: shell pid=16216 from=192.168.4.5 May 17 01:56:41 client-5 rshd[16217]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:56:41 client-5 xinetd[3391]: EXIT: shell status=0 pid=16216 duration=0(sec) May 17 01:56:41 client-5 xinetd[3391]: START: shell pid=16388 from=192.168.4.5 May 17 01:56:41 client-5 rshd[16389]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 118i: Fix error before timeout in recoverable error ================================== 01:56:41 \(1305622601\)");echo XXRETCODE:$?' May 17 01:56:41 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 118i: Fix error before timeout in recoverable error ================================== 01:56:41 (1305622601) May 17 01:56:41 client-5 xinetd[3391]: EXIT: shell status=0 pid=16388 duration=0(sec) May 17 01:57:27 client-5 xinetd[3391]: START: shell pid=16595 from=192.168.4.5 May 17 01:57:27 client-5 rshd[16597]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:57:27 client-5 xinetd[3391]: EXIT: shell status=0 pid=16595 duration=0(sec) May 17 01:57:27 client-5 xinetd[3391]: START: shell pid=16768 from=192.168.4.5 May 17 01:57:27 client-5 rshd[16769]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 118j: Simulate unrecoverable OST side error =========================================== 01:57:27 \(1305622647\)");echo XXRETCODE:$?' May 17 01:57:27 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 118j: Simulate unrecoverable OST side error =========================================== 01:57:27 (1305622647) May 17 01:57:27 client-5 xinetd[3391]: EXIT: shell status=0 pid=16768 duration=0(sec) May 17 01:57:27 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.129@o2ib. The ost_write operation failed with -14 May 17 01:57:27 client-5 kernel: LustreError: Skipped 4 previous similar messages May 17 01:57:30 client-5 xinetd[3391]: START: shell pid=16971 from=192.168.4.5 May 17 01:57:30 client-5 rshd[16972]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:57:30 client-5 xinetd[3391]: EXIT: shell status=0 pid=16971 duration=0(sec) May 17 01:57:30 client-5 xinetd[3391]: START: shell pid=17142 from=192.168.4.5 May 17 01:57:30 client-5 rshd[17144]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 118k: bio alloc -ENOMEM and IO TERM handling ========================================== 01:57:30 \(1305622650\)");echo XXRETCODE:$?' May 17 01:57:30 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 118k: bio alloc -ENOMEM and IO TERM handling ========================================== 01:57:30 (1305622650) May 17 01:57:30 client-5 xinetd[3391]: EXIT: shell status=0 pid=17142 duration=0(sec) May 17 01:57:39 client-5 xinetd[3391]: START: shell pid=17363 from=192.168.4.5 May 17 01:57:39 client-5 rshd[17364]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:57:39 client-5 xinetd[3391]: EXIT: shell status=0 pid=17363 duration=0(sec) May 17 01:57:39 client-5 xinetd[3391]: START: shell pid=17533 from=192.168.4.5 May 17 01:57:39 client-5 rshd[17536]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 119a: Short directIO read must return actual read amount ============================== 01:57:39 \(1305622659\)");echo XXRETCODE:$?' May 17 01:57:39 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 119a: Short directIO read must return actual read amount ============================== 01:57:39 (1305622659) May 17 01:57:39 client-5 xinetd[3391]: EXIT: shell status=0 pid=17533 duration=0(sec) May 17 01:57:39 client-5 xinetd[3391]: START: shell pid=17631 from=192.168.4.5 May 17 01:57:39 client-5 rshd[17632]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:57:39 client-5 xinetd[3391]: EXIT: shell status=0 pid=17631 duration=0(sec) May 17 01:57:39 client-5 xinetd[3391]: START: shell pid=17803 from=192.168.4.5 May 17 01:57:39 client-5 rshd[17804]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 119b: Sparse directIO read must return actual read amount ============================= 01:57:39 \(1305622659\)");echo XXRETCODE:$?' May 17 01:57:40 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 119b: Sparse directIO read must return actual read amount ============================= 01:57:39 (1305622659) May 17 01:57:40 client-5 xinetd[3391]: EXIT: shell status=0 pid=17803 duration=1(sec) May 17 01:57:40 client-5 xinetd[3391]: START: shell pid=17899 from=192.168.4.5 May 17 01:57:40 client-5 rshd[17900]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:57:40 client-5 xinetd[3391]: EXIT: shell status=0 pid=17899 duration=0(sec) May 17 01:57:40 client-5 xinetd[3391]: START: shell pid=18071 from=192.168.4.5 May 17 01:57:40 client-5 rshd[18072]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 119c: Testing for direct read hitting hole ============================================ 01:57:40 \(1305622660\)");echo XXRETCODE:$?' May 17 01:57:40 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 119c: Testing for direct read hitting hole ============================================ 01:57:40 (1305622660) May 17 01:57:40 client-5 xinetd[3391]: EXIT: shell status=0 pid=18071 duration=0(sec) May 17 01:57:40 client-5 xinetd[3391]: START: shell pid=18165 from=192.168.4.5 May 17 01:57:40 client-5 rshd[18166]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:57:40 client-5 xinetd[3391]: EXIT: shell status=0 pid=18165 duration=0(sec) May 17 01:57:41 client-5 xinetd[3391]: START: shell pid=18337 from=192.168.4.5 May 17 01:57:41 client-5 rshd[18338]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 119d: The DIO path should try to send a new rpc once one is completed ================= 01:57:41 \(1305622661\)");echo XXRETCODE:$?' May 17 01:57:41 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 119d: The DIO path should try to send a new rpc once one is completed ================= 01:57:41 (1305622661) May 17 01:57:41 client-5 xinetd[3391]: EXIT: shell status=0 pid=18337 duration=0(sec) May 17 01:57:42 client-5 xinetd[3391]: START: shell pid=18437 from=192.168.4.5 May 17 01:57:42 client-5 rshd[18440]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark the DIO writes have completed, now wait for the reads \(should not block very long\)");echo XXRETCODE:$?' May 17 01:57:42 client-5 kernel: Lustre: DEBUG MARKER: the DIO writes have completed, now wait for the reads (should not block very long) May 17 01:57:42 client-5 xinetd[3391]: EXIT: shell status=0 pid=18437 duration=0(sec) May 17 01:57:44 client-5 xinetd[3391]: START: shell pid=18533 from=192.168.4.5 May 17 01:57:44 client-5 rshd[18535]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:57:44 client-5 xinetd[3391]: EXIT: shell status=0 pid=18533 duration=0(sec) May 17 01:57:44 client-5 xinetd[3391]: START: shell pid=18706 from=192.168.4.5 May 17 01:57:44 client-5 rshd[18707]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 120a: Early Lock Cancel: mkdir test =================================================== 01:57:44 \(1305622664\)");echo XXRETCODE:$?' May 17 01:57:44 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 120a: Early Lock Cancel: mkdir test =================================================== 01:57:44 (1305622664) May 17 01:57:44 client-5 xinetd[3391]: EXIT: shell status=0 pid=18706 duration=0(sec) May 17 01:57:45 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc start May 17 01:57:45 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc stop May 17 01:57:45 client-5 xinetd[3391]: START: shell pid=18832 from=192.168.4.5 May 17 01:57:45 client-5 rshd[18835]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:57:45 client-5 xinetd[3391]: EXIT: shell status=0 pid=18832 duration=0(sec) May 17 01:57:45 client-5 xinetd[3391]: START: shell pid=19005 from=192.168.4.5 May 17 01:57:45 client-5 rshd[19007]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 120b: Early Lock Cancel: create test ================================================== 01:57:45 \(1305622665\)");echo XXRETCODE:$?' May 17 01:57:45 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 120b: Early Lock Cancel: create test ================================================== 01:57:45 (1305622665) May 17 01:57:45 client-5 xinetd[3391]: EXIT: shell status=0 pid=19005 duration=0(sec) May 17 01:57:45 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc start May 17 01:57:45 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc stop May 17 01:57:45 client-5 xinetd[3391]: START: shell pid=19134 from=192.168.4.5 May 17 01:57:45 client-5 rshd[19135]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:57:45 client-5 xinetd[3391]: EXIT: shell status=0 pid=19134 duration=0(sec) May 17 01:57:46 client-5 xinetd[3391]: START: shell pid=19306 from=192.168.4.5 May 17 01:57:46 client-5 rshd[19307]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 120c: Early Lock Cancel: link test ==================================================== 01:57:46 \(1305622666\)");echo XXRETCODE:$?' May 17 01:57:46 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 120c: Early Lock Cancel: link test ==================================================== 01:57:46 (1305622666) May 17 01:57:46 client-5 xinetd[3391]: EXIT: shell status=0 pid=19306 duration=0(sec) May 17 01:57:46 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc start May 17 01:57:46 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc stop May 17 01:57:46 client-5 xinetd[3391]: START: shell pid=19436 from=192.168.4.5 May 17 01:57:46 client-5 rshd[19437]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:57:46 client-5 xinetd[3391]: EXIT: shell status=0 pid=19436 duration=0(sec) May 17 01:57:46 client-5 xinetd[3391]: START: shell pid=19608 from=192.168.4.5 May 17 01:57:46 client-5 rshd[19609]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 120d: Early Lock Cancel: setattr test ================================================= 01:57:46 \(1305622666\)");echo XXRETCODE:$?' May 17 01:57:46 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 120d: Early Lock Cancel: setattr test ================================================= 01:57:46 (1305622666) May 17 01:57:46 client-5 xinetd[3391]: EXIT: shell status=0 pid=19608 duration=0(sec) May 17 01:57:46 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc start May 17 01:57:46 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc stop May 17 01:57:46 client-5 xinetd[3391]: START: shell pid=19737 from=192.168.4.5 May 17 01:57:46 client-5 rshd[19738]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:57:47 client-5 xinetd[3391]: EXIT: shell status=0 pid=19737 duration=1(sec) May 17 01:57:47 client-5 xinetd[3391]: START: shell pid=19909 from=192.168.4.5 May 17 01:57:47 client-5 rshd[19910]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 120e: Early Lock Cancel: unlink test ================================================== 01:57:47 \(1305622667\)");echo XXRETCODE:$?' May 17 01:57:47 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 120e: Early Lock Cancel: unlink test ================================================== 01:57:47 (1305622667) May 17 01:57:47 client-5 xinetd[3391]: EXIT: shell status=0 pid=19909 duration=0(sec) May 17 01:57:47 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc start May 17 01:57:47 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc stop May 17 01:57:47 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 01:57:47 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 01:57:47 client-5 xinetd[3391]: START: shell pid=20053 from=192.168.4.5 May 17 01:57:47 client-5 rshd[20054]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:57:47 client-5 xinetd[3391]: EXIT: shell status=0 pid=20053 duration=0(sec) May 17 01:57:47 client-5 xinetd[3391]: START: shell pid=20225 from=192.168.4.5 May 17 01:57:48 client-5 rshd[20226]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 120f: Early Lock Cancel: rename test ================================================== 01:57:47 \(1305622667\)");echo XXRETCODE:$?' May 17 01:57:48 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 120f: Early Lock Cancel: rename test ================================================== 01:57:47 (1305622667) May 17 01:57:48 client-5 xinetd[3391]: EXIT: shell status=0 pid=20225 duration=1(sec) May 17 01:57:48 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc start May 17 01:57:48 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc stop May 17 01:57:48 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 01:57:48 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 01:57:48 client-5 xinetd[3391]: START: shell pid=20371 from=192.168.4.5 May 17 01:57:48 client-5 rshd[20373]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:57:48 client-5 xinetd[3391]: EXIT: shell status=0 pid=20371 duration=0(sec) May 17 01:57:48 client-5 xinetd[3391]: START: shell pid=20543 from=192.168.4.5 May 17 01:57:48 client-5 rshd[20545]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 120g: Early Lock Cancel: performance test ============================================= 01:57:48 \(1305622668\)");echo XXRETCODE:$?' May 17 01:57:48 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 120g: Early Lock Cancel: performance test ============================================= 01:57:48 (1305622668) May 17 01:57:48 client-5 xinetd[3391]: EXIT: shell status=0 pid=20543 duration=0(sec) May 17 01:57:48 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc start May 17 01:57:48 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc stop May 17 01:57:48 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 01:57:48 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 01:58:36 client-5 xinetd[3391]: START: shell pid=20706 from=192.168.4.5 May 17 01:58:36 client-5 rshd[20707]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:58:36 client-5 xinetd[3391]: EXIT: shell status=0 pid=20706 duration=0(sec) May 17 01:58:36 client-5 xinetd[3391]: START: shell pid=20878 from=192.168.4.5 May 17 01:58:36 client-5 rshd[20879]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 121: read cancel race ================================================================= 01:58:36 \(1305622716\)");echo XXRETCODE:$?' May 17 01:58:36 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 121: read cancel race ================================================================= 01:58:36 (1305622716) May 17 01:58:36 client-5 xinetd[3391]: EXIT: shell status=0 pid=20878 duration=0(sec) May 17 01:58:36 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 01:58:36 client-5 kernel: LustreError: 988:0:(obd_support.h:456:obd_fail_check_set()) *** obd_fail_loc=310 *** May 17 01:58:36 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 01:58:36 client-5 xinetd[3391]: START: shell pid=20991 from=192.168.4.5 May 17 01:58:36 client-5 rshd[20993]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 01:58:36 client-5 xinetd[3391]: EXIT: shell status=0 pid=20991 duration=0(sec) May 17 01:58:37 client-5 xinetd[3391]: START: shell pid=21164 from=192.168.4.5 May 17 01:58:37 client-5 rshd[21165]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 123a: verify statahead work =========================================================== 01:58:36 \(1305622716\)");echo XXRETCODE:$?' May 17 01:58:37 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 123a: verify statahead work =========================================================== 01:58:36 (1305622716) May 17 01:58:37 client-5 xinetd[3391]: EXIT: shell status=0 pid=21164 duration=0(sec) May 17 01:58:37 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc start May 17 01:58:37 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc stop May 17 01:58:37 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 01:58:37 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 01:58:37 client-5 xinetd[3391]: START: shell pid=21297 from=192.168.4.5 May 17 01:58:37 client-5 rshd[21298]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark ls 100 files without statahead: 0 sec");echo XXRETCODE:$?' May 17 01:58:37 client-5 kernel: Lustre: DEBUG MARKER: ls 100 files without statahead: 0 sec May 17 01:58:37 client-5 xinetd[3391]: EXIT: shell status=0 pid=21297 duration=0(sec) May 17 01:58:37 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc start May 17 01:58:37 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc stop May 17 01:58:37 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 01:58:37 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 01:58:38 client-5 xinetd[3391]: START: shell pid=21424 from=192.168.4.5 May 17 01:58:38 client-5 rshd[21426]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark ls 100 files with statahead: 1 sec");echo XXRETCODE:$?' May 17 01:58:38 client-5 kernel: Lustre: DEBUG MARKER: ls 100 files with statahead: 1 sec May 17 01:58:38 client-5 xinetd[3391]: EXIT: shell status=0 pid=21424 duration=0(sec) May 17 01:58:39 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc start May 17 01:58:40 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc stop May 17 01:58:40 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 01:58:40 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 01:58:42 client-5 xinetd[3391]: START: shell pid=21556 from=192.168.4.5 May 17 01:58:42 client-5 rshd[21557]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark ls 1000 files without statahead: 2 sec");echo XXRETCODE:$?' May 17 01:58:42 client-5 kernel: Lustre: DEBUG MARKER: ls 1000 files without statahead: 2 sec May 17 01:58:42 client-5 xinetd[3391]: EXIT: shell status=0 pid=21556 duration=0(sec) May 17 01:58:42 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc start May 17 01:58:42 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc stop May 17 01:58:42 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 01:58:43 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 01:58:44 client-5 xinetd[3391]: START: shell pid=21684 from=192.168.4.5 May 17 01:58:44 client-5 rshd[21685]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark ls 1000 files with statahead: 1 sec");echo XXRETCODE:$?' May 17 01:58:44 client-5 kernel: Lustre: DEBUG MARKER: ls 1000 files with statahead: 1 sec May 17 01:58:44 client-5 xinetd[3391]: EXIT: shell status=0 pid=21684 duration=0(sec) May 17 01:59:01 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc start May 17 01:59:01 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc stop May 17 01:59:01 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 01:59:02 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 01:59:27 client-5 xinetd[3391]: START: shell pid=21815 from=192.168.4.5 May 17 01:59:27 client-5 rshd[21816]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark ls 10000 files without statahead: 25 sec");echo XXRETCODE:$?' May 17 01:59:27 client-5 kernel: Lustre: DEBUG MARKER: ls 10000 files without statahead: 25 sec May 17 01:59:27 client-5 xinetd[3391]: EXIT: shell status=0 pid=21815 duration=0(sec) May 17 01:59:27 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc start May 17 01:59:28 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc stop May 17 01:59:28 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 01:59:29 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 01:59:40 client-5 xinetd[3391]: START: shell pid=21943 from=192.168.4.5 May 17 01:59:40 client-5 rshd[21944]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark ls 10000 files with statahead: 11 sec");echo XXRETCODE:$?' May 17 01:59:40 client-5 kernel: Lustre: DEBUG MARKER: ls 10000 files with statahead: 11 sec May 17 01:59:40 client-5 xinetd[3391]: EXIT: shell status=0 pid=21943 duration=0(sec) May 17 01:59:41 client-5 xinetd[3391]: START: shell pid=22041 from=192.168.4.5 May 17 01:59:41 client-5 rshd[22042]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark ls done");echo XXRETCODE:$?' May 17 01:59:41 client-5 kernel: Lustre: DEBUG MARKER: ls done May 17 01:59:41 client-5 xinetd[3391]: EXIT: shell status=0 pid=22041 duration=0(sec) May 17 02:00:01 client-5 xinetd[3391]: START: shell pid=22143 from=192.168.4.5 May 17 02:00:01 client-5 rshd[22144]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark rm -r \/mnt\/lustre\/d0.sanity\/d123\/: 20 seconds");echo XXRETCODE:$?' May 17 02:00:01 client-5 kernel: Lustre: DEBUG MARKER: rm -r /mnt/lustre/d0.sanity/d123/: 20 seconds May 17 02:00:01 client-5 xinetd[3391]: EXIT: shell status=0 pid=22143 duration=0(sec) May 17 02:00:01 client-5 xinetd[3391]: START: shell pid=22236 from=192.168.4.5 May 17 02:00:01 client-5 rshd[22237]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark rm done");echo XXRETCODE:$?' May 17 02:00:02 client-5 kernel: Lustre: DEBUG MARKER: rm done May 17 02:00:02 client-5 xinetd[3391]: EXIT: shell status=0 pid=22236 duration=1(sec) May 17 02:00:02 client-5 xinetd[3391]: START: shell pid=22328 from=192.168.4.5 May 17 02:00:02 client-5 rshd[22329]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 02:00:02 client-5 xinetd[3391]: EXIT: shell status=0 pid=22328 duration=0(sec) May 17 02:00:02 client-5 xinetd[3391]: START: shell pid=22500 from=192.168.4.5 May 17 02:00:02 client-5 rshd[22501]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 123b: not panic with network error in statahead enqueue \(bug 15027\) =================== 02:00:02 \(1305622802\)");echo XXRETCODE:$?' May 17 02:00:02 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 123b: not panic with network error in statahead enqueue (bug 15027) =================== 02:00:02 (1305622802) May 17 02:00:02 client-5 xinetd[3391]: EXIT: shell status=0 pid=22500 duration=0(sec) May 17 02:00:04 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc start May 17 02:00:04 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc stop May 17 02:00:04 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 02:00:04 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 02:00:04 client-5 kernel: LustreError: 4123:0:(obd_support.h:456:obd_fail_check_set()) *** obd_fail_loc=803 *** May 17 02:00:04 client-5 kernel: LustreError: 4123:0:(mdc_locks.c:996:mdc_intent_getattr_async_interpret()) ldlm_cli_enqueue_fini: -110 May 17 02:00:05 client-5 xinetd[3391]: START: shell pid=22621 from=192.168.4.5 May 17 02:00:05 client-5 rshd[22622]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark ls done");echo XXRETCODE:$?' May 17 02:00:05 client-5 kernel: Lustre: DEBUG MARKER: ls done May 17 02:00:05 client-5 xinetd[3391]: EXIT: shell status=0 pid=22621 duration=0(sec) May 17 02:00:07 client-5 xinetd[3391]: START: shell pid=22717 from=192.168.4.5 May 17 02:00:08 client-5 rshd[22718]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 02:00:08 client-5 xinetd[3391]: EXIT: shell status=0 pid=22717 duration=1(sec) May 17 02:00:08 client-5 xinetd[3391]: START: shell pid=22889 from=192.168.4.5 May 17 02:00:08 client-5 rshd[22890]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 124a: lru resize ====================================================================== 02:00:08 \(1305622808\)");echo XXRETCODE:$?' May 17 02:00:08 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 124a: lru resize ====================================================================== 02:00:08 (1305622808) May 17 02:00:08 client-5 xinetd[3391]: EXIT: shell status=0 pid=22889 duration=0(sec) May 17 02:00:08 client-5 xinetd[3391]: START: shell pid=22986 from=192.168.4.5 May 17 02:00:08 client-5 rshd[22987]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark create 2000 files at \/mnt\/lustre\/d0.sanity\/d124");echo XXRETCODE:$?' May 17 02:00:08 client-5 kernel: Lustre: DEBUG MARKER: create 2000 files at /mnt/lustre/d0.sanity/d124 May 17 02:00:08 client-5 xinetd[3391]: EXIT: shell status=0 pid=22986 duration=0(sec) May 17 02:00:12 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc start May 17 02:00:12 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc stop May 17 02:00:15 client-5 xinetd[3391]: START: shell pid=23102 from=192.168.4.5 May 17 02:00:15 client-5 rshd[23104]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark NS=ldlm.namespaces.lustre-MDT0000-mdc-ffff8103201c4000");echo XXRETCODE:$?' May 17 02:00:15 client-5 kernel: Lustre: DEBUG MARKER: NS=ldlm.namespaces.lustre-MDT0000-mdc-ffff8103201c4000 May 17 02:00:15 client-5 xinetd[3391]: EXIT: shell status=0 pid=23102 duration=0(sec) May 17 02:00:15 client-5 xinetd[3391]: START: shell pid=23199 from=192.168.4.5 May 17 02:00:15 client-5 rshd[23200]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark LRU=2003");echo XXRETCODE:$?' May 17 02:00:15 client-5 kernel: Lustre: DEBUG MARKER: LRU=2003 May 17 02:00:15 client-5 xinetd[3391]: EXIT: shell status=0 pid=23199 duration=0(sec) May 17 02:00:15 client-5 xinetd[3391]: START: shell pid=23293 from=192.168.4.5 May 17 02:00:15 client-5 rshd[23294]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark LVF=766841");echo XXRETCODE:$?' May 17 02:00:15 client-5 kernel: Lustre: DEBUG MARKER: LVF=766841 May 17 02:00:15 client-5 xinetd[3391]: EXIT: shell status=0 pid=23293 duration=0(sec) May 17 02:00:15 client-5 xinetd[3391]: START: shell pid=23388 from=192.168.4.5 May 17 02:00:15 client-5 rshd[23389]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark Sleep 50 sec");echo XXRETCODE:$?' May 17 02:00:15 client-5 kernel: Lustre: DEBUG MARKER: Sleep 50 sec May 17 02:00:15 client-5 xinetd[3391]: EXIT: shell status=0 pid=23388 duration=0(sec) May 17 02:01:05 client-5 xinetd[3391]: START: shell pid=23513 from=192.168.4.5 May 17 02:01:05 client-5 rshd[23514]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark Dropped 725 locks in 50s");echo XXRETCODE:$?' May 17 02:01:05 client-5 kernel: Lustre: DEBUG MARKER: Dropped 725 locks in 50s May 17 02:01:05 client-5 xinetd[3391]: EXIT: shell status=0 pid=23513 duration=0(sec) May 17 02:01:05 client-5 xinetd[3391]: START: shell pid=23606 from=192.168.4.5 May 17 02:01:05 client-5 rshd[23607]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark unlink 2000 files at \/mnt\/lustre\/d0.sanity\/d124");echo XXRETCODE:$?' May 17 02:01:06 client-5 kernel: Lustre: DEBUG MARKER: unlink 2000 files at /mnt/lustre/d0.sanity/d124 May 17 02:01:06 client-5 xinetd[3391]: EXIT: shell status=0 pid=23606 duration=1(sec) May 17 02:01:09 client-5 xinetd[3391]: START: shell pid=23697 from=192.168.4.5 May 17 02:01:09 client-5 rshd[23699]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 02:01:09 client-5 xinetd[3391]: EXIT: shell status=0 pid=23697 duration=0(sec) May 17 02:01:09 client-5 xinetd[3391]: START: shell pid=23868 from=192.168.4.5 May 17 02:01:09 client-5 rshd[23871]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 124b: lru resize \(performance test\) =================================================== 02:01:09 \(1305622869\)");echo XXRETCODE:$?' May 17 02:01:09 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 124b: lru resize (performance test) =================================================== 02:01:09 (1305622869) May 17 02:01:09 client-5 xinetd[3391]: EXIT: shell status=0 pid=23868 duration=0(sec) May 17 02:01:25 client-5 xinetd[3391]: START: shell pid=23978 from=192.168.4.5 May 17 02:01:25 client-5 rshd[23979]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark doing ls -la \/mnt\/lustre\/d0.sanity\/d124\/disable_lru_resize 3 times");echo XXRETCODE:$?' May 17 02:01:25 client-5 kernel: Lustre: DEBUG MARKER: doing ls -la /mnt/lustre/d0.sanity/d124/disable_lru_resize 3 times May 17 02:01:25 client-5 xinetd[3391]: EXIT: shell status=0 pid=23978 duration=0(sec) May 17 02:01:25 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc start May 17 02:01:25 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc stop May 17 02:01:51 client-5 xinetd[3391]: START: shell pid=24088 from=192.168.4.5 May 17 02:01:51 client-5 rshd[24089]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark ls -la time: 26 seconds");echo XXRETCODE:$?' May 17 02:01:51 client-5 kernel: Lustre: DEBUG MARKER: ls -la time: 26 seconds May 17 02:01:51 client-5 xinetd[3391]: EXIT: shell status=0 pid=24088 duration=0(sec) May 17 02:01:51 client-5 xinetd[3391]: START: shell pid=24182 from=192.168.4.5 May 17 02:01:51 client-5 rshd[24183]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark lru_size = 400");echo XXRETCODE:$?' May 17 02:01:51 client-5 kernel: Lustre: DEBUG MARKER: lru_size = 400 May 17 02:01:51 client-5 xinetd[3391]: EXIT: shell status=0 pid=24182 duration=0(sec) May 17 02:02:28 client-5 xinetd[3391]: START: shell pid=24283 from=192.168.4.5 May 17 02:02:28 client-5 rshd[24284]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark doing ls -la \/mnt\/lustre\/d0.sanity\/d124\/enable_lru_resize 3 times");echo XXRETCODE:$?' May 17 02:02:28 client-5 kernel: Lustre: DEBUG MARKER: doing ls -la /mnt/lustre/d0.sanity/d124/enable_lru_resize 3 times May 17 02:02:28 client-5 xinetd[3391]: EXIT: shell status=0 pid=24283 duration=0(sec) May 17 02:02:28 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc start May 17 02:02:28 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc stop May 17 02:02:36 client-5 xinetd[3391]: START: shell pid=24393 from=192.168.4.5 May 17 02:02:36 client-5 rshd[24394]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark ls -la time: 8 seconds");echo XXRETCODE:$?' May 17 02:02:36 client-5 kernel: Lustre: DEBUG MARKER: ls -la time: 8 seconds May 17 02:02:36 client-5 xinetd[3391]: EXIT: shell status=0 pid=24393 duration=0(sec) May 17 02:02:36 client-5 xinetd[3391]: START: shell pid=24487 from=192.168.4.5 May 17 02:02:36 client-5 rshd[24488]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark lru_size = 8004");echo XXRETCODE:$?' May 17 02:02:36 client-5 kernel: Lustre: DEBUG MARKER: lru_size = 8004 May 17 02:02:36 client-5 xinetd[3391]: EXIT: shell status=0 pid=24487 duration=0(sec) May 17 02:02:36 client-5 xinetd[3391]: START: shell pid=24580 from=192.168.4.5 May 17 02:02:36 client-5 rshd[24581]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark ls -la is 69% faster with lru resize enabled");echo XXRETCODE:$?' May 17 02:02:36 client-5 kernel: Lustre: DEBUG MARKER: ls -la is 69% faster with lru resize enabled May 17 02:02:36 client-5 xinetd[3391]: EXIT: shell status=0 pid=24580 duration=0(sec) May 17 02:02:48 client-5 xinetd[3391]: START: shell pid=24672 from=192.168.4.5 May 17 02:02:48 client-5 rshd[24673]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 02:02:48 client-5 xinetd[3391]: EXIT: shell status=0 pid=24672 duration=0(sec) May 17 02:02:48 client-5 xinetd[3391]: START: shell pid=24844 from=192.168.4.5 May 17 02:02:48 client-5 rshd[24845]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 125: don\'t return EPROTO when a dir has a non-default striping and ACLs =============== 02:02:48 \(1305622968\)");echo XXRETCODE:$?' May 17 02:02:48 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 125: don't return EPROTO when a dir has a non-default striping and ACLs =============== 02:02:48 (1305622968) May 17 02:02:48 client-5 xinetd[3391]: EXIT: shell status=0 pid=24844 duration=0(sec) May 17 02:02:49 client-5 xinetd[3391]: START: shell pid=24945 from=192.168.4.5 May 17 02:02:49 client-5 rshd[24946]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 02:02:49 client-5 xinetd[3391]: EXIT: shell status=0 pid=24945 duration=0(sec) May 17 02:02:49 client-5 xinetd[3391]: START: shell pid=25115 from=192.168.4.5 May 17 02:02:49 client-5 rshd[25118]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 126: check that the fsgid provided by the client is taken into account ================ 02:02:49 \(1305622969\)");echo XXRETCODE:$?' May 17 02:02:49 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 126: check that the fsgid provided by the client is taken into account ================ 02:02:49 (1305622969) May 17 02:02:49 client-5 xinetd[3391]: EXIT: shell status=0 pid=25115 duration=0(sec) May 17 02:02:49 client-5 xinetd[3391]: START: shell pid=25216 from=192.168.4.5 May 17 02:02:49 client-5 rshd[25217]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 02:02:49 client-5 xinetd[3391]: EXIT: shell status=0 pid=25216 duration=0(sec) May 17 02:02:49 client-5 xinetd[3391]: START: shell pid=25386 from=192.168.4.5 May 17 02:02:50 client-5 rshd[25389]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 127: verify the client stats are sane ================================================= 02:02:49 \(1305622969\)");echo XXRETCODE:$?' May 17 02:02:50 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 127: verify the client stats are sane ================================================= 02:02:49 (1305622969) May 17 02:02:50 client-5 xinetd[3391]: EXIT: shell status=0 pid=25386 duration=1(sec) May 17 02:02:50 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 02:02:50 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 02:02:50 client-5 xinetd[3391]: START: shell pid=25499 from=192.168.4.5 May 17 02:02:50 client-5 rshd[25500]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 02:02:50 client-5 xinetd[3391]: EXIT: shell status=0 pid=25499 duration=0(sec) May 17 02:02:50 client-5 xinetd[3391]: START: shell pid=25671 from=192.168.4.5 May 17 02:02:50 client-5 rshd[25672]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 128: interactive lfs for 2 consecutive find\'s ========================================= 02:02:50 \(1305622970\)");echo XXRETCODE:$?' May 17 02:02:50 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 128: interactive lfs for 2 consecutive find's ========================================= 02:02:50 (1305622970) May 17 02:02:50 client-5 xinetd[3391]: EXIT: shell status=0 pid=25671 duration=0(sec) May 17 02:02:50 client-5 xinetd[3391]: START: shell pid=25767 from=192.168.4.5 May 17 02:02:50 client-5 rshd[25768]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 02:02:50 client-5 xinetd[3391]: EXIT: shell status=0 pid=25767 duration=0(sec) May 17 02:02:51 client-5 xinetd[3391]: START: shell pid=25939 from=192.168.4.5 May 17 02:02:51 client-5 rshd[25940]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 129: test directory size limit ======================================================== 02:02:51 \(1305622971\)");echo XXRETCODE:$?' May 17 02:02:51 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 129: test directory size limit ======================================================== 02:02:51 (1305622971) May 17 02:02:51 client-5 xinetd[3391]: EXIT: shell status=0 pid=25939 duration=0(sec) May 17 02:02:53 client-5 xinetd[3391]: START: shell pid=26618 from=192.168.4.5 May 17 02:02:53 client-5 rshd[26619]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 02:02:54 client-5 xinetd[3391]: EXIT: shell status=0 pid=26618 duration=1(sec) May 17 02:02:54 client-5 xinetd[3391]: START: shell pid=26790 from=192.168.4.5 May 17 02:02:54 client-5 rshd[26791]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 130a: FIEMAP \(1-stripe file\) ========================================================== 02:02:54 \(1305622974\)");echo XXRETCODE:$?' May 17 02:02:54 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 130a: FIEMAP (1-stripe file) ========================================================== 02:02:54 (1305622974) May 17 02:02:54 client-5 xinetd[3391]: EXIT: shell status=0 pid=26790 duration=0(sec) May 17 02:02:54 client-5 xinetd[3391]: START: shell pid=26886 from=192.168.4.5 May 17 02:02:54 client-5 rshd[26887]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark SKIP: sanity test_130a filefrag does not support FIEMAP");echo XXRETCODE:$?' May 17 02:02:54 client-5 kernel: Lustre: DEBUG MARKER: SKIP: sanity test_130a filefrag does not support FIEMAP May 17 02:02:54 client-5 xinetd[3391]: EXIT: shell status=0 pid=26886 duration=0(sec) May 17 02:02:54 client-5 xinetd[3391]: START: shell pid=26976 from=192.168.4.5 May 17 02:02:54 client-5 rshd[26978]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 02:02:54 client-5 xinetd[3391]: EXIT: shell status=0 pid=26976 duration=0(sec) May 17 02:02:55 client-5 xinetd[3391]: START: shell pid=27149 from=192.168.4.5 May 17 02:02:55 client-5 rshd[27150]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 130b: FIEMAP \(2-stripe file\) ========================================================== 02:02:54 \(1305622974\)");echo XXRETCODE:$?' May 17 02:02:55 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 130b: FIEMAP (2-stripe file) ========================================================== 02:02:54 (1305622974) May 17 02:02:55 client-5 xinetd[3391]: EXIT: shell status=0 pid=27149 duration=0(sec) May 17 02:02:55 client-5 xinetd[3391]: START: shell pid=27243 from=192.168.4.5 May 17 02:02:55 client-5 rshd[27246]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark SKIP: sanity test_130b filefrag does not support FIEMAP");echo XXRETCODE:$?' May 17 02:02:55 client-5 kernel: Lustre: DEBUG MARKER: SKIP: sanity test_130b filefrag does not support FIEMAP May 17 02:02:55 client-5 xinetd[3391]: EXIT: shell status=0 pid=27243 duration=0(sec) May 17 02:02:55 client-5 xinetd[3391]: START: shell pid=27336 from=192.168.4.5 May 17 02:02:55 client-5 rshd[27337]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 02:02:55 client-5 xinetd[3391]: EXIT: shell status=0 pid=27336 duration=0(sec) May 17 02:02:55 client-5 xinetd[3391]: START: shell pid=27508 from=192.168.4.5 May 17 02:02:55 client-5 rshd[27509]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 130c: FIEMAP \(2-stripe file with hole\) ================================================ 02:02:55 \(1305622975\)");echo XXRETCODE:$?' May 17 02:02:55 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 130c: FIEMAP (2-stripe file with hole) ================================================ 02:02:55 (1305622975) May 17 02:02:55 client-5 xinetd[3391]: EXIT: shell status=0 pid=27508 duration=0(sec) May 17 02:02:55 client-5 xinetd[3391]: START: shell pid=27604 from=192.168.4.5 May 17 02:02:56 client-5 rshd[27605]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark SKIP: sanity test_130c filefrag does not support FIEMAP");echo XXRETCODE:$?' May 17 02:02:56 client-5 kernel: Lustre: DEBUG MARKER: SKIP: sanity test_130c filefrag does not support FIEMAP May 17 02:02:56 client-5 xinetd[3391]: EXIT: shell status=0 pid=27604 duration=1(sec) May 17 02:02:56 client-5 xinetd[3391]: START: shell pid=27694 from=192.168.4.5 May 17 02:02:56 client-5 rshd[27696]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 02:02:56 client-5 xinetd[3391]: EXIT: shell status=0 pid=27694 duration=0(sec) May 17 02:02:56 client-5 xinetd[3391]: START: shell pid=27867 from=192.168.4.5 May 17 02:02:56 client-5 rshd[27868]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 130d: FIEMAP \(N-stripe file\) ========================================================== 02:02:56 \(1305622976\)");echo XXRETCODE:$?' May 17 02:02:56 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 130d: FIEMAP (N-stripe file) ========================================================== 02:02:56 (1305622976) May 17 02:02:56 client-5 xinetd[3391]: EXIT: shell status=0 pid=27867 duration=0(sec) May 17 02:02:56 client-5 xinetd[3391]: START: shell pid=27961 from=192.168.4.5 May 17 02:02:56 client-5 rshd[27964]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark SKIP: sanity test_130d filefrag does not support FIEMAP");echo XXRETCODE:$?' May 17 02:02:56 client-5 kernel: Lustre: DEBUG MARKER: SKIP: sanity test_130d filefrag does not support FIEMAP May 17 02:02:56 client-5 xinetd[3391]: EXIT: shell status=0 pid=27961 duration=0(sec) May 17 02:02:56 client-5 xinetd[3391]: START: shell pid=28054 from=192.168.4.5 May 17 02:02:56 client-5 rshd[28055]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 02:02:57 client-5 xinetd[3391]: EXIT: shell status=0 pid=28054 duration=1(sec) May 17 02:02:57 client-5 xinetd[3391]: START: shell pid=28226 from=192.168.4.5 May 17 02:02:57 client-5 rshd[28227]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 130e: FIEMAP \(test continuation FIEMAP calls\) ========================================= 02:02:57 \(1305622977\)");echo XXRETCODE:$?' May 17 02:02:57 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 130e: FIEMAP (test continuation FIEMAP calls) ========================================= 02:02:57 (1305622977) May 17 02:02:57 client-5 xinetd[3391]: EXIT: shell status=0 pid=28226 duration=0(sec) May 17 02:02:57 client-5 xinetd[3391]: START: shell pid=28321 from=192.168.4.5 May 17 02:02:57 client-5 rshd[28323]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark SKIP: sanity test_130e filefrag does not support FIEMAP");echo XXRETCODE:$?' May 17 02:02:57 client-5 kernel: Lustre: DEBUG MARKER: SKIP: sanity test_130e filefrag does not support FIEMAP May 17 02:02:57 client-5 xinetd[3391]: EXIT: shell status=0 pid=28321 duration=0(sec) May 17 02:02:57 client-5 xinetd[3391]: START: shell pid=28413 from=192.168.4.5 May 17 02:02:57 client-5 rshd[28414]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 02:02:57 client-5 xinetd[3391]: EXIT: shell status=0 pid=28413 duration=0(sec) May 17 02:02:57 client-5 xinetd[3391]: START: shell pid=28585 from=192.168.4.5 May 17 02:02:58 client-5 rshd[28586]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 131a: test iov\'s crossing stripe boundary for writev\/readv ============================ 02:02:57 \(1305622977\)");echo XXRETCODE:$?' May 17 02:02:58 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 131a: test iov's crossing stripe boundary for writev/readv ============================ 02:02:57 (1305622977) May 17 02:02:58 client-5 xinetd[3391]: EXIT: shell status=0 pid=28585 duration=1(sec) May 17 02:02:58 client-5 xinetd[3391]: START: shell pid=28679 from=192.168.4.5 May 17 02:02:58 client-5 rshd[28680]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 02:02:58 client-5 xinetd[3391]: EXIT: shell status=0 pid=28679 duration=0(sec) May 17 02:02:58 client-5 xinetd[3391]: START: shell pid=28851 from=192.168.4.5 May 17 02:02:58 client-5 rshd[28852]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 131b: test append writev ============================================================== 02:02:58 \(1305622978\)");echo XXRETCODE:$?' May 17 02:02:58 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 131b: test append writev ============================================================== 02:02:58 (1305622978) May 17 02:02:58 client-5 xinetd[3391]: EXIT: shell status=0 pid=28851 duration=0(sec) May 17 02:02:58 client-5 xinetd[3391]: START: shell pid=28945 from=192.168.4.5 May 17 02:02:58 client-5 rshd[28946]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 02:02:58 client-5 xinetd[3391]: EXIT: shell status=0 pid=28945 duration=0(sec) May 17 02:02:59 client-5 xinetd[3391]: START: shell pid=29117 from=192.168.4.5 May 17 02:02:59 client-5 rshd[29118]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 131c: test read\/write on file w\/o objects ============================================= 02:02:59 \(1305622979\)");echo XXRETCODE:$?' May 17 02:02:59 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 131c: test read/write on file w/o objects ============================================= 02:02:59 (1305622979) May 17 02:02:59 client-5 xinetd[3391]: EXIT: shell status=0 pid=29117 duration=0(sec) May 17 02:02:59 client-5 xinetd[3391]: START: shell pid=29208 from=192.168.4.5 May 17 02:02:59 client-5 rshd[29210]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 02:02:59 client-5 xinetd[3391]: EXIT: shell status=0 pid=29208 duration=0(sec) May 17 02:02:59 client-5 xinetd[3391]: START: shell pid=29381 from=192.168.4.5 May 17 02:02:59 client-5 rshd[29382]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 131d: test short read ================================================================= 02:02:59 \(1305622979\)");echo XXRETCODE:$?' May 17 02:02:59 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 131d: test short read ================================================================= 02:02:59 (1305622979) May 17 02:02:59 client-5 xinetd[3391]: EXIT: shell status=0 pid=29381 duration=0(sec) May 17 02:02:59 client-5 xinetd[3391]: START: shell pid=29477 from=192.168.4.5 May 17 02:02:59 client-5 rshd[29478]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 02:02:59 client-5 xinetd[3391]: EXIT: shell status=0 pid=29477 duration=0(sec) May 17 02:03:00 client-5 xinetd[3391]: START: shell pid=29649 from=192.168.4.5 May 17 02:03:00 client-5 rshd[29650]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 131e: test read hitting hole ========================================================== 02:03:00 \(1305622980\)");echo XXRETCODE:$?' May 17 02:03:00 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 131e: test read hitting hole ========================================================== 02:03:00 (1305622980) May 17 02:03:00 client-5 xinetd[3391]: EXIT: shell status=0 pid=29649 duration=0(sec) May 17 02:03:00 client-5 xinetd[3391]: START: shell pid=29743 from=192.168.4.5 May 17 02:03:00 client-5 rshd[29744]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 02:03:00 client-5 xinetd[3391]: EXIT: shell status=0 pid=29743 duration=0(sec) May 17 02:03:00 client-5 xinetd[3391]: START: shell pid=29915 from=192.168.4.5 May 17 02:03:00 client-5 rshd[29916]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 132: som avoids glimpse rpc =========================================================== 02:03:00 \(1305622980\)");echo XXRETCODE:$?' May 17 02:03:00 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 132: som avoids glimpse rpc =========================================================== 02:03:00 (1305622980) May 17 02:03:00 client-5 xinetd[3391]: EXIT: shell status=0 pid=29915 duration=0(sec) May 17 02:03:02 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 02:03:03 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 02:03:03 client-5 xinetd[3391]: START: shell pid=30080 from=192.168.4.5 May 17 02:03:03 client-5 rshd[30081]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:; lsof -t /mnt/lustre || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre is still busy, wait one second && sleep 1; umount /mnt/lustre; fi fi");echo XXRETCODE:$?' May 17 02:03:03 client-5 kernel: LustreError: 30122:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 02:03:03 client-5 kernel: LustreError: 30122:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Skipped 1 previous similar message May 17 02:03:03 client-5 kernel: LustreError: 30122:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 02:03:03 client-5 kernel: LustreError: 30122:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) Skipped 1 previous similar message May 17 02:03:03 client-5 kernel: Lustre: client ffff8103201c4000 umount complete May 17 02:03:03 client-5 xinetd[3391]: EXIT: shell status=0 pid=30080 duration=0(sec) May 17 02:03:04 client-5 xinetd[3391]: START: shell pid=30131 from=192.168.4.5 May 17 02:03:04 client-5 rshd[30132]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:; lsof -t /mnt/lustre2 || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre2 | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre2 is still busy, wait one second && sleep 1; umount /mnt/lustre2; fi fi");echo XXRETCODE:$?' May 17 02:03:04 client-5 xinetd[3391]: EXIT: shell status=0 pid=30131 duration=0(sec) May 17 02:03:30 client-5 xinetd[3391]: START: shell pid=30497 from=192.168.4.5 May 17 02:03:30 client-5 rshd[30498]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); mpts=\$(mount | grep -w -c /mnt/lustre); if [ \$running -ne \$mpts ]; then echo \$(hostname) env are INSANE!; exit 1; fi");echo XXRETCODE:$?' May 17 02:03:30 client-5 xinetd[3391]: EXIT: shell status=0 pid=30497 duration=0(sec) May 17 02:03:30 client-5 xinetd[3391]: START: shell pid=30544 from=192.168.4.5 May 17 02:03:30 client-5 rshd[30545]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); mpts=\$(mount | grep -w -c /mnt/lustre2); if [ \$running -ne \$mpts ]; then echo \$(hostname) env are INSANE!; exit 1; fi");echo XXRETCODE:$?' May 17 02:03:30 client-5 xinetd[3391]: EXIT: shell status=0 pid=30544 duration=0(sec) May 17 02:03:30 client-5 kernel: Lustre: 30587:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 02:03:30 client-5 kernel: Lustre: 30587:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message May 17 02:03:35 client-5 kernel: Lustre: 30917:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 02:03:35 client-5 kernel: Lustre: 30917:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 7 previous similar messages May 17 02:03:35 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 02:03:35 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 02:03:35 client-5 kernel: Lustre: Increasing default stripe size to min 1048576 May 17 02:03:35 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 02:03:35 client-5 kernel: LustreError: Skipped 82 previous similar messages May 17 02:03:45 client-5 kernel: Lustre: 30917:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-OST0000-osc-ffff810313e1ec00->192.168.4.129@o2ib netid 50000: select flavor null May 17 02:03:45 client-5 kernel: Lustre: 30917:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 1 previous similar message May 17 02:03:45 client-5 kernel: Lustre: Client lustre-client has started May 17 02:03:45 client-5 kernel: Lustre: 30934:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 02:03:45 client-5 kernel: Lustre: 30934:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message May 17 02:03:45 client-5 xinetd[3391]: START: shell pid=30944 from=192.168.4.5 May 17 02:03:45 client-5 rshd[30945]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c " running=\$(mount | grep -c /mnt/lustre' '); rc=0; if [ \$running -eq 0 ] ; then mkdir -p /mnt/lustre; mount -t lustre -o flock,user_xattr,acl,som_preview fat-intel-1-ib@o2ib:/lustre /mnt/lustre; rc=\$?; fi; exit \$rc");echo XXRETCODE:$?' May 17 02:03:45 client-5 xinetd[3391]: EXIT: shell status=0 pid=30944 duration=0(sec) May 17 02:03:45 client-5 xinetd[3391]: START: shell pid=30988 from=192.168.4.5 May 17 02:03:45 client-5 rshd[30990]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mount | grep -w /mnt/lustre");echo XXRETCODE:$?' May 17 02:03:45 client-5 xinetd[3391]: EXIT: shell status=0 pid=30988 duration=0(sec) May 17 02:03:45 client-5 xinetd[3391]: START: shell pid=31033 from=192.168.4.5 May 17 02:03:45 client-5 rshd[31034]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param debug=\"0x33f0406\"; lctl set_param subsystem_debug=\"0xffb7e3ff\"; lctl set_param debug_mb=32;");echo XXRETCODE:$?' May 17 02:03:45 client-5 xinetd[3391]: EXIT: shell status=0 pid=31033 duration=0(sec) May 17 02:03:46 client-5 xinetd[3391]: START: shell pid=31203 from=192.168.4.5 May 17 02:03:46 client-5 rshd[31204]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark Using TIMEOUT=20");echo XXRETCODE:$?' May 17 02:03:46 client-5 kernel: Lustre: DEBUG MARKER: Using TIMEOUT=20 May 17 02:03:46 client-5 xinetd[3391]: EXIT: shell status=0 pid=31203 duration=0(sec) May 17 02:03:46 client-5 kernel: LustreError: 31257:0:(quota_ctl.c:328:client_quota_ctl()) ptlrpc_queue_wait failed, rc: -114 May 17 02:03:51 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 02:03:51 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 02:03:52 client-5 xinetd[3391]: START: shell pid=31377 from=192.168.4.5 May 17 02:03:52 client-5 rshd[31378]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:; lsof -t /mnt/lustre || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre is still busy, wait one second && sleep 1; umount /mnt/lustre; fi fi");echo XXRETCODE:$?' May 17 02:03:52 client-5 kernel: LustreError: 31419:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 02:03:52 client-5 kernel: LustreError: 31419:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Skipped 6 previous similar messages May 17 02:03:52 client-5 kernel: LustreError: 31419:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 02:03:52 client-5 kernel: LustreError: 31419:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) Skipped 6 previous similar messages May 17 02:03:52 client-5 kernel: Lustre: client ffff810313e1ec00 umount complete May 17 02:03:52 client-5 xinetd[3391]: EXIT: shell status=0 pid=31377 duration=0(sec) May 17 02:03:52 client-5 xinetd[3391]: START: shell pid=31428 from=192.168.4.5 May 17 02:03:52 client-5 rshd[31429]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:; lsof -t /mnt/lustre2 || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre2 | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre2 is still busy, wait one second && sleep 1; umount /mnt/lustre2; fi fi");echo XXRETCODE:$?' May 17 02:03:52 client-5 xinetd[3391]: EXIT: shell status=0 pid=31428 duration=0(sec) May 17 02:04:30 client-5 xinetd[3391]: START: shell pid=31789 from=192.168.4.5 May 17 02:04:31 client-5 rshd[31791]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); mpts=\$(mount | grep -w -c /mnt/lustre); if [ \$running -ne \$mpts ]; then echo \$(hostname) env are INSANE!; exit 1; fi");echo XXRETCODE:$?' May 17 02:04:31 client-5 xinetd[3391]: EXIT: shell status=0 pid=31789 duration=1(sec) May 17 02:04:31 client-5 xinetd[3391]: START: shell pid=31837 from=192.168.4.5 May 17 02:04:31 client-5 rshd[31838]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); mpts=\$(mount | grep -w -c /mnt/lustre2); if [ \$running -ne \$mpts ]; then echo \$(hostname) env are INSANE!; exit 1; fi");echo XXRETCODE:$?' May 17 02:04:31 client-5 xinetd[3391]: EXIT: shell status=0 pid=31837 duration=0(sec) May 17 02:04:31 client-5 kernel: Lustre: 31880:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 02:04:31 client-5 kernel: Lustre: 31880:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 3 previous similar messages May 17 02:04:35 client-5 kernel: Lustre: 32210:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 02:04:35 client-5 kernel: Lustre: 32210:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 5 previous similar messages May 17 02:04:35 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 02:04:35 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 02:04:35 client-5 kernel: Lustre: Increasing default stripe size to min 1048576 May 17 02:04:45 client-5 kernel: Lustre: 32210:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-OST0000-osc-ffff810326170400->192.168.4.129@o2ib netid 50000: select flavor null May 17 02:04:45 client-5 kernel: Lustre: 32210:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 1 previous similar message May 17 02:04:45 client-5 kernel: Lustre: Client lustre-client has started May 17 02:04:45 client-5 kernel: Lustre: 32231:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 02:04:45 client-5 kernel: Lustre: 32231:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message May 17 02:04:45 client-5 xinetd[3391]: START: shell pid=32241 from=192.168.4.5 May 17 02:04:45 client-5 rshd[32242]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c " running=\$(mount | grep -c /mnt/lustre' '); rc=0; if [ \$running -eq 0 ] ; then mkdir -p /mnt/lustre; mount -t lustre -o flock,user_xattr,acl, fat-intel-1-ib@o2ib:/lustre /mnt/lustre; rc=\$?; fi; exit \$rc");echo XXRETCODE:$?' May 17 02:04:46 client-5 xinetd[3391]: EXIT: shell status=0 pid=32241 duration=1(sec) May 17 02:04:46 client-5 xinetd[3391]: START: shell pid=32286 from=192.168.4.5 May 17 02:04:46 client-5 rshd[32287]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mount | grep -w /mnt/lustre");echo XXRETCODE:$?' May 17 02:04:46 client-5 xinetd[3391]: START: shell pid=32330 from=192.168.4.5 May 17 02:04:46 client-5 xinetd[3391]: EXIT: shell status=0 pid=32286 duration=0(sec) May 17 02:04:46 client-5 rshd[32331]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param debug=\"0x33f0406\"; lctl set_param subsystem_debug=\"0xffb7e3ff\"; lctl set_param debug_mb=32;");echo XXRETCODE:$?' May 17 02:04:46 client-5 xinetd[3391]: EXIT: shell status=0 pid=32330 duration=0(sec) May 17 02:04:46 client-5 xinetd[3391]: START: shell pid=32499 from=192.168.4.5 May 17 02:04:46 client-5 rshd[32501]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark Using TIMEOUT=20");echo XXRETCODE:$?' May 17 02:04:46 client-5 kernel: Lustre: DEBUG MARKER: Using TIMEOUT=20 May 17 02:04:46 client-5 xinetd[3391]: EXIT: shell status=0 pid=32499 duration=0(sec) May 17 02:04:47 client-5 kernel: LustreError: 32554:0:(quota_ctl.c:328:client_quota_ctl()) ptlrpc_queue_wait failed, rc: -114 May 17 02:04:52 client-5 xinetd[3391]: START: shell pid=32610 from=192.168.4.5 May 17 02:04:52 client-5 rshd[32613]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 02:04:52 client-5 xinetd[3391]: EXIT: shell status=0 pid=32610 duration=0(sec) May 17 02:04:52 client-5 xinetd[3391]: START: shell pid=317 from=192.168.4.5 May 17 02:04:52 client-5 rshd[319]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 140: Check reasonable stack depth \(shouldn\'t LBUG\) ==================================== 02:04:52 \(1305623092\)");echo XXRETCODE:$?' May 17 02:04:52 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 140: Check reasonable stack depth (shouldn't LBUG) ==================================== 02:04:52 (1305623092) May 17 02:04:52 client-5 xinetd[3391]: EXIT: shell status=0 pid=317 duration=0(sec) May 17 02:04:52 client-5 xinetd[3391]: START: shell pid=452 from=192.168.4.5 May 17 02:04:52 client-5 rshd[453]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 02:04:53 client-5 xinetd[3391]: EXIT: shell status=0 pid=452 duration=1(sec) May 17 02:04:53 client-5 xinetd[3391]: START: shell pid=636 from=192.168.4.5 May 17 02:04:53 client-5 rshd[638]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 150: truncate\/append tests ============================================================ 02:04:53 \(1305623093\)");echo XXRETCODE:$?' May 17 02:04:53 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 150: truncate/append tests ============================================================ 02:04:53 (1305623093) May 17 02:04:53 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 02:04:53 client-5 xinetd[3391]: EXIT: shell status=0 pid=636 duration=0(sec) May 17 02:04:53 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 02:04:53 client-5 kernel: LustreError: 710:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 02:04:53 client-5 kernel: LustreError: 710:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Skipped 1 previous similar message May 17 02:04:53 client-5 kernel: LustreError: 710:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 02:04:53 client-5 kernel: LustreError: 710:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) Skipped 1 previous similar message May 17 02:04:53 client-5 kernel: Lustre: client ffff810326170400 umount complete May 17 02:04:54 client-5 kernel: Lustre: 720:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 02:04:54 client-5 kernel: Lustre: 720:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 5 previous similar messages May 17 02:04:54 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 02:04:54 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 02:04:54 client-5 kernel: Lustre: Increasing default stripe size to min 1048576 May 17 02:04:54 client-5 kernel: Lustre: Client lustre-client has started May 17 02:04:54 client-5 kernel: Lustre: 737:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 02:04:54 client-5 kernel: Lustre: 737:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 3 previous similar messages May 17 02:04:54 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 02:04:54 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 02:04:54 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 02:04:54 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 02:04:54 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 02:04:54 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 02:04:54 client-5 xinetd[3391]: START: shell pid=846 from=192.168.4.5 May 17 02:04:54 client-5 rshd[847]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 02:04:54 client-5 xinetd[3391]: EXIT: shell status=0 pid=846 duration=0(sec) May 17 02:04:54 client-5 xinetd[3391]: START: shell pid=1018 from=192.168.4.5 May 17 02:04:54 client-5 rshd[1019]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 151: test cache on oss and controls =================================================== 02:04:54 \(1305623094\)");echo XXRETCODE:$?' May 17 02:04:54 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 151: test cache on oss and controls =================================================== 02:04:54 (1305623094) May 17 02:04:54 client-5 xinetd[3391]: EXIT: shell status=0 pid=1018 duration=0(sec) May 17 02:04:55 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 02:04:55 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 02:04:55 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 02:04:55 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 02:04:55 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 02:04:55 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 02:04:56 client-5 xinetd[3391]: START: shell pid=1401 from=192.168.4.5 May 17 02:04:56 client-5 rshd[1404]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 02:04:56 client-5 xinetd[3391]: EXIT: shell status=0 pid=1401 duration=0(sec) May 17 02:04:56 client-5 xinetd[3391]: START: shell pid=1576 from=192.168.4.5 May 17 02:04:56 client-5 rshd[1577]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 152: test read\/write with enomem ====================================================== 02:04:56 \(1305623096\)");echo XXRETCODE:$?' May 17 02:04:56 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 152: test read/write with enomem ====================================================== 02:04:56 (1305623096) May 17 02:04:56 client-5 xinetd[3391]: EXIT: shell status=0 pid=1576 duration=0(sec) May 17 02:04:56 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 02:04:56 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 02:04:56 client-5 xinetd[3391]: START: shell pid=1690 from=192.168.4.5 May 17 02:04:56 client-5 rshd[1692]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 02:04:56 client-5 xinetd[3391]: EXIT: shell status=0 pid=1690 duration=0(sec) May 17 02:04:57 client-5 xinetd[3391]: START: shell pid=1865 from=192.168.4.5 May 17 02:04:57 client-5 rshd[1866]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 153: test if fdatasync does not crash ================================================= 02:04:57 \(1305623097\)");echo XXRETCODE:$?' May 17 02:04:57 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 153: test if fdatasync does not crash ================================================= 02:04:57 (1305623097) May 17 02:04:57 client-5 xinetd[3391]: EXIT: shell status=0 pid=1865 duration=0(sec) May 17 02:04:57 client-5 xinetd[3391]: START: shell pid=1960 from=192.168.4.5 May 17 02:04:57 client-5 rshd[1963]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 02:04:57 client-5 xinetd[3391]: EXIT: shell status=0 pid=1960 duration=0(sec) May 17 02:04:57 client-5 xinetd[3391]: START: shell pid=2135 from=192.168.4.5 May 17 02:04:57 client-5 rshd[2138]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 154: Opening a file by FID ============================================================ 02:04:57 \(1305623097\)");echo XXRETCODE:$?' May 17 02:04:57 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 154: Opening a file by FID ============================================================ 02:04:57 (1305623097) May 17 02:04:57 client-5 xinetd[3391]: EXIT: shell status=0 pid=2135 duration=0(sec) May 17 02:04:57 client-5 kernel: LustreError: 2072:0:(dir.c:379:ll_get_dir_page()) read cache page: [0x200000002:0x2:0x0] at 0: rc -1 May 17 02:04:57 client-5 kernel: LustreError: 2072:0:(dir.c:549:ll_readdir()) error reading dir [0x200000002:0x2:0x0] at 0: rc -1 May 17 02:04:58 client-5 xinetd[3391]: START: shell pid=2235 from=192.168.4.5 May 17 02:04:58 client-5 rshd[2236]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 02:04:58 client-5 xinetd[3391]: EXIT: shell status=0 pid=2235 duration=0(sec) May 17 02:04:58 client-5 xinetd[3391]: START: shell pid=2426 from=192.168.4.5 May 17 02:04:58 client-5 rshd[2427]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 155a: Verification of correctness: read cache:on write_cache:on ======================= 02:04:58 \(1305623098\)");echo XXRETCODE:$?' May 17 02:04:58 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 155a: Verification of correctness: read cache:on write_cache:on ======================= 02:04:58 (1305623098) May 17 02:04:58 client-5 xinetd[3391]: EXIT: shell status=0 pid=2426 duration=0(sec) May 17 02:04:58 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 02:04:58 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 02:04:58 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 02:04:58 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 02:04:59 client-5 xinetd[3391]: START: shell pid=2667 from=192.168.4.5 May 17 02:04:59 client-5 rshd[2669]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark cache size on OSS is 294912 KB");echo XXRETCODE:$?' May 17 02:04:59 client-5 kernel: Lustre: DEBUG MARKER: cache size on OSS is 294912 KB May 17 02:04:59 client-5 xinetd[3391]: EXIT: shell status=0 pid=2667 duration=0(sec) May 17 02:04:59 client-5 xinetd[3391]: START: shell pid=2761 from=192.168.4.5 May 17 02:04:59 client-5 rshd[2762]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark large file size is 589824 KB");echo XXRETCODE:$?' May 17 02:04:59 client-5 kernel: Lustre: DEBUG MARKER: large file size is 589824 KB May 17 02:04:59 client-5 xinetd[3391]: EXIT: shell status=0 pid=2761 duration=0(sec) May 17 02:04:59 client-5 xinetd[3391]: START: shell pid=2853 from=192.168.4.5 May 17 02:04:59 client-5 rshd[2855]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark min available OST size is 8904820 KB");echo XXRETCODE:$?' May 17 02:04:59 client-5 kernel: Lustre: DEBUG MARKER: min available OST size is 8904820 KB May 17 02:04:59 client-5 xinetd[3391]: EXIT: shell status=0 pid=2853 duration=0(sec) May 17 02:04:59 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 02:04:59 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 02:06:33 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 02:06:34 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 02:06:37 client-5 xinetd[3391]: START: shell pid=2990 from=192.168.4.5 May 17 02:06:37 client-5 rshd[2991]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 02:06:37 client-5 xinetd[3391]: EXIT: shell status=0 pid=2990 duration=0(sec) May 17 02:06:37 client-5 xinetd[3391]: START: shell pid=3181 from=192.168.4.5 May 17 02:06:38 client-5 rshd[3182]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 155b: Verification of correctness: read cache:on write_cache:off ====================== 02:06:37 \(1305623197\)");echo XXRETCODE:$?' May 17 02:06:38 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 155b: Verification of correctness: read cache:on write_cache:off ====================== 02:06:37 (1305623197) May 17 02:06:38 client-5 xinetd[3391]: EXIT: shell status=0 pid=3181 duration=1(sec) May 17 02:06:38 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 02:06:38 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 02:06:38 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 02:06:38 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 02:06:38 client-5 xinetd[3391]: START: shell pid=3445 from=192.168.4.5 May 17 02:06:38 client-5 rshd[3446]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark cache size on OSS is 294912 KB");echo XXRETCODE:$?' May 17 02:06:38 client-5 kernel: Lustre: DEBUG MARKER: cache size on OSS is 294912 KB May 17 02:06:38 client-5 xinetd[3391]: EXIT: shell status=0 pid=3445 duration=0(sec) May 17 02:06:38 client-5 xinetd[3391]: START: shell pid=3550 from=192.168.4.5 May 17 02:06:38 client-5 rshd[3551]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark large file size is 589824 KB");echo XXRETCODE:$?' May 17 02:06:38 client-5 kernel: Lustre: DEBUG MARKER: large file size is 589824 KB May 17 02:06:38 client-5 xinetd[3391]: EXIT: shell status=0 pid=3550 duration=0(sec) May 17 02:06:38 client-5 xinetd[3391]: START: shell pid=3647 from=192.168.4.5 May 17 02:06:38 client-5 rshd[3649]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark min available OST size is 8904820 KB");echo XXRETCODE:$?' May 17 02:06:39 client-5 kernel: Lustre: DEBUG MARKER: min available OST size is 8904820 KB May 17 02:06:39 client-5 xinetd[3391]: EXIT: shell status=0 pid=3647 duration=1(sec) May 17 02:06:39 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 02:06:39 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 02:08:14 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 02:08:15 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 02:08:22 client-5 xinetd[3391]: START: shell pid=3784 from=192.168.4.5 May 17 02:08:22 client-5 rshd[3786]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 02:08:22 client-5 xinetd[3391]: EXIT: shell status=0 pid=3784 duration=0(sec) May 17 02:08:22 client-5 xinetd[3391]: START: shell pid=3957 from=192.168.4.5 May 17 02:08:22 client-5 rshd[3958]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 155c: Verification of correctness: read cache:off write_cache:on ====================== 02:08:22 \(1305623302\)");echo XXRETCODE:$?' May 17 02:08:22 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 155c: Verification of correctness: read cache:off write_cache:on ====================== 02:08:22 (1305623302) May 17 02:08:22 client-5 xinetd[3391]: EXIT: shell status=0 pid=3957 duration=0(sec) May 17 02:08:22 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 02:08:22 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 02:08:22 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 02:08:22 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 02:08:23 client-5 xinetd[3391]: START: shell pid=4223 from=192.168.4.5 May 17 02:08:23 client-5 rshd[4225]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark cache size on OSS is 294912 KB");echo XXRETCODE:$?' May 17 02:08:23 client-5 kernel: Lustre: DEBUG MARKER: cache size on OSS is 294912 KB May 17 02:08:23 client-5 xinetd[3391]: EXIT: shell status=0 pid=4223 duration=0(sec) May 17 02:08:23 client-5 xinetd[3391]: START: shell pid=4318 from=192.168.4.5 May 17 02:08:23 client-5 rshd[4319]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark large file size is 589824 KB");echo XXRETCODE:$?' May 17 02:08:23 client-5 kernel: Lustre: DEBUG MARKER: large file size is 589824 KB May 17 02:08:23 client-5 xinetd[3391]: EXIT: shell status=0 pid=4318 duration=0(sec) May 17 02:08:23 client-5 xinetd[3391]: START: shell pid=4411 from=192.168.4.5 May 17 02:08:23 client-5 rshd[4412]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark min available OST size is 8904820 KB");echo XXRETCODE:$?' May 17 02:08:23 client-5 kernel: Lustre: DEBUG MARKER: min available OST size is 8904820 KB May 17 02:08:23 client-5 xinetd[3391]: EXIT: shell status=0 pid=4411 duration=0(sec) May 17 02:08:23 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 02:08:23 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 02:10:03 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 02:10:04 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 02:10:08 client-5 xinetd[3391]: START: shell pid=4551 from=192.168.4.5 May 17 02:10:08 client-5 rshd[4552]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 02:10:08 client-5 xinetd[3391]: EXIT: shell status=0 pid=4551 duration=0(sec) May 17 02:10:08 client-5 xinetd[3391]: START: shell pid=4723 from=192.168.4.5 May 17 02:10:08 client-5 rshd[4724]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 155d: Verification of correctness: read cache:off write_cache:off ===================== 02:10:08 \(1305623408\)");echo XXRETCODE:$?' May 17 02:10:08 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 155d: Verification of correctness: read cache:off write_cache:off ===================== 02:10:08 (1305623408) May 17 02:10:08 client-5 xinetd[3391]: EXIT: shell status=0 pid=4723 duration=0(sec) May 17 02:10:08 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 02:10:08 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 02:10:08 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 02:10:08 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 02:10:09 client-5 xinetd[3391]: START: shell pid=4959 from=192.168.4.5 May 17 02:10:09 client-5 rshd[4960]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark cache size on OSS is 294912 KB");echo XXRETCODE:$?' May 17 02:10:09 client-5 kernel: Lustre: DEBUG MARKER: cache size on OSS is 294912 KB May 17 02:10:09 client-5 xinetd[3391]: EXIT: shell status=0 pid=4959 duration=0(sec) May 17 02:10:09 client-5 xinetd[3391]: START: shell pid=5052 from=192.168.4.5 May 17 02:10:09 client-5 rshd[5053]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark large file size is 589824 KB");echo XXRETCODE:$?' May 17 02:10:09 client-5 kernel: Lustre: DEBUG MARKER: large file size is 589824 KB May 17 02:10:09 client-5 xinetd[3391]: EXIT: shell status=0 pid=5052 duration=0(sec) May 17 02:10:09 client-5 xinetd[3391]: START: shell pid=5143 from=192.168.4.5 May 17 02:10:09 client-5 rshd[5146]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark min available OST size is 8904820 KB");echo XXRETCODE:$?' May 17 02:10:09 client-5 kernel: Lustre: DEBUG MARKER: min available OST size is 8904820 KB May 17 02:10:09 client-5 xinetd[3391]: EXIT: shell status=0 pid=5143 duration=0(sec) May 17 02:10:09 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 02:10:09 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 02:11:51 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 02:11:52 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 02:12:00 client-5 xinetd[3391]: START: shell pid=5284 from=192.168.4.5 May 17 02:12:00 client-5 rshd[5285]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 02:12:00 client-5 xinetd[3391]: EXIT: shell status=0 pid=5284 duration=0(sec) May 17 02:12:00 client-5 xinetd[3391]: START: shell pid=5456 from=192.168.4.5 May 17 02:12:00 client-5 rshd[5457]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 156: Verification of tunables ========================================================= 02:12:00 \(1305623520\)");echo XXRETCODE:$?' May 17 02:12:00 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 156: Verification of tunables ========================================================= 02:12:00 (1305623520) May 17 02:12:00 client-5 xinetd[3391]: EXIT: shell status=0 pid=5456 duration=0(sec) May 17 02:12:00 client-5 xinetd[3391]: START: shell pid=5548 from=192.168.4.5 May 17 02:12:00 client-5 rshd[5550]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark Turn on read and write cache");echo XXRETCODE:$?' May 17 02:12:00 client-5 kernel: Lustre: DEBUG MARKER: Turn on read and write cache May 17 02:12:00 client-5 xinetd[3391]: EXIT: shell status=0 pid=5548 duration=0(sec) May 17 02:12:01 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 02:12:01 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 02:12:01 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 02:12:01 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 02:12:01 client-5 xinetd[3391]: START: shell pid=5742 from=192.168.4.5 May 17 02:12:01 client-5 rshd[5743]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark Write data and read it back.");echo XXRETCODE:$?' May 17 02:12:01 client-5 kernel: Lustre: DEBUG MARKER: Write data and read it back. May 17 02:12:01 client-5 xinetd[3391]: EXIT: shell status=0 pid=5742 duration=0(sec) May 17 02:12:01 client-5 xinetd[3391]: START: shell pid=5833 from=192.168.4.5 May 17 02:12:01 client-5 rshd[5836]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark Read should be satisfied from the cache.");echo XXRETCODE:$?' May 17 02:12:01 client-5 kernel: Lustre: DEBUG MARKER: Read should be satisfied from the cache. May 17 02:12:01 client-5 xinetd[3391]: EXIT: shell status=0 pid=5833 duration=0(sec) May 17 02:12:01 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 02:12:01 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 02:12:01 client-5 xinetd[3391]: START: shell pid=6024 from=192.168.4.5 May 17 02:12:02 client-5 rshd[6025]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark cache hits:: before: 294940, after: 294943");echo XXRETCODE:$?' May 17 02:12:02 client-5 kernel: Lustre: DEBUG MARKER: cache hits:: before: 294940, after: 294943 May 17 02:12:02 client-5 xinetd[3391]: EXIT: shell status=0 pid=6024 duration=1(sec) May 17 02:12:02 client-5 xinetd[3391]: START: shell pid=6117 from=192.168.4.5 May 17 02:12:02 client-5 rshd[6118]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark Read again\; it should be satisfied from the cache.");echo XXRETCODE:$?' May 17 02:12:02 client-5 kernel: Lustre: DEBUG MARKER: Read again; it should be satisfied from the cache. May 17 02:12:02 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 02:12:02 client-5 xinetd[3391]: EXIT: shell status=0 pid=6117 duration=0(sec) May 17 02:12:02 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 02:12:02 client-5 xinetd[3391]: START: shell pid=6266 from=192.168.4.5 May 17 02:12:02 client-5 rshd[6267]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark cache hits:: before: 294943, after: 294946");echo XXRETCODE:$?' May 17 02:12:02 client-5 kernel: Lustre: DEBUG MARKER: cache hits:: before: 294943, after: 294946 May 17 02:12:02 client-5 xinetd[3391]: EXIT: shell status=0 pid=6266 duration=0(sec) May 17 02:12:02 client-5 xinetd[3391]: START: shell pid=6359 from=192.168.4.5 May 17 02:12:02 client-5 rshd[6360]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark Turn off the read cache and turn on the write cache");echo XXRETCODE:$?' May 17 02:12:02 client-5 kernel: Lustre: DEBUG MARKER: Turn off the read cache and turn on the write cache May 17 02:12:02 client-5 xinetd[3391]: EXIT: shell status=0 pid=6359 duration=0(sec) May 17 02:12:02 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 02:12:02 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 02:12:03 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 02:12:03 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 02:12:03 client-5 xinetd[3391]: START: shell pid=6552 from=192.168.4.5 May 17 02:12:03 client-5 rshd[6553]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark Read again\; it should be satisfied from the cache.");echo XXRETCODE:$?' May 17 02:12:03 client-5 kernel: Lustre: DEBUG MARKER: Read again; it should be satisfied from the cache. May 17 02:12:03 client-5 xinetd[3391]: EXIT: shell status=0 pid=6552 duration=0(sec) May 17 02:12:03 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 02:12:03 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 02:12:03 client-5 xinetd[3391]: START: shell pid=6740 from=192.168.4.5 May 17 02:12:03 client-5 rshd[6741]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark cache hits:: before: 294946, after: 294949");echo XXRETCODE:$?' May 17 02:12:03 client-5 kernel: Lustre: DEBUG MARKER: cache hits:: before: 294946, after: 294949 May 17 02:12:03 client-5 xinetd[3391]: EXIT: shell status=0 pid=6740 duration=0(sec) May 17 02:12:03 client-5 xinetd[3391]: START: shell pid=6832 from=192.168.4.5 May 17 02:12:03 client-5 rshd[6834]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark Read again\; it should not be satisfied from the cache.");echo XXRETCODE:$?' May 17 02:12:03 client-5 kernel: Lustre: DEBUG MARKER: Read again; it should not be satisfied from the cache. May 17 02:12:03 client-5 xinetd[3391]: EXIT: shell status=0 pid=6832 duration=0(sec) May 17 02:12:03 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 02:12:03 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 02:12:04 client-5 xinetd[3391]: START: shell pid=6981 from=192.168.4.5 May 17 02:12:04 client-5 rshd[6982]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark cache hits:: before: 294949, after: 294949");echo XXRETCODE:$?' May 17 02:12:04 client-5 kernel: Lustre: DEBUG MARKER: cache hits:: before: 294949, after: 294949 May 17 02:12:04 client-5 xinetd[3391]: EXIT: shell status=0 pid=6981 duration=0(sec) May 17 02:12:04 client-5 xinetd[3391]: START: shell pid=7074 from=192.168.4.5 May 17 02:12:04 client-5 rshd[7075]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark Write data and read it back.");echo XXRETCODE:$?' May 17 02:12:04 client-5 kernel: Lustre: DEBUG MARKER: Write data and read it back. May 17 02:12:04 client-5 xinetd[3391]: EXIT: shell status=0 pid=7074 duration=0(sec) May 17 02:12:04 client-5 xinetd[3391]: START: shell pid=7167 from=192.168.4.5 May 17 02:12:04 client-5 rshd[7168]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark Read should be satisfied from the cache.");echo XXRETCODE:$?' May 17 02:12:04 client-5 kernel: Lustre: DEBUG MARKER: Read should be satisfied from the cache. May 17 02:12:04 client-5 xinetd[3391]: EXIT: shell status=0 pid=7167 duration=0(sec) May 17 02:12:04 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 02:12:04 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 02:12:04 client-5 xinetd[3391]: START: shell pid=7356 from=192.168.4.5 May 17 02:12:04 client-5 rshd[7357]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark cache hits:: before: 294949, after: 294952");echo XXRETCODE:$?' May 17 02:12:05 client-5 kernel: Lustre: DEBUG MARKER: cache hits:: before: 294949, after: 294952 May 17 02:12:05 client-5 xinetd[3391]: EXIT: shell status=0 pid=7356 duration=1(sec) May 17 02:12:05 client-5 xinetd[3391]: START: shell pid=7449 from=192.168.4.5 May 17 02:12:05 client-5 rshd[7450]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark Read again\; it should not be satisfied from the cache.");echo XXRETCODE:$?' May 17 02:12:05 client-5 kernel: Lustre: DEBUG MARKER: Read again; it should not be satisfied from the cache. May 17 02:12:05 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 02:12:05 client-5 xinetd[3391]: EXIT: shell status=0 pid=7449 duration=0(sec) May 17 02:12:05 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 02:12:05 client-5 xinetd[3391]: START: shell pid=7597 from=192.168.4.5 May 17 02:12:05 client-5 rshd[7598]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark cache hits:: before: 294952, after: 294952");echo XXRETCODE:$?' May 17 02:12:05 client-5 kernel: Lustre: DEBUG MARKER: cache hits:: before: 294952, after: 294952 May 17 02:12:05 client-5 xinetd[3391]: EXIT: shell status=0 pid=7597 duration=0(sec) May 17 02:12:05 client-5 xinetd[3391]: START: shell pid=7690 from=192.168.4.5 May 17 02:12:05 client-5 rshd[7691]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark Turn off read and write cache");echo XXRETCODE:$?' May 17 02:12:05 client-5 kernel: Lustre: DEBUG MARKER: Turn off read and write cache May 17 02:12:05 client-5 xinetd[3391]: EXIT: shell status=0 pid=7690 duration=0(sec) May 17 02:12:05 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 02:12:05 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 02:12:06 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 02:12:06 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 02:12:06 client-5 xinetd[3391]: START: shell pid=7882 from=192.168.4.5 May 17 02:12:06 client-5 rshd[7884]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark Write data and read it back");echo XXRETCODE:$?' May 17 02:12:06 client-5 kernel: Lustre: DEBUG MARKER: Write data and read it back May 17 02:12:06 client-5 xinetd[3391]: EXIT: shell status=0 pid=7882 duration=0(sec) May 17 02:12:06 client-5 xinetd[3391]: START: shell pid=7976 from=192.168.4.5 May 17 02:12:06 client-5 rshd[7977]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark It should not be satisfied from the cache.");echo XXRETCODE:$?' May 17 02:12:06 client-5 kernel: Lustre: DEBUG MARKER: It should not be satisfied from the cache. May 17 02:12:06 client-5 xinetd[3391]: EXIT: shell status=0 pid=7976 duration=0(sec) May 17 02:12:06 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 02:12:06 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 02:12:06 client-5 xinetd[3391]: START: shell pid=8166 from=192.168.4.5 May 17 02:12:06 client-5 rshd[8167]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark cache hits:: before: 294952, after: 294952");echo XXRETCODE:$?' May 17 02:12:06 client-5 kernel: Lustre: DEBUG MARKER: cache hits:: before: 294952, after: 294952 May 17 02:12:06 client-5 xinetd[3391]: EXIT: shell status=0 pid=8166 duration=0(sec) May 17 02:12:06 client-5 xinetd[3391]: START: shell pid=8259 from=192.168.4.5 May 17 02:12:06 client-5 rshd[8260]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark Turn on the read cache and turn off the write cache");echo XXRETCODE:$?' May 17 02:12:07 client-5 kernel: Lustre: DEBUG MARKER: Turn on the read cache and turn off the write cache May 17 02:12:07 client-5 xinetd[3391]: EXIT: shell status=0 pid=8259 duration=1(sec) May 17 02:12:07 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 02:12:07 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 02:12:07 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 02:12:07 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 02:12:07 client-5 xinetd[3391]: START: shell pid=8450 from=192.168.4.5 May 17 02:12:07 client-5 rshd[8453]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark Write data and read it back");echo XXRETCODE:$?' May 17 02:12:07 client-5 kernel: Lustre: DEBUG MARKER: Write data and read it back May 17 02:12:07 client-5 xinetd[3391]: EXIT: shell status=0 pid=8450 duration=0(sec) May 17 02:12:07 client-5 xinetd[3391]: START: shell pid=8544 from=192.168.4.5 May 17 02:12:07 client-5 rshd[8546]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark It should not be satisfied from the cache.");echo XXRETCODE:$?' May 17 02:12:07 client-5 kernel: Lustre: DEBUG MARKER: It should not be satisfied from the cache. May 17 02:12:07 client-5 xinetd[3391]: EXIT: shell status=0 pid=8544 duration=0(sec) May 17 02:12:07 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 02:12:07 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 02:12:08 client-5 xinetd[3391]: START: shell pid=8733 from=192.168.4.5 May 17 02:12:08 client-5 rshd[8736]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark cache hits:: before: 294952, after: 294952");echo XXRETCODE:$?' May 17 02:12:08 client-5 kernel: Lustre: DEBUG MARKER: cache hits:: before: 294952, after: 294952 May 17 02:12:08 client-5 xinetd[3391]: EXIT: shell status=0 pid=8733 duration=0(sec) May 17 02:12:08 client-5 xinetd[3391]: START: shell pid=8828 from=192.168.4.5 May 17 02:12:08 client-5 rshd[8829]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark Read again\; it should be satisfied from the cache.");echo XXRETCODE:$?' May 17 02:12:08 client-5 kernel: Lustre: DEBUG MARKER: Read again; it should be satisfied from the cache. May 17 02:12:08 client-5 xinetd[3391]: EXIT: shell status=0 pid=8828 duration=0(sec) May 17 02:12:08 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 02:12:08 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 02:12:08 client-5 xinetd[3391]: START: shell pid=9016 from=192.168.4.5 May 17 02:12:08 client-5 rshd[9017]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark cache hits:: before: 294952, after: 294955");echo XXRETCODE:$?' May 17 02:12:08 client-5 kernel: Lustre: DEBUG MARKER: cache hits:: before: 294952, after: 294955 May 17 02:12:08 client-5 xinetd[3391]: EXIT: shell status=0 pid=9016 duration=0(sec) May 17 02:12:08 client-5 xinetd[3391]: START: shell pid=9108 from=192.168.4.5 May 17 02:12:08 client-5 rshd[9109]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 02:12:08 client-5 xinetd[3391]: EXIT: shell status=0 pid=9108 duration=0(sec) May 17 02:12:09 client-5 xinetd[3391]: START: shell pid=9280 from=192.168.4.5 May 17 02:12:09 client-5 rshd[9281]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 160: changelog sanity ================================================================= 02:12:09 \(1305623529\)");echo XXRETCODE:$?' May 17 02:12:09 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 160: changelog sanity ================================================================= 02:12:09 (1305623529) May 17 02:12:09 client-5 xinetd[3391]: EXIT: shell status=0 pid=9280 duration=0(sec) May 17 02:12:10 client-5 xinetd[3391]: START: shell pid=9544 from=192.168.4.5 May 17 02:12:10 client-5 rshd[9545]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 02:12:10 client-5 xinetd[3391]: EXIT: shell status=0 pid=9544 duration=0(sec) May 17 02:12:11 client-5 xinetd[3391]: START: shell pid=9716 from=192.168.4.5 May 17 02:12:11 client-5 rshd[9717]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 161: link ea sanity =================================================================== 02:12:11 \(1305623531\)");echo XXRETCODE:$?' May 17 02:12:11 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 161: link ea sanity =================================================================== 02:12:11 (1305623531) May 17 02:12:11 client-5 xinetd[3391]: EXIT: shell status=0 pid=9716 duration=0(sec) May 17 02:12:13 client-5 xinetd[3391]: START: shell pid=9832 from=192.168.4.5 May 17 02:12:13 client-5 rshd[9833]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 02:12:13 client-5 xinetd[3391]: EXIT: shell status=0 pid=9832 duration=0(sec) May 17 02:12:13 client-5 xinetd[3391]: START: shell pid=10003 from=192.168.4.5 May 17 02:12:13 client-5 rshd[10005]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 162: path lookup sanity =============================================================== 02:12:13 \(1305623533\)");echo XXRETCODE:$?' May 17 02:12:13 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 162: path lookup sanity =============================================================== 02:12:13 (1305623533) May 17 02:12:13 client-5 xinetd[3391]: EXIT: shell status=0 pid=10003 duration=0(sec) May 17 02:12:13 client-5 xinetd[3391]: START: shell pid=10127 from=192.168.4.5 May 17 02:12:13 client-5 rshd[10128]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 02:12:13 client-5 xinetd[3391]: EXIT: shell status=0 pid=10127 duration=0(sec) May 17 02:12:14 client-5 xinetd[3391]: START: shell pid=10298 from=192.168.4.5 May 17 02:12:14 client-5 rshd[10300]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 163: kernel \<-\> userspace comms ======================================================= 02:12:14 \(1305623534\)");echo XXRETCODE:$?' May 17 02:12:14 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 163: kernel <-> userspace comms ======================================================= 02:12:14 (1305623534) May 17 02:12:14 client-5 xinetd[3391]: EXIT: shell status=0 pid=10298 duration=0(sec) May 17 02:12:15 client-5 xinetd[3391]: START: shell pid=10413 from=192.168.4.5 May 17 02:12:15 client-5 rshd[10415]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 02:12:15 client-5 xinetd[3391]: EXIT: shell status=0 pid=10413 duration=0(sec) May 17 02:12:15 client-5 xinetd[3391]: START: shell pid=10584 from=192.168.4.5 May 17 02:12:15 client-5 rshd[10587]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 169: parallel read and truncate should not deadlock =================================== 02:12:15 \(1305623535\)");echo XXRETCODE:$?' May 17 02:12:16 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 169: parallel read and truncate should not deadlock =================================== 02:12:15 (1305623535) May 17 02:12:16 client-5 xinetd[3391]: EXIT: shell status=0 pid=10584 duration=1(sec) May 17 02:12:16 client-5 xinetd[3391]: START: shell pid=10679 from=192.168.4.5 May 17 02:12:16 client-5 rshd[10680]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark creating a 10 Mb file");echo XXRETCODE:$?' May 17 02:12:16 client-5 kernel: Lustre: DEBUG MARKER: creating a 10 Mb file May 17 02:12:16 client-5 xinetd[3391]: EXIT: shell status=0 pid=10679 duration=0(sec) May 17 02:12:16 client-5 xinetd[3391]: START: shell pid=10773 from=192.168.4.5 May 17 02:12:16 client-5 rshd[10774]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark starting reads");echo XXRETCODE:$?' May 17 02:12:16 client-5 kernel: Lustre: DEBUG MARKER: starting reads May 17 02:12:16 client-5 xinetd[3391]: EXIT: shell status=0 pid=10773 duration=0(sec) May 17 02:12:16 client-5 xinetd[3391]: START: shell pid=10867 from=192.168.4.5 May 17 02:12:16 client-5 rshd[10868]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark truncating the file");echo XXRETCODE:$?' May 17 02:12:16 client-5 kernel: Lustre: DEBUG MARKER: truncating the file May 17 02:12:16 client-5 xinetd[3391]: EXIT: shell status=0 pid=10867 duration=0(sec) May 17 02:12:16 client-5 xinetd[3391]: START: shell pid=10961 from=192.168.4.5 May 17 02:12:16 client-5 rshd[10962]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark killing dd");echo XXRETCODE:$?' May 17 02:12:16 client-5 kernel: Lustre: DEBUG MARKER: killing dd May 17 02:12:16 client-5 xinetd[3391]: EXIT: shell status=0 pid=10961 duration=0(sec) May 17 02:12:16 client-5 xinetd[3391]: START: shell pid=11053 from=192.168.4.5 May 17 02:12:16 client-5 rshd[11055]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark removing the temporary file");echo XXRETCODE:$?' May 17 02:12:17 client-5 kernel: Lustre: DEBUG MARKER: removing the temporary file May 17 02:12:17 client-5 xinetd[3391]: EXIT: shell status=0 pid=11053 duration=1(sec) May 17 02:12:17 client-5 xinetd[3391]: START: shell pid=11146 from=192.168.4.5 May 17 02:12:17 client-5 rshd[11147]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 02:12:17 client-5 xinetd[3391]: EXIT: shell status=0 pid=11146 duration=0(sec) May 17 02:12:17 client-5 xinetd[3391]: START: shell pid=11318 from=192.168.4.5 May 17 02:12:17 client-5 rshd[11319]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 170: test lctl df to handle corrupted log ============================================= 02:12:17 \(1305623537\)");echo XXRETCODE:$?' May 17 02:12:17 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 170: test lctl df to handle corrupted log ============================================= 02:12:17 (1305623537) May 17 02:12:17 client-5 xinetd[3391]: EXIT: shell status=0 pid=11318 duration=0(sec) May 17 02:12:17 client-5 kernel: Lustre: debug daemon will attempt to start writing to /tmp/f170_log_good (512000kB max) May 17 02:12:17 client-5 kernel: Lustre: shutting down debug daemon thread... May 17 02:12:18 client-5 kernel: Lustre: debug daemon will attempt to start writing to /tmp/f170_log_good (512000kB max) May 17 02:12:18 client-5 kernel: Lustre: shutting down debug daemon thread... May 17 02:12:19 client-5 xinetd[3391]: START: shell pid=11440 from=192.168.4.5 May 17 02:12:19 client-5 rshd[11442]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 02:12:19 client-5 xinetd[3391]: EXIT: shell status=0 pid=11440 duration=0(sec) May 17 02:12:20 client-5 xinetd[3391]: START: shell pid=11613 from=192.168.4.5 May 17 02:12:20 client-5 rshd[11614]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 171: test libcfs_debug_dumplog_thread stuck in do_exit\(\) ============================== 02:12:19 \(1305623539\)");echo XXRETCODE:$?' May 17 02:12:20 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 171: test libcfs_debug_dumplog_thread stuck in do_exit() ============================== 02:12:19 (1305623539) May 17 02:12:20 client-5 xinetd[3391]: EXIT: shell status=0 pid=11613 duration=0(sec) May 17 02:12:20 client-5 kernel: LustreError: 11651:0:(lvfs_lib.c:150:__obd_fail_timeout_set()) obd_fail_timeout id 50e sleeping for 3000ms May 17 02:12:23 client-5 kernel: LustreError: 11651:0:(lvfs_lib.c:154:__obd_fail_timeout_set()) obd_fail_timeout id 50e awake May 17 02:12:23 client-5 kernel: LustreError: dumping log to /tmp/lustre-log.1305623543.11651 May 17 02:12:23 client-5 xinetd[3391]: START: shell pid=11716 from=192.168.4.5 May 17 02:12:23 client-5 rshd[11717]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 02:12:23 client-5 xinetd[3391]: EXIT: shell status=0 pid=11716 duration=0(sec) May 17 02:12:23 client-5 xinetd[3391]: START: shell pid=11888 from=192.168.4.5 May 17 02:12:23 client-5 rshd[11889]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 180a: test obdecho on osc ============================================================= 02:12:23 \(1305623543\)");echo XXRETCODE:$?' May 17 02:12:23 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 180a: test obdecho on osc ============================================================= 02:12:23 (1305623543) May 17 02:12:23 client-5 xinetd[3391]: EXIT: shell status=0 pid=11888 duration=0(sec) May 17 02:12:23 client-5 kernel: Lustre: Echo OBD driver; http://www.lustre.org/ May 17 02:12:23 client-5 kernel: Lustre: 11958:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-OST0000_osc->192.168.4.129@o2ib netid 50000: select flavor null May 17 02:12:23 client-5 kernel: Lustre: 11958:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 7 previous similar messages May 17 02:12:24 client-5 xinetd[3391]: START: shell pid=12053 from=192.168.4.5 May 17 02:12:24 client-5 rshd[12055]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 02:12:24 client-5 xinetd[3391]: EXIT: shell status=0 pid=12053 duration=0(sec) May 17 02:12:24 client-5 xinetd[3391]: START: shell pid=12226 from=192.168.4.5 May 17 02:12:24 client-5 rshd[12227]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 180b: test obdecho directly on obdfilter ============================================== 02:12:24 \(1305623544\)");echo XXRETCODE:$?' May 17 02:12:24 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 180b: test obdecho directly on obdfilter ============================================== 02:12:24 (1305623544) May 17 02:12:24 client-5 xinetd[3391]: EXIT: shell status=0 pid=12226 duration=0(sec) May 17 02:12:25 client-5 xinetd[3391]: START: shell pid=12395 from=192.168.4.5 May 17 02:12:25 client-5 rshd[12396]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 02:12:26 client-5 xinetd[3391]: EXIT: shell status=0 pid=12395 duration=1(sec) May 17 02:12:26 client-5 xinetd[3391]: START: shell pid=12567 from=192.168.4.5 May 17 02:12:26 client-5 rshd[12568]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 181: Test open-unlinked dir =========================================================== 02:12:26 \(1305623546\)");echo XXRETCODE:$?' May 17 02:12:26 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 181: Test open-unlinked dir =========================================================== 02:12:26 (1305623546) May 17 02:12:26 client-5 xinetd[3391]: EXIT: shell status=0 pid=12567 duration=0(sec) May 17 02:12:33 client-5 xinetd[3391]: START: shell pid=12667 from=192.168.4.5 May 17 02:12:33 client-5 rshd[12668]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 02:12:34 client-5 xinetd[3391]: EXIT: shell status=0 pid=12667 duration=1(sec) May 17 02:12:34 client-5 xinetd[3391]: START: shell pid=12840 from=192.168.4.5 May 17 02:12:34 client-5 rshd[12841]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 200a: Create new pool ================================================================= 02:12:34 \(1305623554\)");echo XXRETCODE:$?' May 17 02:12:34 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 200a: Create new pool ================================================================= 02:12:34 (1305623554) May 17 02:12:34 client-5 xinetd[3391]: EXIT: shell status=0 pid=12840 duration=0(sec) May 17 02:12:43 client-5 xinetd[3391]: START: shell pid=12970 from=192.168.4.5 May 17 02:12:43 client-5 rshd[12972]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 02:12:43 client-5 xinetd[3391]: EXIT: shell status=0 pid=12970 duration=0(sec) May 17 02:12:43 client-5 xinetd[3391]: START: shell pid=13143 from=192.168.4.5 May 17 02:12:44 client-5 rshd[13144]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 200b: Add targets to a pool =========================================================== 02:12:43 \(1305623563\)");echo XXRETCODE:$?' May 17 02:12:44 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 200b: Add targets to a pool =========================================================== 02:12:43 (1305623563) May 17 02:12:44 client-5 xinetd[3391]: EXIT: shell status=0 pid=13143 duration=1(sec) May 17 02:12:50 client-5 xinetd[3391]: START: shell pid=13258 from=192.168.4.5 May 17 02:12:50 client-5 rshd[13259]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 02:12:50 client-5 xinetd[3391]: EXIT: shell status=0 pid=13258 duration=0(sec) May 17 02:12:50 client-5 xinetd[3391]: START: shell pid=13430 from=192.168.4.5 May 17 02:12:50 client-5 rshd[13431]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 200c: Set pool on a directory ========================================================= 02:12:50 \(1305623570\)");echo XXRETCODE:$?' May 17 02:12:50 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 200c: Set pool on a directory ========================================================= 02:12:50 (1305623570) May 17 02:12:50 client-5 xinetd[3391]: EXIT: shell status=0 pid=13430 duration=0(sec) May 17 02:12:50 client-5 xinetd[3391]: START: shell pid=13531 from=192.168.4.5 May 17 02:12:50 client-5 rshd[13534]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 02:12:50 client-5 xinetd[3391]: EXIT: shell status=0 pid=13531 duration=0(sec) May 17 02:12:51 client-5 xinetd[3391]: START: shell pid=13705 from=192.168.4.5 May 17 02:12:51 client-5 rshd[13706]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 200d: Check pool on a directory ======================================================= 02:12:51 \(1305623571\)");echo XXRETCODE:$?' May 17 02:12:51 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 200d: Check pool on a directory ======================================================= 02:12:51 (1305623571) May 17 02:12:51 client-5 xinetd[3391]: EXIT: shell status=0 pid=13705 duration=0(sec) May 17 02:12:51 client-5 xinetd[3391]: START: shell pid=13799 from=192.168.4.5 May 17 02:12:51 client-5 rshd[13800]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 02:12:51 client-5 xinetd[3391]: EXIT: shell status=0 pid=13799 duration=0(sec) May 17 02:12:51 client-5 xinetd[3391]: START: shell pid=13970 from=192.168.4.5 May 17 02:12:51 client-5 rshd[13972]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 200e: Check files allocation from directory pool ====================================== 02:12:51 \(1305623571\)");echo XXRETCODE:$?' May 17 02:12:51 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 200e: Check files allocation from directory pool ====================================== 02:12:51 (1305623571) May 17 02:12:51 client-5 xinetd[3391]: EXIT: shell status=0 pid=13970 duration=0(sec) May 17 02:12:52 client-5 xinetd[3391]: START: shell pid=14298 from=192.168.4.5 May 17 02:12:52 client-5 rshd[14300]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 02:12:52 client-5 xinetd[3391]: EXIT: shell status=0 pid=14298 duration=0(sec) May 17 02:12:52 client-5 xinetd[3391]: START: shell pid=14471 from=192.168.4.5 May 17 02:12:52 client-5 rshd[14472]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 200f: Create files in a pool ========================================================== 02:12:52 \(1305623572\)");echo XXRETCODE:$?' May 17 02:12:52 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 200f: Create files in a pool ========================================================== 02:12:52 (1305623572) May 17 02:12:52 client-5 xinetd[3391]: EXIT: shell status=0 pid=14471 duration=0(sec) May 17 02:12:53 client-5 xinetd[3391]: START: shell pid=14727 from=192.168.4.5 May 17 02:12:53 client-5 rshd[14729]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 02:12:53 client-5 xinetd[3391]: EXIT: shell status=0 pid=14727 duration=0(sec) May 17 02:12:53 client-5 xinetd[3391]: START: shell pid=14900 from=192.168.4.5 May 17 02:12:53 client-5 rshd[14901]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 200g: lfs df a pool =================================================================== 02:12:53 \(1305623573\)");echo XXRETCODE:$?' May 17 02:12:53 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 200g: lfs df a pool =================================================================== 02:12:53 (1305623573) May 17 02:12:53 client-5 xinetd[3391]: EXIT: shell status=0 pid=14900 duration=0(sec) May 17 02:12:53 client-5 xinetd[3391]: START: shell pid=15001 from=192.168.4.5 May 17 02:12:53 client-5 rshd[15002]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 02:12:53 client-5 xinetd[3391]: EXIT: shell status=0 pid=15001 duration=0(sec) May 17 02:12:53 client-5 xinetd[3391]: START: shell pid=15173 from=192.168.4.5 May 17 02:12:53 client-5 rshd[15174]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 200h: Create files in a pool with relative pathname =================================== 02:12:53 \(1305623573\)");echo XXRETCODE:$?' May 17 02:12:54 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 200h: Create files in a pool with relative pathname =================================== 02:12:53 (1305623573) May 17 02:12:54 client-5 xinetd[3391]: EXIT: shell status=0 pid=15173 duration=1(sec) May 17 02:12:54 client-5 xinetd[3391]: START: shell pid=15265 from=192.168.4.5 May 17 02:12:54 client-5 rshd[15268]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 02:12:54 client-5 xinetd[3391]: EXIT: shell status=0 pid=15265 duration=0(sec) May 17 02:12:54 client-5 xinetd[3391]: START: shell pid=15439 from=192.168.4.5 May 17 02:12:54 client-5 rshd[15440]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 201a: Remove a target from a pool ===================================================== 02:12:54 \(1305623574\)");echo XXRETCODE:$?' May 17 02:12:54 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 201a: Remove a target from a pool ===================================================== 02:12:54 (1305623574) May 17 02:12:54 client-5 xinetd[3391]: EXIT: shell status=0 pid=15439 duration=0(sec) May 17 02:13:00 client-5 xinetd[3391]: START: shell pid=15552 from=192.168.4.5 May 17 02:13:00 client-5 rshd[15553]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 02:13:00 client-5 xinetd[3391]: EXIT: shell status=0 pid=15552 duration=0(sec) May 17 02:13:01 client-5 xinetd[3391]: START: shell pid=15724 from=192.168.4.5 May 17 02:13:01 client-5 rshd[15725]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 201b: Remove all targets from a pool ================================================== 02:13:01 \(1305623581\)");echo XXRETCODE:$?' May 17 02:13:01 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 201b: Remove all targets from a pool ================================================== 02:13:01 (1305623581) May 17 02:13:01 client-5 xinetd[3391]: EXIT: shell status=0 pid=15724 duration=0(sec) May 17 02:13:09 client-5 xinetd[3391]: START: shell pid=15845 from=192.168.4.5 May 17 02:13:09 client-5 rshd[15846]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 02:13:09 client-5 xinetd[3391]: EXIT: shell status=0 pid=15845 duration=0(sec) May 17 02:13:09 client-5 xinetd[3391]: START: shell pid=16017 from=192.168.4.5 May 17 02:13:10 client-5 rshd[16018]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 201c: Remove a pool =================================================================== 02:13:09 \(1305623589\)");echo XXRETCODE:$?' May 17 02:13:10 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 201c: Remove a pool =================================================================== 02:13:09 (1305623589) May 17 02:13:10 client-5 xinetd[3391]: EXIT: shell status=0 pid=16017 duration=1(sec) May 17 02:13:16 client-5 xinetd[3391]: START: shell pid=16136 from=192.168.4.5 May 17 02:13:16 client-5 rshd[16137]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 02:13:16 client-5 xinetd[3391]: EXIT: shell status=0 pid=16136 duration=0(sec) May 17 02:13:16 client-5 xinetd[3391]: START: shell pid=16316 from=192.168.4.5 May 17 02:13:16 client-5 rshd[16317]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 212: Sendfile test ==================================================================== 02:13:16 \(1305623596\)");echo XXRETCODE:$?' May 17 02:13:16 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 212: Sendfile test ==================================================================== 02:13:16 (1305623596) May 17 02:13:16 client-5 xinetd[3391]: EXIT: shell status=0 pid=16316 duration=0(sec) May 17 02:13:19 client-5 xinetd[3391]: START: shell pid=16412 from=192.168.4.5 May 17 02:13:19 client-5 rshd[16413]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 02:13:19 client-5 xinetd[3391]: EXIT: shell status=0 pid=16412 duration=0(sec) May 17 02:13:19 client-5 xinetd[3391]: START: shell pid=16584 from=192.168.4.5 May 17 02:13:19 client-5 rshd[16585]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 213: OSC lock completion and cancel race don\'t crash - bug 18829 ====================== 02:13:19 \(1305623599\)");echo XXRETCODE:$?' May 17 02:13:19 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 213: OSC lock completion and cancel race don't crash - bug 18829 ====================== 02:13:19 (1305623599) May 17 02:13:19 client-5 xinetd[3391]: EXIT: shell status=0 pid=16584 duration=0(sec) May 17 02:13:19 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 02:13:19 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 02:13:19 client-5 kernel: LustreError: 4123:0:(lvfs_lib.c:150:__obd_fail_timeout_set()) obd_fail_timeout id 40f sleeping for 10000ms May 17 02:13:19 client-5 kernel: LustreError: 4123:0:(lvfs_lib.c:154:__obd_fail_timeout_set()) obd_fail_timeout id 40f awake May 17 02:13:19 client-5 xinetd[3391]: START: shell pid=16694 from=192.168.4.5 May 17 02:13:19 client-5 rshd[16695]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 02:13:19 client-5 xinetd[3391]: EXIT: shell status=0 pid=16694 duration=0(sec) May 17 02:13:20 client-5 xinetd[3391]: START: shell pid=16866 from=192.168.4.5 May 17 02:13:20 client-5 rshd[16867]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 214: hash-indexed directory test - bug 20133 ========================================== 02:13:19 \(1305623599\)");echo XXRETCODE:$?' May 17 02:13:20 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 214: hash-indexed directory test - bug 20133 ========================================== 02:13:19 (1305623599) May 17 02:13:20 client-5 xinetd[3391]: EXIT: shell status=0 pid=16866 duration=0(sec) May 17 02:13:21 client-5 xinetd[3391]: START: shell pid=17303 from=192.168.4.5 May 17 02:13:21 client-5 rshd[17304]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 02:13:21 client-5 xinetd[3391]: EXIT: shell status=0 pid=17303 duration=0(sec) May 17 02:13:21 client-5 xinetd[3391]: START: shell pid=17475 from=192.168.4.5 May 17 02:13:21 client-5 rshd[17476]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 215: \/proc\/sys\/lnet exists and has proper content - bugs 18102, 21079, 21517 ========== 02:13:21 \(1305623601\)");echo XXRETCODE:$?' May 17 02:13:21 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 215: /proc/sys/lnet exists and has proper content - bugs 18102, 21079, 21517 ========== 02:13:21 (1305623601) May 17 02:13:21 client-5 xinetd[3391]: EXIT: shell status=0 pid=17475 duration=0(sec) May 17 02:13:22 client-5 xinetd[3391]: START: shell pid=17679 from=192.168.4.5 May 17 02:13:22 client-5 rshd[17680]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 02:13:22 client-5 xinetd[3391]: EXIT: shell status=0 pid=17679 duration=0(sec) May 17 02:13:22 client-5 xinetd[3391]: START: shell pid=17851 from=192.168.4.5 May 17 02:13:22 client-5 rshd[17852]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 216: check lockless direct write works and updates file size and kms correctly ======== 02:13:22 \(1305623602\)");echo XXRETCODE:$?' May 17 02:13:22 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 216: check lockless direct write works and updates file size and kms correctly ======== 02:13:22 (1305623602) May 17 02:13:22 client-5 xinetd[3391]: EXIT: shell status=0 pid=17851 duration=0(sec) May 17 02:13:25 client-5 xinetd[3391]: START: shell pid=18169 from=192.168.4.5 May 17 02:13:25 client-5 rshd[18170]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 02:13:25 client-5 xinetd[3391]: EXIT: shell status=0 pid=18169 duration=0(sec) May 17 02:13:25 client-5 xinetd[3391]: START: shell pid=18341 from=192.168.4.5 May 17 02:13:25 client-5 rshd[18342]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 217: check lctl ping for hostnames with hiphen \(\'-\'\) ================================== 02:13:25 \(1305623605\)");echo XXRETCODE:$?' May 17 02:13:25 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 217: check lctl ping for hostnames with hiphen ('-') ================================== 02:13:25 (1305623605) May 17 02:13:25 client-5 xinetd[3391]: EXIT: shell status=0 pid=18341 duration=0(sec) May 17 02:13:25 client-5 xinetd[3391]: START: shell pid=18477 from=192.168.4.5 May 17 02:13:25 client-5 rshd[18478]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 02:13:25 client-5 xinetd[3391]: EXIT: shell status=0 pid=18477 duration=0(sec) May 17 02:13:26 client-5 xinetd[3391]: START: shell pid=18649 from=192.168.4.5 May 17 02:13:26 client-5 rshd[18650]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 218: parallel read and truncate should not deadlock =================================== 02:13:26 \(1305623606\)");echo XXRETCODE:$?' May 17 02:13:26 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 218: parallel read and truncate should not deadlock =================================== 02:13:26 (1305623606) May 17 02:13:26 client-5 xinetd[3391]: EXIT: shell status=0 pid=18649 duration=0(sec) May 17 02:13:26 client-5 xinetd[3391]: START: shell pid=18742 from=192.168.4.5 May 17 02:13:26 client-5 rshd[18743]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark creating a 10 Mb file");echo XXRETCODE:$?' May 17 02:13:26 client-5 kernel: Lustre: DEBUG MARKER: creating a 10 Mb file May 17 02:13:26 client-5 xinetd[3391]: EXIT: shell status=0 pid=18742 duration=0(sec) May 17 02:13:26 client-5 xinetd[3391]: START: shell pid=18836 from=192.168.4.5 May 17 02:13:26 client-5 rshd[18837]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark starting reads");echo XXRETCODE:$?' May 17 02:13:26 client-5 kernel: Lustre: DEBUG MARKER: starting reads May 17 02:13:26 client-5 xinetd[3391]: EXIT: shell status=0 pid=18836 duration=0(sec) May 17 02:13:26 client-5 xinetd[3391]: START: shell pid=18929 from=192.168.4.5 May 17 02:13:26 client-5 rshd[18931]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark truncating the file");echo XXRETCODE:$?' May 17 02:13:26 client-5 kernel: Lustre: DEBUG MARKER: truncating the file May 17 02:13:26 client-5 xinetd[3391]: EXIT: shell status=0 pid=18929 duration=0(sec) May 17 02:13:26 client-5 xinetd[3391]: START: shell pid=19023 from=192.168.4.5 May 17 02:13:27 client-5 rshd[19025]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark killing dd");echo XXRETCODE:$?' May 17 02:13:27 client-5 kernel: Lustre: DEBUG MARKER: killing dd May 17 02:13:27 client-5 xinetd[3391]: EXIT: shell status=0 pid=19023 duration=1(sec) May 17 02:13:27 client-5 xinetd[3391]: START: shell pid=19117 from=192.168.4.5 May 17 02:13:27 client-5 rshd[19118]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark removing the temporary file");echo XXRETCODE:$?' May 17 02:13:27 client-5 kernel: Lustre: DEBUG MARKER: removing the temporary file May 17 02:13:27 client-5 xinetd[3391]: EXIT: shell status=0 pid=19117 duration=0(sec) May 17 02:13:27 client-5 xinetd[3391]: START: shell pid=19208 from=192.168.4.5 May 17 02:13:27 client-5 rshd[19210]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 02:13:27 client-5 xinetd[3391]: EXIT: shell status=0 pid=19208 duration=0(sec) May 17 02:13:27 client-5 xinetd[3391]: START: shell pid=19381 from=192.168.4.5 May 17 02:13:27 client-5 rshd[19382]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity test 900: umount should not race with any mgc requeue thread =============================== 02:13:27 \(1305623607\)");echo XXRETCODE:$?' May 17 02:13:27 client-5 kernel: Lustre: DEBUG MARKER: == sanity test 900: umount should not race with any mgc requeue thread =============================== 02:13:27 (1305623607) May 17 02:13:27 client-5 xinetd[3391]: EXIT: shell status=0 pid=19381 duration=0(sec) May 17 02:13:27 client-5 xinetd[3391]: START: shell pid=19426 from=192.168.4.5 May 17 02:13:27 client-5 rshd[19427]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:; lsof -t /mnt/lustre || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre is still busy, wait one second && sleep 1; umount /mnt/lustre; fi fi");echo XXRETCODE:$?' May 17 02:13:28 client-5 kernel: LustreError: 19468:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 02:13:28 client-5 kernel: LustreError: 19468:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Skipped 13 previous similar messages May 17 02:13:28 client-5 kernel: LustreError: 19468:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 02:13:28 client-5 kernel: LustreError: 19468:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) Skipped 13 previous similar messages May 17 02:13:28 client-5 kernel: Lustre: client ffff8102f8e5f800 umount complete May 17 02:13:28 client-5 xinetd[3391]: EXIT: shell status=0 pid=19426 duration=1(sec) May 17 02:13:28 client-5 xinetd[3391]: START: shell pid=19477 from=192.168.4.5 May 17 02:13:28 client-5 rshd[19478]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:; lsof -t /mnt/lustre2 || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre2 | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre2 is still busy, wait one second && sleep 1; umount /mnt/lustre2; fi fi");echo XXRETCODE:$?' May 17 02:13:28 client-5 xinetd[3391]: EXIT: shell status=0 pid=19477 duration=0(sec) May 17 02:13:58 client-5 kernel: Lustre: Acceptor stopping May 17 02:13:59 client-5 kernel: Lustre: Removed LNI 10.10.4.5@tcp May 17 02:14:02 client-5 kernel: Lustre: Removed LNI 192.168.4.5@o2ib May 17 02:14:03 client-5 kernel: Lustre: 20088:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 02:14:03 client-5 kernel: Lustre: 20089:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 02:14:03 client-5 kernel: Lustre: OBD class driver, http://www.lustre.org/ May 17 02:14:03 client-5 kernel: Lustre: Lustre Version: 2.0.60 May 17 02:14:03 client-5 kernel: Lustre: Build Version: jenkins-g985ac5b-PRISTINE-2.6.18-194.17.1.el5 May 17 02:14:03 client-5 kernel: Lustre: Lustre LU module (ffffffff888eb1c0). May 17 02:14:03 client-5 kernel: Lustre: Register global MR array, MR size: 0xffffffffffffffff, array size: 1 May 17 02:14:03 client-5 kernel: Lustre: Added LNI 192.168.4.5@o2ib [8/64/0/180] May 17 02:14:03 client-5 kernel: Lustre: Added LNI 10.10.4.5@tcp [8/256/0/180] May 17 02:14:03 client-5 kernel: Lustre: Accept all, port 7988 May 17 02:14:03 client-5 kernel: Lustre: Lustre OSC module (ffffffff88b3c6c0). May 17 02:14:03 client-5 kernel: Lustre: Lustre LOV module (ffffffff88bab540). May 17 02:14:03 client-5 kernel: Lustre: Lustre client module (ffffffff88c643a0). May 17 02:14:04 client-5 xinetd[3391]: START: shell pid=20548 from=192.168.4.5 May 17 02:14:04 client-5 rshd[20549]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); mpts=\$(mount | grep -w -c /mnt/lustre); if [ \$running -ne \$mpts ]; then echo \$(hostname) env are INSANE!; exit 1; fi");echo XXRETCODE:$?' May 17 02:14:04 client-5 xinetd[3391]: EXIT: shell status=0 pid=20548 duration=0(sec) May 17 02:14:04 client-5 xinetd[3391]: START: shell pid=20595 from=192.168.4.5 May 17 02:14:04 client-5 rshd[20596]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); mpts=\$(mount | grep -w -c /mnt/lustre2); if [ \$running -ne \$mpts ]; then echo \$(hostname) env are INSANE!; exit 1; fi");echo XXRETCODE:$?' May 17 02:14:04 client-5 xinetd[3391]: EXIT: shell status=0 pid=20595 duration=0(sec) May 17 02:14:04 client-5 kernel: Lustre: 20638:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 02:14:09 client-5 kernel: Lustre: 20968:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 02:14:09 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 02:14:09 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 02:14:09 client-5 kernel: Lustre: Increasing default stripe size to min 1048576 May 17 02:14:09 client-5 kernel: Lustre: 20968:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff810321d30800->192.168.4.128@o2ib netid 50000: select flavor null May 17 02:14:09 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 02:14:14 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 02:14:19 client-5 kernel: Lustre: 20968:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-OST0000-osc-ffff810321d30800->192.168.4.129@o2ib netid 50000: select flavor null May 17 02:14:19 client-5 kernel: Lustre: Client lustre-client has started May 17 02:14:19 client-5 kernel: Lustre: 20985:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 02:14:19 client-5 kernel: Lustre: 20985:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message May 17 02:14:19 client-5 xinetd[3391]: START: shell pid=20995 from=192.168.4.5 May 17 02:14:19 client-5 rshd[20996]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c " running=\$(mount | grep -c /mnt/lustre' '); rc=0; if [ \$running -eq 0 ] ; then mkdir -p /mnt/lustre; mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre; rc=\$?; fi; exit \$rc");echo XXRETCODE:$?' May 17 02:14:19 client-5 xinetd[3391]: EXIT: shell status=0 pid=20995 duration=0(sec) May 17 02:14:19 client-5 xinetd[3391]: START: shell pid=21040 from=192.168.4.5 May 17 02:14:19 client-5 rshd[21041]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mount | grep -w /mnt/lustre");echo XXRETCODE:$?' May 17 02:14:19 client-5 xinetd[3391]: EXIT: shell status=0 pid=21040 duration=0(sec) May 17 02:14:19 client-5 xinetd[3391]: START: shell pid=21084 from=192.168.4.5 May 17 02:14:20 client-5 rshd[21085]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param debug=\"0x33f0406\"; lctl set_param subsystem_debug=\"0xffb7e3ff\"; lctl set_param debug_mb=32;");echo XXRETCODE:$?' May 17 02:14:20 client-5 xinetd[3391]: EXIT: shell status=0 pid=21084 duration=1(sec) May 17 02:14:20 client-5 xinetd[3391]: START: shell pid=21253 from=192.168.4.5 May 17 02:14:20 client-5 rshd[21255]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark Using TIMEOUT=20");echo XXRETCODE:$?' May 17 02:14:20 client-5 kernel: Lustre: DEBUG MARKER: Using TIMEOUT=20 May 17 02:14:20 client-5 xinetd[3391]: EXIT: shell status=0 pid=21253 duration=0(sec) May 17 02:14:20 client-5 kernel: LustreError: 21308:0:(quota_ctl.c:328:client_quota_ctl()) ptlrpc_queue_wait failed, rc: -114 May 17 02:14:25 client-5 xinetd[3391]: START: shell pid=21366 from=192.168.4.5 May 17 02:14:25 client-5 rshd[21367]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 02:14:26 client-5 xinetd[3391]: EXIT: shell status=0 pid=21366 duration=1(sec) May 17 02:14:26 client-5 xinetd[3391]: START: shell pid=21531 from=192.168.4.5 May 17 02:14:26 client-5 rshd[21532]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity sanity.sh test complete, duration 7201 sec ================================================= 02:14:26 \(1305623666\)");echo XXRETCODE:$?' May 17 02:14:26 client-5 kernel: Lustre: DEBUG MARKER: == sanity sanity.sh test complete, duration 7201 sec ================================================= 02:14:26 (1305623666) May 17 02:14:26 client-5 xinetd[3391]: EXIT: shell status=0 pid=21531 duration=0(sec) May 17 02:14:27 client-5 xinetd[3391]: START: shell pid=21678 from=192.168.4.5 May 17 02:14:27 client-5 rshd[21681]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark -----============= acceptance-small: metadata-updates ============----- Tue May 17 02:14:27 PDT 2011");echo XXRETCODE:$?' May 17 02:14:27 client-5 kernel: Lustre: DEBUG MARKER: -----============= acceptance-small: metadata-updates ============----- Tue May 17 02:14:27 PDT 2011 May 17 02:14:27 client-5 xinetd[3391]: EXIT: shell status=0 pid=21678 duration=0(sec) May 17 02:14:27 client-5 xinetd[3391]: START: shell pid=21763 from=192.168.4.5 May 17 02:14:27 client-5 rshd[21764]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "echo \$(hostname)");echo XXRETCODE:$?' May 17 02:14:27 client-5 xinetd[3391]: EXIT: shell status=0 pid=21763 duration=0(sec) May 17 02:14:27 client-5 xinetd[3391]: START: shell pid=21867 from=192.168.4.5 May 17 02:14:27 client-5 rshd[21868]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark ===== \/usr\/lib64\/lustre\/tests\/metadata-updates.sh ====== ");echo XXRETCODE:$?' May 17 02:14:27 client-5 kernel: Lustre: DEBUG MARKER: ===== /usr/lib64/lustre/tests/metadata-updates.sh ====== May 17 02:14:27 client-5 xinetd[3391]: EXIT: shell status=0 pid=21867 duration=0(sec) May 17 02:14:27 client-5 xinetd[3391]: START: shell pid=21918 from=192.168.4.5 May 17 02:14:27 client-5 rshd[21919]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/tests:/usr/lib64/openmpi/1.4-gcc/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/usr/lib64/lustre/utils:/usr/lib64/lustre/utils/gss:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey::/usr/lib64/lustre/tests/mpi NAME=sarah sh rpc.sh check_config_client /mnt/lustre ");echo XXRETCODE:$?' May 17 02:14:27 client-5 xinetd[3391]: EXIT: shell status=0 pid=21918 duration=0(sec) May 17 02:14:28 client-5 xinetd[3391]: START: shell pid=22118 from=192.168.4.5 May 17 02:14:28 client-5 rshd[22119]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); mpts=\$(mount | grep -w -c /mnt/lustre); if [ \$running -ne \$mpts ]; then echo \$(hostname) env are INSANE!; exit 1; fi");echo XXRETCODE:$?' May 17 02:14:28 client-5 xinetd[3391]: EXIT: shell status=0 pid=22118 duration=0(sec) May 17 02:14:28 client-5 xinetd[3391]: START: shell pid=22165 from=192.168.4.5 May 17 02:14:28 client-5 rshd[22166]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); mpts=\$(mount | grep -w -c /mnt/lustre2); if [ \$running -ne \$mpts ]; then echo \$(hostname) env are INSANE!; exit 1; fi");echo XXRETCODE:$?' May 17 02:14:28 client-5 xinetd[3391]: EXIT: shell status=0 pid=22165 duration=0(sec) May 17 02:14:30 client-5 xinetd[3391]: START: shell pid=22526 from=192.168.4.5 May 17 02:14:30 client-5 rshd[22528]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark Using TIMEOUT=20");echo XXRETCODE:$?' May 17 02:14:30 client-5 kernel: Lustre: DEBUG MARKER: Using TIMEOUT=20 May 17 02:14:30 client-5 xinetd[3391]: EXIT: shell status=0 pid=22526 duration=0(sec) May 17 02:14:31 client-5 kernel: LustreError: 22581:0:(quota_ctl.c:328:client_quota_ctl()) ptlrpc_queue_wait failed, rc: -114 May 17 02:14:31 client-5 xinetd[3391]: START: shell pid=22638 from=192.168.4.5 May 17 02:14:31 client-5 rshd[22639]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param debug=\"0x33f0406\"; lctl set_param subsystem_debug=\"0xffb7e3ff\"; lctl set_param debug_mb=32; sync");echo XXRETCODE:$?' May 17 02:14:31 client-5 kernel: Lustre: 22673:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 02:14:31 client-5 kernel: Lustre: 22673:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 3 previous similar messages May 17 02:14:31 client-5 xinetd[3391]: EXIT: shell status=0 pid=22638 duration=0(sec) May 17 02:14:31 client-5 xinetd[3391]: START: shell pid=22690 from=192.168.4.5 May 17 02:14:31 client-5 rshd[22691]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "cat /proc/mounts");echo XXRETCODE:$?' May 17 02:14:31 client-5 xinetd[3391]: EXIT: shell status=0 pid=22690 duration=0(sec) May 17 02:14:32 client-5 xinetd[3391]: START: shell pid=22902 from=192.168.4.5 May 17 02:14:32 client-5 rshd[22903]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "set +x; DIR=/mnt/lustre/d0.metadata-updates/\$(hostname); TESTFILE=\$DIR/testfile; rm -f \$TESTFILE; rm -f /mnt/lustre/d0.metadata-updates/mdsum; rmdir \$DIR 2>/dev/null; mkdir -p \$DIR");echo XXRETCODE:$?' May 17 02:14:32 client-5 xinetd[3391]: EXIT: shell status=0 pid=22902 duration=0(sec) May 17 02:14:32 client-5 xinetd[3391]: START: shell pid=22949 from=192.168.4.5 May 17 02:14:32 client-5 rshd[22950]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "set +x; TESTFILE=/mnt/lustre/d0.metadata-updates/\$(hostname)/testfile; mcreate \$TESTFILE; ");echo XXRETCODE:$?' May 17 02:14:32 client-5 xinetd[3391]: EXIT: shell status=0 pid=22949 duration=0(sec) May 17 02:14:32 client-5 xinetd[3391]: START: shell pid=22993 from=192.168.4.5 May 17 02:14:32 client-5 rshd[22994]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "set +x; TESTFILE=/mnt/lustre/d0.metadata-updates/\$(hostname)/testfile; dd if=/dev/zero of=\$TESTFILE bs=1024 count=1 2>/dev/null || exit 54; echo \$(hostname) | dd of=\$TESTFILE conv=notrunc 2>/dev/null || exit 55; md5sum \$TESTFILE >> /mnt/lustre/d0.metadata-updates/mdsum; ");echo XXRETCODE:$?' May 17 02:14:32 client-5 xinetd[3391]: EXIT: shell status=0 pid=22993 duration=0(sec) May 17 02:14:32 client-5 xinetd[3391]: START: shell pid=23043 from=192.168.4.5 May 17 02:14:32 client-5 rshd[23044]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "md5sum --check /mnt/lustre/d0.metadata-updates/mdsum");echo XXRETCODE:$?' May 17 02:14:33 client-5 xinetd[3391]: EXIT: shell status=0 pid=23043 duration=1(sec) May 17 02:14:33 client-5 xinetd[3391]: START: shell pid=23084 from=192.168.4.5 May 17 02:14:33 client-5 rshd[23085]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "set +x; for HOST in client-18.lab.whamcloud.com client-5-ib ; do TESTFILE=/mnt/lustre/d0.metadata-updates/\$HOST/testfile; tmp=\$(stat -c \"%u %g %s 0%a\" \$TESTFILE); echo \"\$TESTFILE [ uid gid size mode ] expected : 0 0 1024 0644 ; got : \$tmp \"; if [ x\"\$tmp\" != x\"0 0 1024 0644\" ] ; then echo \"Wrong file attributes\"; exit 56; fi; done ");echo XXRETCODE:$?' May 17 02:14:33 client-5 xinetd[3391]: EXIT: shell status=0 pid=23084 duration=0(sec) May 17 02:14:33 client-5 xinetd[3391]: START: shell pid=23130 from=192.168.4.5 May 17 02:14:33 client-5 rshd[23131]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "set +x; TESTFILE=/mnt/lustre/d0.metadata-updates/\$(hostname)/testfile; chmod 0222 \$TESTFILE");echo XXRETCODE:$?' May 17 02:14:33 client-5 xinetd[3391]: EXIT: shell status=0 pid=23130 duration=0(sec) May 17 02:14:33 client-5 xinetd[3391]: START: shell pid=23173 from=192.168.4.5 May 17 02:14:33 client-5 rshd[23174]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "set +x; for HOST in client-18.lab.whamcloud.com client-5-ib ; do TESTFILE=/mnt/lustre/d0.metadata-updates/\$HOST/testfile; tmp=\$(stat -c \"%u %g %s 0%a\" \$TESTFILE); echo \"\$TESTFILE [ uid gid size mode ] expected : 0 0 1024 0222 ; got : \$tmp \"; if [ x\"\$tmp\" != x\"0 0 1024 0222\" ] ; then echo \"Wrong file attributes\"; exit 56; fi; done ");echo XXRETCODE:$?' May 17 02:14:33 client-5 xinetd[3391]: EXIT: shell status=0 pid=23173 duration=0(sec) May 17 02:14:33 client-5 xinetd[3391]: START: shell pid=23219 from=192.168.4.5 May 17 02:14:33 client-5 rshd[23220]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c " set +x; TESTFILE=/mnt/lustre/d0.metadata-updates/\$(hostname)/testfile; touch -c --date=\"2001-01-01 GMT\" -a \$TESTFILE; touch -c --date=\"2005-05-05 GMT\" -m \$TESTFILE ");echo XXRETCODE:$?' May 17 02:14:33 client-5 xinetd[3391]: EXIT: shell status=0 pid=23219 duration=0(sec) May 17 02:14:33 client-5 xinetd[3391]: START: shell pid=23265 from=192.168.4.5 May 17 02:14:33 client-5 rshd[23266]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "set +x; for HOST in client-18.lab.whamcloud.com client-5-ib ; do TESTFILE=/mnt/lustre/d0.metadata-updates/\$HOST/testfile; tmp=\$(stat -c \"%X %Y\" \$TESTFILE); if [ x\"\$tmp\" != x\"978307200 1115251200\" ] ; then echo \"\$TESTFILE [ atime mtime ] expected : 978307200 1115251200 ; got : \$tmp \"; RC=57; fi; done; exit \$RC");echo XXRETCODE:$?' May 17 02:14:33 client-5 xinetd[3391]: EXIT: shell status=0 pid=23265 duration=0(sec) May 17 02:14:33 client-5 xinetd[3391]: START: shell pid=23311 from=192.168.4.5 May 17 02:14:33 client-5 rshd[23312]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "set +x; TESTFILE=/mnt/lustre/d0.metadata-updates/\$(hostname)/testfile; /usr/lib64/lustre/tests/truncate \$TESTFILE 0");echo XXRETCODE:$?' May 17 02:14:33 client-5 xinetd[3391]: EXIT: shell status=0 pid=23311 duration=0(sec) May 17 02:14:33 client-5 xinetd[3391]: START: shell pid=23354 from=192.168.4.5 May 17 02:14:33 client-5 rshd[23355]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "set +x; for HOST in client-18.lab.whamcloud.com client-5-ib ; do TESTFILE=/mnt/lustre/d0.metadata-updates/\$HOST/testfile; tmp=\$(stat -c \"%u %g %s 0%a\" \$TESTFILE); echo \"\$TESTFILE [ uid gid size mode ] expected : 0 0 0 0222 ; got : \$tmp \"; if [ x\"\$tmp\" != x\"0 0 0 0222\" ] ; then echo \"Wrong file attributes\"; exit 56; fi; done ");echo XXRETCODE:$?' May 17 02:14:33 client-5 xinetd[3391]: EXIT: shell status=0 pid=23354 duration=0(sec) May 17 02:14:33 client-5 xinetd[3391]: START: shell pid=23400 from=192.168.4.5 May 17 02:14:33 client-5 rshd[23401]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "set +x; TESTFILE=/mnt/lustre/d0.metadata-updates/\$(hostname)/testfile; rm -f \$TESTFILE; DIR=/mnt/lustre/d0.metadata-updates/\$(hostname); for i in \$(seq 1000) ; do touch \$DIR/f\$i; done ");echo XXRETCODE:$?' May 17 02:14:36 client-5 xinetd[3391]: EXIT: shell status=0 pid=23400 duration=3(sec) May 17 02:14:36 client-5 xinetd[3391]: START: shell pid=24449 from=192.168.4.5 May 17 02:14:36 client-5 rshd[24450]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "set +x; for HOST in client-18.lab.whamcloud.com client-5-ib ; do DIR=/mnt/lustre/d0.metadata-updates/\$HOST; for i in \$(seq 1000 -1 1) ; do if ! [ -f \$DIR/f\$i ] ; then echo \"ERROR: file \$DIR/f\$i should exist\"; RC=1; fi; done; for i in \$(seq 0) ; do if [ -f \$DIR/f\$i ] ; then echo \"ERROR: deleted file \$DIR/f\$i exists\"; RC=1; fi; done; done; exit \$RC ");echo XXRETCODE:$?' May 17 02:14:38 client-5 xinetd[3391]: EXIT: shell status=0 pid=24449 duration=2(sec) May 17 02:14:39 client-5 xinetd[3391]: START: shell pid=24499 from=192.168.4.5 May 17 02:14:39 client-5 rshd[24500]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "set +x; DIR=/mnt/lustre/d0.metadata-updates/\$(hostname); for i in \$(seq 500) ; do if ! rm -f \$DIR/f\$i ; then exit 1; fi; done ");echo XXRETCODE:$?' May 17 02:14:39 client-5 xinetd[3391]: EXIT: shell status=0 pid=24499 duration=0(sec) May 17 02:14:40 client-5 xinetd[3391]: START: shell pid=25044 from=192.168.4.5 May 17 02:14:40 client-5 rshd[25046]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "set +x; for HOST in client-18.lab.whamcloud.com client-5-ib ; do DIR=/mnt/lustre/d0.metadata-updates/\$HOST; for i in \$(seq 1000 -1 501) ; do if ! [ -f \$DIR/f\$i ] ; then echo \"ERROR: file \$DIR/f\$i should exist\"; RC=1; fi; done; for i in \$(seq 500) ; do if [ -f \$DIR/f\$i ] ; then echo \"ERROR: deleted file \$DIR/f\$i exists\"; RC=1; fi; done; done; exit \$RC ");echo XXRETCODE:$?' May 17 02:14:40 client-5 xinetd[3391]: EXIT: shell status=0 pid=25044 duration=0(sec) May 17 02:18:06 client-5 init: Trying to re-exec init May 17 12:31:41 client-5 xinetd[3391]: START: shell pid=26242 from=192.168.4.5 May 17 12:31:41 client-5 rshd[26243]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == metadata-updates metadata-updates.sh test complete, duration 37034 sec ============================ 12:31:41 \(1305660701\)");echo XXRETCODE:$?' May 17 12:31:41 client-5 kernel: Lustre: DEBUG MARKER: == metadata-updates metadata-updates.sh test complete, duration 37034 sec ============================ 12:31:41 (1305660701) May 17 12:31:41 client-5 xinetd[3391]: EXIT: shell status=0 pid=26242 duration=0(sec) May 17 12:31:42 client-5 xinetd[3391]: START: shell pid=26354 from=192.168.4.5 May 17 12:31:42 client-5 rshd[26357]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark -----============= acceptance-small: sanity-benchmark ============----- Tue May 17 12:31:42 PDT 2011");echo XXRETCODE:$?' May 17 12:31:42 client-5 kernel: Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-benchmark ============----- Tue May 17 12:31:42 PDT 2011 May 17 12:31:42 client-5 xinetd[3391]: EXIT: shell status=0 pid=26354 duration=0(sec) May 17 12:31:42 client-5 xinetd[3391]: START: shell pid=26492 from=192.168.4.5 May 17 12:31:42 client-5 rshd[26495]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark excepting tests: ");echo XXRETCODE:$?' May 17 12:31:42 client-5 kernel: Lustre: DEBUG MARKER: excepting tests: May 17 12:31:42 client-5 xinetd[3391]: EXIT: shell status=0 pid=26492 duration=0(sec) May 17 12:31:42 client-5 xinetd[3391]: START: shell pid=26545 from=192.168.4.5 May 17 12:31:42 client-5 rshd[26546]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/tests:/usr/lib64/openmpi/1.4-gcc/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/usr/lib64/lustre/utils:/usr/lib64/lustre/utils/gss:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey::/usr/lib64/lustre/tests/mpi NAME=sarah sh rpc.sh check_config_client /mnt/lustre ");echo XXRETCODE:$?' May 17 12:31:43 client-5 xinetd[3391]: EXIT: shell status=0 pid=26545 duration=1(sec) May 17 12:31:43 client-5 xinetd[3391]: START: shell pid=26745 from=192.168.4.5 May 17 12:31:44 client-5 rshd[26746]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); mpts=\$(mount | grep -w -c /mnt/lustre); if [ \$running -ne \$mpts ]; then echo \$(hostname) env are INSANE!; exit 1; fi");echo XXRETCODE:$?' May 17 12:31:44 client-5 xinetd[3391]: EXIT: shell status=0 pid=26745 duration=1(sec) May 17 12:31:44 client-5 xinetd[3391]: START: shell pid=26792 from=192.168.4.5 May 17 12:31:44 client-5 rshd[26793]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); mpts=\$(mount | grep -w -c /mnt/lustre2); if [ \$running -ne \$mpts ]; then echo \$(hostname) env are INSANE!; exit 1; fi");echo XXRETCODE:$?' May 17 12:31:44 client-5 xinetd[3391]: EXIT: shell status=0 pid=26792 duration=0(sec) May 17 12:31:46 client-5 xinetd[3391]: START: shell pid=27154 from=192.168.4.5 May 17 12:31:46 client-5 rshd[27155]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark Using TIMEOUT=20");echo XXRETCODE:$?' May 17 12:31:46 client-5 kernel: Lustre: DEBUG MARKER: Using TIMEOUT=20 May 17 12:31:46 client-5 xinetd[3391]: EXIT: shell status=0 pid=27154 duration=0(sec) May 17 12:31:46 client-5 kernel: LustreError: 27208:0:(quota_ctl.c:328:client_quota_ctl()) ptlrpc_queue_wait failed, rc: -114 May 17 12:31:46 client-5 xinetd[3391]: START: shell pid=27265 from=192.168.4.5 May 17 12:31:46 client-5 rshd[27266]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param debug=\"0x33f0406\"; lctl set_param subsystem_debug=\"0xffb7e3ff\"; lctl set_param debug_mb=32; sync");echo XXRETCODE:$?' May 17 12:31:46 client-5 kernel: Lustre: 27300:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 12:31:46 client-5 kernel: Lustre: 27300:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message May 17 12:31:46 client-5 xinetd[3391]: EXIT: shell status=0 pid=27265 duration=0(sec) May 17 12:31:46 client-5 xinetd[3391]: START: shell pid=27317 from=192.168.4.5 May 17 12:31:46 client-5 rshd[27318]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "cat /proc/mounts");echo XXRETCODE:$?' May 17 12:31:46 client-5 xinetd[3391]: EXIT: shell status=0 pid=27317 duration=0(sec) May 17 12:31:47 client-5 xinetd[3391]: START: shell pid=27587 from=192.168.4.5 May 17 12:31:47 client-5 rshd[27589]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity-benchmark test dbench: dbench ============================================================== 12:31:47 \(1305660707\)");echo XXRETCODE:$?' May 17 12:31:47 client-5 kernel: Lustre: DEBUG MARKER: == sanity-benchmark test dbench: dbench ============================================================== 12:31:47 (1305660707) May 17 12:31:47 client-5 xinetd[3391]: EXIT: shell status=0 pid=27587 duration=0(sec) May 17 12:31:47 client-5 kernel: Lustre: 27629:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 12:31:47 client-5 kernel: Lustre: 27629:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message May 17 12:43:48 client-5 kernel: Lustre: 27737:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 12:50:14 client-5 xinetd[3391]: START: login pid=27790 from=10.10.0.1 May 17 12:52:11 client-5 xinetd[3391]: EXIT: login status=0 pid=27790 duration=117(sec) May 17 12:55:51 client-5 kernel: Lustre: 28083:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 12:55:51 client-5 kernel: Lustre: 28083:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message May 17 12:55:52 client-5 xinetd[3391]: START: shell pid=28364 from=192.168.4.5 May 17 12:55:52 client-5 rshd[28365]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 12:55:52 client-5 xinetd[3391]: EXIT: shell status=0 pid=28364 duration=0(sec) May 17 12:55:52 client-5 xinetd[3391]: START: shell pid=28534 from=192.168.4.5 May 17 12:55:52 client-5 rshd[28537]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity-benchmark test bonnie: bonnie++ ============================================================ 12:55:52 \(1305662152\)");echo XXRETCODE:$?' May 17 12:55:52 client-5 kernel: Lustre: DEBUG MARKER: == sanity-benchmark test bonnie: bonnie++ ============================================================ 12:55:52 (1305662152) May 17 12:55:52 client-5 xinetd[3391]: EXIT: shell status=0 pid=28534 duration=0(sec) May 17 12:55:52 client-5 xinetd[3391]: START: shell pid=28635 from=192.168.4.5 May 17 12:55:52 client-5 rshd[28638]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark min OST has 8907404kB available, using 24580944kB file size");echo XXRETCODE:$?' May 17 12:55:52 client-5 kernel: Lustre: DEBUG MARKER: min OST has 8907404kB available, using 24580944kB file size May 17 12:55:52 client-5 xinetd[3391]: EXIT: shell status=0 pid=28635 duration=0(sec) May 17 12:55:52 client-5 kernel: Lustre: 28672:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 13:08:07 client-5 kernel: Lustre: 28730:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 13:08:07 client-5 xinetd[3391]: START: shell pid=28787 from=192.168.4.5 May 17 13:08:07 client-5 rshd[28788]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 13:08:08 client-5 xinetd[3391]: EXIT: shell status=0 pid=28787 duration=1(sec) May 17 13:08:08 client-5 xinetd[3391]: START: shell pid=28959 from=192.168.4.5 May 17 13:08:08 client-5 rshd[28960]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity-benchmark test iozone: iozone ============================================================== 13:08:08 \(1305662888\)");echo XXRETCODE:$?' May 17 13:08:09 client-5 kernel: Lustre: DEBUG MARKER: == sanity-benchmark test iozone: iozone ============================================================== 13:08:08 (1305662888) May 17 13:08:09 client-5 xinetd[3391]: EXIT: shell status=0 pid=28959 duration=1(sec) May 17 13:08:09 client-5 xinetd[3391]: START: shell pid=29060 from=192.168.4.5 May 17 13:08:09 client-5 rshd[29061]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark min OST has 8917752kB available, using 24580944kB file size");echo XXRETCODE:$?' May 17 13:08:09 client-5 kernel: Lustre: DEBUG MARKER: min OST has 8917752kB available, using 24580944kB file size May 17 13:08:09 client-5 kernel: Lustre: 29095:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 13:08:09 client-5 xinetd[3391]: EXIT: shell status=0 pid=29060 duration=0(sec) May 17 13:24:52 client-5 kernel: Lustre: 29166:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 13:24:52 client-5 kernel: Lustre: 29170:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 13:59:14 client-5 kernel: Lustre: 29284:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 13:59:15 client-5 xinetd[3391]: START: shell pid=29349 from=192.168.4.5 May 17 13:59:15 client-5 rshd[29350]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 13:59:16 client-5 xinetd[3391]: EXIT: shell status=0 pid=29349 duration=1(sec) May 17 13:59:16 client-5 xinetd[3391]: START: shell pid=29521 from=192.168.4.5 May 17 13:59:16 client-5 rshd[29522]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity-benchmark test fsx: fsx ==================================================================== 13:59:16 \(1305665956\)");echo XXRETCODE:$?' May 17 13:59:16 client-5 kernel: Lustre: DEBUG MARKER: == sanity-benchmark test fsx: fsx ==================================================================== 13:59:16 (1305665956) May 17 13:59:16 client-5 xinetd[3391]: EXIT: shell status=0 pid=29521 duration=0(sec) May 17 13:59:16 client-5 kernel: Lustre: 29560:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 14:02:54 client-5 kernel: Lustre: 29583:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 14:02:54 client-5 xinetd[3391]: START: shell pid=29640 from=192.168.4.5 May 17 14:02:54 client-5 rshd[29641]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:02:54 client-5 xinetd[3391]: EXIT: shell status=0 pid=29640 duration=0(sec) May 17 14:02:55 client-5 xinetd[3391]: START: shell pid=29812 from=192.168.4.5 May 17 14:02:55 client-5 rshd[29813]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity-benchmark test pios_ssf: pios shared single file =========================================== 14:02:55 \(1305666175\)");echo XXRETCODE:$?' May 17 14:02:55 client-5 kernel: Lustre: DEBUG MARKER: == sanity-benchmark test pios_ssf: pios shared single file =========================================== 14:02:55 (1305666175) May 17 14:02:55 client-5 xinetd[3391]: EXIT: shell status=0 pid=29812 duration=0(sec) May 17 14:02:55 client-5 xinetd[3391]: START: shell pid=29905 from=192.168.4.5 May 17 14:02:55 client-5 rshd[29906]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark SKIP: sanity-benchmark test_pios_ssf \/usr\/lib64\/lustre\/tests\/sanity-benchmark.sh : pios not found PIOSBIN=");echo XXRETCODE:$?' May 17 14:02:55 client-5 kernel: Lustre: DEBUG MARKER: SKIP: sanity-benchmark test_pios_ssf /usr/lib64/lustre/tests/sanity-benchmark.sh : pios not found PIOSBIN= May 17 14:02:55 client-5 xinetd[3391]: EXIT: shell status=0 pid=29905 duration=0(sec) May 17 14:02:55 client-5 xinetd[3391]: START: shell pid=29996 from=192.168.4.5 May 17 14:02:55 client-5 rshd[29997]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:02:55 client-5 xinetd[3391]: EXIT: shell status=0 pid=29996 duration=0(sec) May 17 14:02:55 client-5 xinetd[3391]: START: shell pid=30166 from=192.168.4.5 May 17 14:02:55 client-5 rshd[30169]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity-benchmark test pios_fpp: pios file per process ============================================= 14:02:55 \(1305666175\)");echo XXRETCODE:$?' May 17 14:02:55 client-5 kernel: Lustre: DEBUG MARKER: == sanity-benchmark test pios_fpp: pios file per process ============================================= 14:02:55 (1305666175) May 17 14:02:55 client-5 xinetd[3391]: EXIT: shell status=0 pid=30166 duration=0(sec) May 17 14:02:55 client-5 xinetd[3391]: START: shell pid=30261 from=192.168.4.5 May 17 14:02:55 client-5 rshd[30262]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark SKIP: sanity-benchmark test_pios_fpp pios not found PIOSBIN=");echo XXRETCODE:$?' May 17 14:02:55 client-5 kernel: Lustre: DEBUG MARKER: SKIP: sanity-benchmark test_pios_fpp pios not found PIOSBIN= May 17 14:02:56 client-5 xinetd[3391]: EXIT: shell status=0 pid=30261 duration=1(sec) May 17 14:02:56 client-5 xinetd[3391]: START: shell pid=30351 from=192.168.4.5 May 17 14:02:56 client-5 rshd[30353]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:02:56 client-5 xinetd[3391]: EXIT: shell status=0 pid=30351 duration=0(sec) May 17 14:02:56 client-5 xinetd[3391]: START: shell pid=30517 from=192.168.4.5 May 17 14:02:56 client-5 rshd[30518]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity-benchmark sanity-benchmark.sh test complete, duration 5474 sec ============================= 14:02:56 \(1305666176\)");echo XXRETCODE:$?' May 17 14:02:56 client-5 kernel: Lustre: DEBUG MARKER: == sanity-benchmark sanity-benchmark.sh test complete, duration 5474 sec ============================= 14:02:56 (1305666176) May 17 14:02:56 client-5 xinetd[3391]: EXIT: shell status=0 pid=30517 duration=0(sec) May 17 14:02:56 client-5 xinetd[3391]: START: shell pid=30627 from=192.168.4.5 May 17 14:02:56 client-5 rshd[30628]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark -----============= acceptance-small: sanityn ============----- Tue May 17 14:02:56 PDT 2011");echo XXRETCODE:$?' May 17 14:02:56 client-5 kernel: Lustre: DEBUG MARKER: -----============= acceptance-small: sanityn ============----- Tue May 17 14:02:56 PDT 2011 May 17 14:02:56 client-5 xinetd[3391]: EXIT: shell status=0 pid=30627 duration=0(sec) May 17 14:02:56 client-5 xinetd[3391]: START: shell pid=30719 from=192.168.4.5 May 17 14:02:56 client-5 rshd[30720]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c " running=\$(mount | grep -c /mnt/lustre2' '); rc=0; if [ \$running -eq 0 ] ; then mkdir -p /mnt/lustre2; mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre2; rc=\$?; fi; exit \$rc");echo XXRETCODE:$?' May 17 14:02:57 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 14:02:57 client-5 kernel: Lustre: Increasing default stripe size to min 1048576 May 17 14:02:57 client-5 kernel: Lustre: 30759:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff8102f6091800->192.168.4.128@o2ib netid 50000: select flavor null May 17 14:02:57 client-5 kernel: Lustre: 30759:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 5 previous similar messages May 17 14:02:57 client-5 kernel: Lustre: Client lustre-client has started May 17 14:02:57 client-5 xinetd[3391]: EXIT: shell status=0 pid=30719 duration=1(sec) May 17 14:02:57 client-5 xinetd[3391]: START: shell pid=30769 from=192.168.4.5 May 17 14:02:57 client-5 rshd[30770]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mount | grep -w /mnt/lustre2");echo XXRETCODE:$?' May 17 14:02:57 client-5 xinetd[3391]: EXIT: shell status=0 pid=30769 duration=0(sec) May 17 14:02:57 client-5 xinetd[3391]: START: shell pid=30813 from=192.168.4.5 May 17 14:02:57 client-5 rshd[30814]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param debug=\"0x33f0406\"; lctl set_param subsystem_debug=\"0xffb7e3ff\"; lctl set_param debug_mb=32;");echo XXRETCODE:$?' May 17 14:02:57 client-5 kernel: Lustre: 30848:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 14:02:57 client-5 xinetd[3391]: EXIT: shell status=0 pid=30813 duration=0(sec) May 17 14:02:57 client-5 xinetd[3391]: START: shell pid=30855 from=192.168.4.5 May 17 14:02:57 client-5 rshd[30857]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/tests:/usr/lib64/openmpi/1.4-gcc/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/usr/lib64/lustre/utils:/usr/lib64/lustre/utils/gss:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey::/usr/lib64/lustre/tests/mpi NAME=sarah sh rpc.sh check_config_client /mnt/lustre ");echo XXRETCODE:$?' May 17 14:02:57 client-5 xinetd[3391]: EXIT: shell status=0 pid=30855 duration=0(sec) May 17 14:02:58 client-5 xinetd[3391]: START: shell pid=31056 from=192.168.4.5 May 17 14:02:58 client-5 rshd[31057]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); mpts=\$(mount | grep -w -c /mnt/lustre); if [ \$running -ne \$mpts ]; then echo \$(hostname) env are INSANE!; exit 1; fi");echo XXRETCODE:$?' May 17 14:02:58 client-5 xinetd[3391]: EXIT: shell status=0 pid=31056 duration=0(sec) May 17 14:02:58 client-5 xinetd[3391]: START: shell pid=31103 from=192.168.4.5 May 17 14:02:58 client-5 rshd[31104]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); mpts=\$(mount | grep -w -c /mnt/lustre2); if [ \$running -ne \$mpts ]; then echo \$(hostname) env are INSANE!; exit 1; fi");echo XXRETCODE:$?' May 17 14:02:58 client-5 xinetd[3391]: EXIT: shell status=0 pid=31103 duration=0(sec) May 17 14:03:00 client-5 xinetd[3391]: START: shell pid=31464 from=192.168.4.5 May 17 14:03:00 client-5 rshd[31466]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark Using TIMEOUT=20");echo XXRETCODE:$?' May 17 14:03:00 client-5 kernel: Lustre: DEBUG MARKER: Using TIMEOUT=20 May 17 14:03:00 client-5 xinetd[3391]: EXIT: shell status=0 pid=31464 duration=0(sec) May 17 14:03:01 client-5 kernel: LustreError: 31519:0:(quota_ctl.c:328:client_quota_ctl()) ptlrpc_queue_wait failed, rc: -114 May 17 14:03:01 client-5 xinetd[3391]: START: shell pid=31576 from=192.168.4.5 May 17 14:03:01 client-5 rshd[31577]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param debug=\"0x33f0406\"; lctl set_param subsystem_debug=\"0xffb7e3ff\"; lctl set_param debug_mb=32; sync");echo XXRETCODE:$?' May 17 14:03:01 client-5 kernel: Lustre: 31611:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 14:03:01 client-5 kernel: Lustre: 31611:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message May 17 14:03:01 client-5 xinetd[3391]: EXIT: shell status=0 pid=31576 duration=0(sec) May 17 14:03:01 client-5 xinetd[3391]: START: shell pid=31628 from=192.168.4.5 May 17 14:03:01 client-5 rshd[31629]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "cat /proc/mounts");echo XXRETCODE:$?' May 17 14:03:01 client-5 xinetd[3391]: EXIT: shell status=0 pid=31628 duration=0(sec) May 17 14:03:02 client-5 xinetd[3391]: START: shell pid=31904 from=192.168.4.5 May 17 14:03:02 client-5 rshd[31905]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark excepting tests: 22 14b 19 22 28 29 35");echo XXRETCODE:$?' May 17 14:03:02 client-5 kernel: Lustre: DEBUG MARKER: excepting tests: 22 14b 19 22 28 29 35 May 17 14:03:02 client-5 xinetd[3391]: EXIT: shell status=0 pid=31904 duration=0(sec) May 17 14:03:02 client-5 xinetd[3391]: START: shell pid=32009 from=192.168.4.5 May 17 14:03:02 client-5 rshd[32010]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 1a: check create on 2 mtpt\'s ========================================================= 14:03:02 \(1305666182\)");echo XXRETCODE:$?' May 17 14:03:02 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 1a: check create on 2 mtpt's ========================================================= 14:03:02 (1305666182) May 17 14:03:02 client-5 xinetd[3391]: EXIT: shell status=0 pid=32009 duration=0(sec) May 17 14:03:02 client-5 xinetd[3391]: START: shell pid=32100 from=192.168.4.5 May 17 14:03:02 client-5 rshd[32102]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:03:03 client-5 xinetd[3391]: EXIT: shell status=0 pid=32100 duration=1(sec) May 17 14:03:03 client-5 xinetd[3391]: START: shell pid=32273 from=192.168.4.5 May 17 14:03:03 client-5 rshd[32274]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 1b: check attribute updates on 2 mtpt\'s ============================================== 14:03:03 \(1305666183\)");echo XXRETCODE:$?' May 17 14:03:03 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 1b: check attribute updates on 2 mtpt's ============================================== 14:03:03 (1305666183) May 17 14:03:03 client-5 xinetd[3391]: EXIT: shell status=0 pid=32273 duration=0(sec) May 17 14:03:03 client-5 xinetd[3391]: START: shell pid=32367 from=192.168.4.5 May 17 14:03:03 client-5 rshd[32368]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:03:03 client-5 xinetd[3391]: EXIT: shell status=0 pid=32367 duration=0(sec) May 17 14:03:03 client-5 xinetd[3391]: START: shell pid=32539 from=192.168.4.5 May 17 14:03:03 client-5 rshd[32540]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 1c: check after remount attribute updates on 2 mtpt\'s ================================ 14:03:03 \(1305666183\)");echo XXRETCODE:$?' May 17 14:03:03 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 1c: check after remount attribute updates on 2 mtpt's ================================ 14:03:03 (1305666183) May 17 14:03:03 client-5 xinetd[3391]: EXIT: shell status=0 pid=32539 duration=0(sec) May 17 14:03:03 client-5 xinetd[3391]: START: shell pid=32630 from=192.168.4.5 May 17 14:03:03 client-5 rshd[32632]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:03:04 client-5 xinetd[3391]: EXIT: shell status=0 pid=32630 duration=1(sec) May 17 14:03:04 client-5 xinetd[3391]: START: shell pid=337 from=192.168.4.5 May 17 14:03:04 client-5 rshd[338]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 1d: unlink on one mountpoint removes file on other =================================== 14:03:04 \(1305666184\)");echo XXRETCODE:$?' May 17 14:03:04 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 1d: unlink on one mountpoint removes file on other =================================== 14:03:04 (1305666184) May 17 14:03:04 client-5 xinetd[3391]: EXIT: shell status=0 pid=337 duration=0(sec) May 17 14:03:04 client-5 xinetd[3391]: START: shell pid=438 from=192.168.4.5 May 17 14:03:04 client-5 rshd[439]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:03:04 client-5 xinetd[3391]: EXIT: shell status=0 pid=438 duration=0(sec) May 17 14:03:04 client-5 xinetd[3391]: START: shell pid=622 from=192.168.4.5 May 17 14:03:04 client-5 rshd[624]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 2a: check cached attribute updates on 2 mtpt\'s ======================================= 14:03:04 \(1305666184\)");echo XXRETCODE:$?' May 17 14:03:04 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 2a: check cached attribute updates on 2 mtpt's ======================================= 14:03:04 (1305666184) May 17 14:03:04 client-5 xinetd[3391]: EXIT: shell status=0 pid=622 duration=0(sec) May 17 14:03:04 client-5 xinetd[3391]: START: shell pid=720 from=192.168.4.5 May 17 14:03:04 client-5 rshd[722]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:03:05 client-5 xinetd[3391]: EXIT: shell status=0 pid=720 duration=1(sec) May 17 14:03:05 client-5 xinetd[3391]: START: shell pid=893 from=192.168.4.5 May 17 14:03:05 client-5 rshd[894]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 2b: check cached attribute updates on 2 mtpt\'s ======================================= 14:03:05 \(1305666185\)");echo XXRETCODE:$?' May 17 14:03:05 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 2b: check cached attribute updates on 2 mtpt's ======================================= 14:03:05 (1305666185) May 17 14:03:05 client-5 xinetd[3391]: EXIT: shell status=0 pid=893 duration=0(sec) May 17 14:03:05 client-5 xinetd[3391]: START: shell pid=986 from=192.168.4.5 May 17 14:03:05 client-5 rshd[989]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:03:05 client-5 xinetd[3391]: EXIT: shell status=0 pid=986 duration=0(sec) May 17 14:03:05 client-5 xinetd[3391]: START: shell pid=1160 from=192.168.4.5 May 17 14:03:05 client-5 rshd[1161]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 2c: check cached attribute updates on 2 mtpt\'s root ================================== 14:03:05 \(1305666185\)");echo XXRETCODE:$?' May 17 14:03:05 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 2c: check cached attribute updates on 2 mtpt's root ================================== 14:03:05 (1305666185) May 17 14:03:05 client-5 xinetd[3391]: EXIT: shell status=0 pid=1160 duration=0(sec) May 17 14:03:05 client-5 xinetd[3391]: START: shell pid=1253 from=192.168.4.5 May 17 14:03:05 client-5 rshd[1254]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:03:06 client-5 xinetd[3391]: EXIT: shell status=0 pid=1253 duration=1(sec) May 17 14:03:06 client-5 xinetd[3391]: START: shell pid=1424 from=192.168.4.5 May 17 14:03:06 client-5 rshd[1426]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 2d: check cached attribute updates on 2 mtpt\'s root ================================== 14:03:06 \(1305666186\)");echo XXRETCODE:$?' May 17 14:03:06 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 2d: check cached attribute updates on 2 mtpt's root ================================== 14:03:06 (1305666186) May 17 14:03:06 client-5 xinetd[3391]: EXIT: shell status=0 pid=1424 duration=0(sec) May 17 14:03:06 client-5 xinetd[3391]: START: shell pid=1519 from=192.168.4.5 May 17 14:03:06 client-5 rshd[1520]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:03:06 client-5 xinetd[3391]: EXIT: shell status=0 pid=1519 duration=0(sec) May 17 14:03:06 client-5 xinetd[3391]: START: shell pid=1692 from=192.168.4.5 May 17 14:03:06 client-5 rshd[1693]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 2e: check chmod on root is propagated to others ====================================== 14:03:06 \(1305666186\)");echo XXRETCODE:$?' May 17 14:03:06 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 2e: check chmod on root is propagated to others ====================================== 14:03:06 (1305666186) May 17 14:03:06 client-5 xinetd[3391]: EXIT: shell status=0 pid=1692 duration=0(sec) May 17 14:03:06 client-5 xinetd[3391]: START: shell pid=1791 from=192.168.4.5 May 17 14:03:06 client-5 rshd[1793]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:03:07 client-5 xinetd[3391]: EXIT: shell status=0 pid=1791 duration=1(sec) May 17 14:03:07 client-5 xinetd[3391]: START: shell pid=1969 from=192.168.4.5 May 17 14:03:07 client-5 rshd[1970]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 3: symlink on one mtpt, readlink on another ========================================== 14:03:07 \(1305666187\)");echo XXRETCODE:$?' May 17 14:03:07 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 3: symlink on one mtpt, readlink on another ========================================== 14:03:07 (1305666187) May 17 14:03:07 client-5 xinetd[3391]: EXIT: shell status=0 pid=1969 duration=0(sec) May 17 14:03:07 client-5 xinetd[3391]: START: shell pid=2064 from=192.168.4.5 May 17 14:03:07 client-5 rshd[2065]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:03:07 client-5 xinetd[3391]: EXIT: shell status=0 pid=2064 duration=0(sec) May 17 14:03:07 client-5 xinetd[3391]: START: shell pid=2242 from=192.168.4.5 May 17 14:03:07 client-5 rshd[2243]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 4: fstat validation on multiple mount points ========================================= 14:03:07 \(1305666187\)");echo XXRETCODE:$?' May 17 14:03:07 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 4: fstat validation on multiple mount points ========================================= 14:03:07 (1305666187) May 17 14:03:07 client-5 xinetd[3391]: EXIT: shell status=0 pid=2242 duration=0(sec) May 17 14:03:08 client-5 xinetd[3391]: START: shell pid=2341 from=192.168.4.5 May 17 14:03:08 client-5 rshd[2343]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:03:08 client-5 xinetd[3391]: EXIT: shell status=0 pid=2341 duration=0(sec) May 17 14:03:09 client-5 xinetd[3391]: START: shell pid=2525 from=192.168.4.5 May 17 14:03:09 client-5 rshd[2530]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 5: create a file on one mount, truncate it on the other ============================== 14:03:09 \(1305666189\)");echo XXRETCODE:$?' May 17 14:03:09 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 5: create a file on one mount, truncate it on the other ============================== 14:03:09 (1305666189) May 17 14:03:09 client-5 xinetd[3391]: EXIT: shell status=0 pid=2525 duration=0(sec) May 17 14:03:09 client-5 xinetd[3391]: START: shell pid=2626 from=192.168.4.5 May 17 14:03:09 client-5 rshd[2627]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:03:09 client-5 xinetd[3391]: EXIT: shell status=0 pid=2626 duration=0(sec) May 17 14:03:09 client-5 xinetd[3391]: START: shell pid=2798 from=192.168.4.5 May 17 14:03:09 client-5 rshd[2799]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 6: remove of open file on other node ================================================= 14:03:09 \(1305666189\)");echo XXRETCODE:$?' May 17 14:03:09 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 6: remove of open file on other node ================================================= 14:03:09 (1305666189) May 17 14:03:09 client-5 xinetd[3391]: EXIT: shell status=0 pid=2798 duration=0(sec) May 17 14:03:09 client-5 xinetd[3391]: START: shell pid=2890 from=192.168.4.5 May 17 14:03:09 client-5 rshd[2891]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:03:10 client-5 xinetd[3391]: EXIT: shell status=0 pid=2890 duration=1(sec) May 17 14:03:10 client-5 xinetd[3391]: START: shell pid=3069 from=192.168.4.5 May 17 14:03:10 client-5 rshd[3073]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 7: remove of open directory on other node ============================================ 14:03:10 \(1305666190\)");echo XXRETCODE:$?' May 17 14:03:10 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 7: remove of open directory on other node ============================================ 14:03:10 (1305666190) May 17 14:03:10 client-5 xinetd[3391]: EXIT: shell status=0 pid=3069 duration=0(sec) May 17 14:03:10 client-5 xinetd[3391]: START: shell pid=3171 from=192.168.4.5 May 17 14:03:10 client-5 rshd[3173]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:03:10 client-5 xinetd[3391]: EXIT: shell status=0 pid=3171 duration=0(sec) May 17 14:03:10 client-5 xinetd[3391]: START: shell pid=3363 from=192.168.4.5 May 17 14:03:10 client-5 rshd[3364]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 8: remove of open special file on other node ========================================= 14:03:10 \(1305666190\)");echo XXRETCODE:$?' May 17 14:03:10 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 8: remove of open special file on other node ========================================= 14:03:10 (1305666190) May 17 14:03:10 client-5 xinetd[3391]: EXIT: shell status=0 pid=3363 duration=0(sec) May 17 14:03:11 client-5 xinetd[3391]: START: shell pid=3466 from=192.168.4.5 May 17 14:03:11 client-5 rshd[3467]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:03:11 client-5 xinetd[3391]: EXIT: shell status=0 pid=3466 duration=0(sec) May 17 14:03:11 client-5 xinetd[3391]: START: shell pid=3653 from=192.168.4.5 May 17 14:03:11 client-5 rshd[3654]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 9: append of file with sub-page size on multiple mounts ============================== 14:03:11 \(1305666191\)");echo XXRETCODE:$?' May 17 14:03:11 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 9: append of file with sub-page size on multiple mounts ============================== 14:03:11 (1305666191) May 17 14:03:11 client-5 xinetd[3391]: EXIT: shell status=0 pid=3653 duration=0(sec) May 17 14:03:11 client-5 xinetd[3391]: START: shell pid=3758 from=192.168.4.5 May 17 14:03:11 client-5 rshd[3759]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:03:11 client-5 xinetd[3391]: EXIT: shell status=0 pid=3758 duration=0(sec) May 17 14:03:12 client-5 xinetd[3391]: START: shell pid=3930 from=192.168.4.5 May 17 14:03:12 client-5 rshd[3931]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 10a: write of file with sub-page size on multiple mounts ============================= 14:03:12 \(1305666192\)");echo XXRETCODE:$?' May 17 14:03:12 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 10a: write of file with sub-page size on multiple mounts ============================= 14:03:12 (1305666192) May 17 14:03:12 client-5 xinetd[3391]: EXIT: shell status=0 pid=3930 duration=0(sec) May 17 14:03:12 client-5 xinetd[3391]: START: shell pid=4070 from=192.168.4.5 May 17 14:03:12 client-5 rshd[4071]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:03:12 client-5 xinetd[3391]: EXIT: shell status=0 pid=4070 duration=0(sec) May 17 14:03:12 client-5 xinetd[3391]: START: shell pid=4241 from=192.168.4.5 May 17 14:03:12 client-5 rshd[4243]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 10b: write of file with sub-page size on multiple mounts ============================= 14:03:12 \(1305666192\)");echo XXRETCODE:$?' May 17 14:03:12 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 10b: write of file with sub-page size on multiple mounts ============================= 14:03:12 (1305666192) May 17 14:03:12 client-5 xinetd[3391]: EXIT: shell status=0 pid=4241 duration=0(sec) May 17 14:03:13 client-5 xinetd[3391]: START: shell pid=4342 from=192.168.4.5 May 17 14:03:13 client-5 rshd[4343]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:03:13 client-5 xinetd[3391]: EXIT: shell status=0 pid=4342 duration=0(sec) May 17 14:03:13 client-5 xinetd[3391]: START: shell pid=4514 from=192.168.4.5 May 17 14:03:13 client-5 rshd[4515]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 11: execution of file opened for write should return error =========================== 14:03:13 \(1305666193\)");echo XXRETCODE:$?' May 17 14:03:13 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 11: execution of file opened for write should return error =========================== 14:03:13 (1305666193) May 17 14:03:13 client-5 xinetd[3391]: EXIT: shell status=0 pid=4514 duration=0(sec) May 17 14:03:13 client-5 xinetd[3391]: START: shell pid=4610 from=192.168.4.5 May 17 14:03:13 client-5 rshd[4612]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:03:13 client-5 xinetd[3391]: EXIT: shell status=0 pid=4610 duration=0(sec) May 17 14:03:14 client-5 xinetd[3391]: START: shell pid=4783 from=192.168.4.5 May 17 14:03:14 client-5 rshd[4784]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 12: test lock ordering \(link, stat, unlink\) ========================================== 14:03:13 \(1305666193\)");echo XXRETCODE:$?' May 17 14:03:14 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 12: test lock ordering (link, stat, unlink) ========================================== 14:03:13 (1305666193) May 17 14:03:14 client-5 xinetd[3391]: EXIT: shell status=0 pid=4783 duration=0(sec) May 17 14:03:14 client-5 kernel: Lustre: DEBUG MARKER: start dir: /mnt/lustre/lockdir=144115305952669262 file: /mnt/lustre/lockdir/lockfile=144115305952669260 May 17 14:05:36 client-5 xinetd[3391]: START: shell pid=4903 from=192.168.4.5 May 17 14:05:36 client-5 rshd[4906]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:05:36 client-5 xinetd[3391]: EXIT: shell status=0 pid=4903 duration=0(sec) May 17 14:05:36 client-5 xinetd[3391]: START: shell pid=5077 from=192.168.4.5 May 17 14:05:36 client-5 rshd[5078]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 13: test directory page revocation =================================================== 14:05:36 \(1305666336\)");echo XXRETCODE:$?' May 17 14:05:36 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 13: test directory page revocation =================================================== 14:05:36 (1305666336) May 17 14:05:36 client-5 xinetd[3391]: EXIT: shell status=0 pid=5077 duration=0(sec) May 17 14:05:36 client-5 xinetd[3391]: START: shell pid=5180 from=192.168.4.5 May 17 14:05:36 client-5 rshd[5182]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:05:36 client-5 xinetd[3391]: EXIT: shell status=0 pid=5180 duration=0(sec) May 17 14:05:36 client-5 xinetd[3391]: START: shell pid=5353 from=192.168.4.5 May 17 14:05:36 client-5 rshd[5354]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 14: execution of file open for write returns -ETXTBSY ================================ 14:05:36 \(1305666336\)");echo XXRETCODE:$?' May 17 14:05:37 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 14: execution of file open for write returns -ETXTBSY ================================ 14:05:36 (1305666336) May 17 14:05:37 client-5 xinetd[3391]: EXIT: shell status=0 pid=5353 duration=1(sec) May 17 14:05:37 client-5 xinetd[3391]: START: shell pid=5448 from=192.168.4.5 May 17 14:05:37 client-5 rshd[5451]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:05:37 client-5 xinetd[3391]: EXIT: shell status=0 pid=5448 duration=0(sec) May 17 14:05:37 client-5 xinetd[3391]: START: shell pid=5620 from=192.168.4.5 May 17 14:05:37 client-5 rshd[5623]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 14a: open\(RDWR\) of executing file returns -ETXTBSY =================================== 14:05:37 \(1305666337\)");echo XXRETCODE:$?' May 17 14:05:37 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 14a: open(RDWR) of executing file returns -ETXTBSY =================================== 14:05:37 (1305666337) May 17 14:05:37 client-5 xinetd[3391]: EXIT: shell status=0 pid=5620 duration=0(sec) May 17 14:05:37 client-5 xinetd[3391]: START: shell pid=5720 from=192.168.4.5 May 17 14:05:37 client-5 rshd[5722]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:05:37 client-5 xinetd[3391]: EXIT: shell status=0 pid=5720 duration=0(sec) May 17 14:05:37 client-5 xinetd[3391]: START: shell pid=5884 from=192.168.4.5 May 17 14:05:38 client-5 rshd[5885]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark SKIP: sanityn test_14b skipping ALWAYS excluded test 14b");echo XXRETCODE:$?' May 17 14:05:38 client-5 kernel: Lustre: DEBUG MARKER: SKIP: sanityn test_14b skipping ALWAYS excluded test 14b May 17 14:05:38 client-5 xinetd[3391]: EXIT: shell status=0 pid=5884 duration=1(sec) May 17 14:05:38 client-5 xinetd[3391]: START: shell pid=5989 from=192.168.4.5 May 17 14:05:38 client-5 rshd[5990]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 14c: open\(O_TRUNC\) of executing file return -ETXTBSY ================================= 14:05:38 \(1305666338\)");echo XXRETCODE:$?' May 17 14:05:38 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ================================= 14:05:38 (1305666338) May 17 14:05:38 client-5 xinetd[3391]: EXIT: shell status=0 pid=5989 duration=0(sec) May 17 14:05:38 client-5 xinetd[3391]: START: shell pid=6090 from=192.168.4.5 May 17 14:05:38 client-5 rshd[6091]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:05:38 client-5 xinetd[3391]: EXIT: shell status=0 pid=6090 duration=0(sec) May 17 14:05:38 client-5 xinetd[3391]: START: shell pid=6262 from=192.168.4.5 May 17 14:05:38 client-5 rshd[6263]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 14d: chmod of executing file is still possible ======================================= 14:05:38 \(1305666338\)");echo XXRETCODE:$?' May 17 14:05:38 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 14d: chmod of executing file is still possible ======================================= 14:05:38 (1305666338) May 17 14:05:38 client-5 xinetd[3391]: EXIT: shell status=0 pid=6262 duration=0(sec) May 17 14:05:38 client-5 xinetd[3391]: START: shell pid=6361 from=192.168.4.5 May 17 14:05:38 client-5 rshd[6362]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark chmod");echo XXRETCODE:$?' May 17 14:05:38 client-5 kernel: Lustre: DEBUG MARKER: chmod May 17 14:05:38 client-5 xinetd[3391]: EXIT: shell status=0 pid=6361 duration=0(sec) May 17 14:05:39 client-5 xinetd[3391]: START: shell pid=6455 from=192.168.4.5 May 17 14:05:39 client-5 rshd[6457]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:05:39 client-5 xinetd[3391]: EXIT: shell status=0 pid=6455 duration=0(sec) May 17 14:05:39 client-5 xinetd[3391]: START: shell pid=6628 from=192.168.4.5 May 17 14:05:39 client-5 rshd[6629]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 15: test out-of-space with multiple writers ========================================== 14:05:39 \(1305666339\)");echo XXRETCODE:$?' May 17 14:05:39 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 15: test out-of-space with multiple writers ========================================== 14:05:39 (1305666339) May 17 14:05:39 client-5 xinetd[3391]: EXIT: shell status=0 pid=6628 duration=0(sec) May 17 14:05:40 client-5 xinetd[3391]: START: shell pid=6740 from=192.168.4.5 May 17 14:05:40 client-5 rshd[6741]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark SKIP: sanityn test_15 oos2.sh: 53532968kB free gt MAXFREE 2400000kB, increase 2400000 \(or reduce test fs size\) to proceed");echo XXRETCODE:$?' May 17 14:05:40 client-5 kernel: Lustre: DEBUG MARKER: SKIP: sanityn test_15 oos2.sh: 53532968kB free gt MAXFREE 2400000kB, increase 2400000 (or reduce test fs size) to proceed May 17 14:05:40 client-5 xinetd[3391]: EXIT: shell status=0 pid=6740 duration=0(sec) May 17 14:05:40 client-5 xinetd[3391]: START: shell pid=6834 from=192.168.4.5 May 17 14:05:40 client-5 rshd[6835]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:05:40 client-5 xinetd[3391]: EXIT: shell status=0 pid=6834 duration=0(sec) May 17 14:05:41 client-5 xinetd[3391]: START: shell pid=7006 from=192.168.4.5 May 17 14:05:41 client-5 rshd[7007]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 16: 2500 iterations of dual-mount fsx ================================================ 14:05:41 \(1305666341\)");echo XXRETCODE:$?' May 17 14:05:41 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 16: 2500 iterations of dual-mount fsx ================================================ 14:05:41 (1305666341) May 17 14:05:41 client-5 xinetd[3391]: EXIT: shell status=0 pid=7006 duration=0(sec) May 17 14:05:44 client-5 kernel: format at cl_lock.c:1592:cl_lock_modify doesn't end in newline May 17 14:06:16 client-5 last message repeated 56 times May 17 14:06:45 client-5 last message repeated 57 times May 17 14:06:46 client-5 xinetd[3391]: START: shell pid=7104 from=192.168.4.5 May 17 14:06:46 client-5 rshd[7105]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:06:46 client-5 xinetd[3391]: EXIT: shell status=0 pid=7104 duration=0(sec) May 17 14:06:46 client-5 xinetd[3391]: START: shell pid=7276 from=192.168.4.5 May 17 14:06:46 client-5 rshd[7277]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 17: resource creation\/LVB creation race ============================================== 14:06:46 \(1305666406\)");echo XXRETCODE:$?' May 17 14:06:47 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 17: resource creation/LVB creation race ============================================== 14:06:46 (1305666406) May 17 14:06:47 client-5 xinetd[3391]: EXIT: shell status=0 pid=7276 duration=1(sec) May 17 14:06:47 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 14:06:47 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 14:06:49 client-5 xinetd[3391]: START: shell pid=7425 from=192.168.4.5 May 17 14:06:49 client-5 rshd[7428]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:06:49 client-5 xinetd[3391]: EXIT: shell status=0 pid=7425 duration=0(sec) May 17 14:06:49 client-5 xinetd[3391]: START: shell pid=7599 from=192.168.4.5 May 17 14:06:49 client-5 rshd[7600]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 18: mmap sanity check ================================================================ 14:06:49 \(1305666409\)");echo XXRETCODE:$?' May 17 14:06:49 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 18: mmap sanity check ================================================================ 14:06:49 (1305666409) May 17 14:06:49 client-5 xinetd[3391]: EXIT: shell status=0 pid=7599 duration=0(sec) May 17 14:06:55 client-5 xinetd[3391]: START: shell pid=7699 from=192.168.4.5 May 17 14:06:55 client-5 rshd[7701]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:06:55 client-5 xinetd[3391]: EXIT: shell status=0 pid=7699 duration=0(sec) May 17 14:06:55 client-5 xinetd[3391]: START: shell pid=7863 from=192.168.4.5 May 17 14:06:55 client-5 rshd[7864]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark SKIP: sanityn test_19 skipping ALWAYS excluded test 19");echo XXRETCODE:$?' May 17 14:06:55 client-5 kernel: Lustre: DEBUG MARKER: SKIP: sanityn test_19 skipping ALWAYS excluded test 19 May 17 14:06:55 client-5 xinetd[3391]: EXIT: shell status=0 pid=7863 duration=0(sec) May 17 14:06:56 client-5 xinetd[3391]: START: shell pid=7968 from=192.168.4.5 May 17 14:06:56 client-5 rshd[7969]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 20: test extra readahead page left in cache ========================================== 14:06:55 \(1305666415\)");echo XXRETCODE:$?' May 17 14:06:56 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 20: test extra readahead page left in cache ========================================== 14:06:55 (1305666415) May 17 14:06:56 client-5 xinetd[3391]: EXIT: shell status=0 pid=7968 duration=0(sec) May 17 14:06:56 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 14:06:56 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 14:06:57 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 14:06:57 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 14:06:58 client-5 xinetd[3391]: START: shell pid=8109 from=192.168.4.5 May 17 14:06:58 client-5 rshd[8110]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:06:58 client-5 xinetd[3391]: EXIT: shell status=0 pid=8109 duration=0(sec) May 17 14:06:58 client-5 xinetd[3391]: START: shell pid=8281 from=192.168.4.5 May 17 14:06:58 client-5 rshd[8282]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 21: Try to remove mountpoint on another dir ========================================= 14:06:58 \(1305666418\)");echo XXRETCODE:$?' May 17 14:06:58 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 21: Try to remove mountpoint on another dir ========================================= 14:06:58 (1305666418) May 17 14:06:58 client-5 xinetd[3391]: EXIT: shell status=0 pid=8281 duration=0(sec) May 17 14:06:58 client-5 xinetd[3391]: START: shell pid=8376 from=192.168.4.5 May 17 14:06:58 client-5 rshd[8378]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:06:58 client-5 xinetd[3391]: EXIT: shell status=0 pid=8376 duration=0(sec) May 17 14:06:59 client-5 xinetd[3391]: START: shell pid=8548 from=192.168.4.5 May 17 14:06:59 client-5 rshd[8550]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 23: others should see updated atime while another read=============================== 14:06:58 \(1305666418\)");echo XXRETCODE:$?' May 17 14:06:59 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 23: others should see updated atime while another read=============================== 14:06:58 (1305666418) May 17 14:06:59 client-5 xinetd[3391]: EXIT: shell status=0 pid=8548 duration=0(sec) May 17 14:06:59 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 14:06:59 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 14:08:00 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc start May 17 14:08:00 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc stop May 17 14:08:00 client-5 xinetd[3391]: START: shell pid=8681 from=192.168.4.5 May 17 14:08:00 client-5 rshd[8682]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:08:00 client-5 xinetd[3391]: EXIT: shell status=0 pid=8681 duration=0(sec) May 17 14:08:00 client-5 xinetd[3391]: START: shell pid=8852 from=192.168.4.5 May 17 14:08:00 client-5 rshd[8854]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 24a: lfs df [-ih] [path] test ======================================================== 14:08:00 \(1305666480\)");echo XXRETCODE:$?' May 17 14:08:00 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 24a: lfs df [-ih] [path] test ======================================================== 14:08:00 (1305666480) May 17 14:08:00 client-5 xinetd[3391]: EXIT: shell status=0 pid=8852 duration=0(sec) May 17 14:08:00 client-5 kernel: Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request May 17 14:08:00 client-5 kernel: Lustre: lustre-OST0000-osc-ffff810321d30800: Connection to service lustre-OST0000 via nid 192.168.4.129@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 14:08:00 client-5 kernel: LustreError: 167-0: This client was evicted by lustre-OST0000; in progress operations using this service will fail. May 17 14:08:00 client-5 kernel: Lustre: lustre-OST0000-osc-ffff810321d30800: Connection restored to service lustre-OST0000 using nid 192.168.4.129@o2ib. May 17 14:08:00 client-5 xinetd[3391]: START: shell pid=8959 from=192.168.4.5 May 17 14:08:00 client-5 rshd[8961]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:08:01 client-5 xinetd[3391]: EXIT: shell status=0 pid=8959 duration=1(sec) May 17 14:08:01 client-5 xinetd[3391]: START: shell pid=9132 from=192.168.4.5 May 17 14:08:01 client-5 rshd[9133]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 24b: lfs df should show both filesystems ============================================= 14:08:01 \(1305666481\)");echo XXRETCODE:$?' May 17 14:08:01 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 24b: lfs df should show both filesystems ============================================= 14:08:01 (1305666481) May 17 14:08:01 client-5 xinetd[3391]: EXIT: shell status=0 pid=9132 duration=0(sec) May 17 14:08:01 client-5 xinetd[3391]: START: shell pid=9227 from=192.168.4.5 May 17 14:08:01 client-5 rshd[9228]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:08:01 client-5 xinetd[3391]: EXIT: shell status=0 pid=9227 duration=0(sec) May 17 14:08:01 client-5 xinetd[3391]: START: shell pid=9399 from=192.168.4.5 May 17 14:08:01 client-5 rshd[9400]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 25: change ACL on one mountpoint be seen on another ================================== 14:08:01 \(1305666481\)");echo XXRETCODE:$?' May 17 14:08:01 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 25: change ACL on one mountpoint be seen on another ================================== 14:08:01 (1305666481) May 17 14:08:01 client-5 xinetd[3391]: EXIT: shell status=0 pid=9399 duration=0(sec) May 17 14:08:02 client-5 xinetd[3391]: START: shell pid=9505 from=192.168.4.5 May 17 14:08:02 client-5 rshd[9507]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:08:02 client-5 xinetd[3391]: EXIT: shell status=0 pid=9505 duration=0(sec) May 17 14:08:02 client-5 xinetd[3391]: START: shell pid=9678 from=192.168.4.5 May 17 14:08:02 client-5 rshd[9679]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 26a: allow mtime to get older ======================================================== 14:08:02 \(1305666482\)");echo XXRETCODE:$?' May 17 14:08:02 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 26a: allow mtime to get older ======================================================== 14:08:02 (1305666482) May 17 14:08:02 client-5 xinetd[3391]: EXIT: shell status=0 pid=9678 duration=0(sec) May 17 14:08:03 client-5 xinetd[3391]: START: shell pid=9770 from=192.168.4.5 May 17 14:08:03 client-5 rshd[9771]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:08:03 client-5 xinetd[3391]: EXIT: shell status=0 pid=9770 duration=0(sec) May 17 14:08:03 client-5 xinetd[3391]: START: shell pid=9942 from=192.168.4.5 May 17 14:08:03 client-5 rshd[9943]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 26b: sync mtime between ost and mds ================================================== 14:08:03 \(1305666483\)");echo XXRETCODE:$?' May 17 14:08:03 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 26b: sync mtime between ost and mds ================================================== 14:08:03 (1305666483) May 17 14:08:04 client-5 xinetd[3391]: EXIT: shell status=0 pid=9942 duration=1(sec) May 17 14:08:06 client-5 xinetd[3391]: START: shell pid=10039 from=192.168.4.5 May 17 14:08:06 client-5 rshd[10040]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:08:06 client-5 xinetd[3391]: EXIT: shell status=0 pid=10039 duration=0(sec) May 17 14:08:06 client-5 xinetd[3391]: START: shell pid=10211 from=192.168.4.5 May 17 14:08:06 client-5 rshd[10212]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 27: align non-overlapping extent locks from request ================================== 14:08:06 \(1305666486\)");echo XXRETCODE:$?' May 17 14:08:06 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 27: align non-overlapping extent locks from request ================================== 14:08:06 (1305666486) May 17 14:08:06 client-5 xinetd[3391]: EXIT: shell status=0 pid=10211 duration=0(sec) May 17 14:08:06 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 14:08:06 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 14:08:06 client-5 xinetd[3391]: START: shell pid=10327 from=192.168.4.5 May 17 14:08:06 client-5 rshd[10328]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark dd 1 started");echo XXRETCODE:$?' May 17 14:08:06 client-5 kernel: Lustre: DEBUG MARKER: dd 1 started May 17 14:08:06 client-5 xinetd[3391]: EXIT: shell status=0 pid=10327 duration=0(sec) May 17 14:08:06 client-5 xinetd[3391]: START: shell pid=10419 from=192.168.4.5 May 17 14:08:06 client-5 rshd[10422]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark dd 2 started");echo XXRETCODE:$?' May 17 14:08:06 client-5 kernel: Lustre: DEBUG MARKER: dd 2 started May 17 14:08:06 client-5 xinetd[3391]: EXIT: shell status=0 pid=10419 duration=0(sec) May 17 14:08:07 client-5 xinetd[3391]: START: shell pid=10515 from=192.168.4.5 May 17 14:08:07 client-5 rshd[10517]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark dd 3 finished");echo XXRETCODE:$?' May 17 14:08:08 client-5 kernel: Lustre: DEBUG MARKER: dd 3 finished May 17 14:08:08 client-5 xinetd[3391]: EXIT: shell status=0 pid=10515 duration=1(sec) May 17 14:08:08 client-5 xinetd[3391]: START: shell pid=10608 from=192.168.4.5 May 17 14:08:08 client-5 rshd[10609]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:08:08 client-5 xinetd[3391]: EXIT: shell status=0 pid=10608 duration=0(sec) May 17 14:08:08 client-5 xinetd[3391]: START: shell pid=10771 from=192.168.4.5 May 17 14:08:08 client-5 rshd[10772]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark SKIP: sanityn test_28 skipping ALWAYS excluded test 28");echo XXRETCODE:$?' May 17 14:08:08 client-5 kernel: Lustre: DEBUG MARKER: SKIP: sanityn test_28 skipping ALWAYS excluded test 28 May 17 14:08:08 client-5 xinetd[3391]: EXIT: shell status=0 pid=10771 duration=0(sec) May 17 14:08:08 client-5 xinetd[3391]: START: shell pid=10865 from=192.168.4.5 May 17 14:08:08 client-5 rshd[10868]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark SKIP: sanityn test_29 skipping ALWAYS excluded test 29");echo XXRETCODE:$?' May 17 14:08:08 client-5 kernel: Lustre: DEBUG MARKER: SKIP: sanityn test_29 skipping ALWAYS excluded test 29 May 17 14:08:08 client-5 xinetd[3391]: EXIT: shell status=0 pid=10865 duration=0(sec) May 17 14:08:08 client-5 xinetd[3391]: START: shell pid=10972 from=192.168.4.5 May 17 14:08:08 client-5 rshd[10973]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 30: recreate file race =============================================================== 14:08:08 \(1305666488\)");echo XXRETCODE:$?' May 17 14:08:08 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 30: recreate file race =============================================================== 14:08:08 (1305666488) May 17 14:08:08 client-5 xinetd[3391]: EXIT: shell status=0 pid=10972 duration=0(sec) May 17 14:08:10 client-5 xinetd[3391]: START: shell pid=11072 from=192.168.4.5 May 17 14:08:10 client-5 rshd[11073]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:08:11 client-5 xinetd[3391]: EXIT: shell status=0 pid=11072 duration=1(sec) May 17 14:08:11 client-5 xinetd[3391]: START: shell pid=11244 from=192.168.4.5 May 17 14:08:11 client-5 rshd[11245]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 31a: voluntary cancel \/ blocking ast race============================================= 14:08:11 \(1305666491\)");echo XXRETCODE:$?' May 17 14:08:11 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 31a: voluntary cancel / blocking ast race============================================= 14:08:11 (1305666491) May 17 14:08:11 client-5 xinetd[3391]: EXIT: shell status=0 pid=11244 duration=0(sec) May 17 14:08:11 client-5 kernel: LustreError: 20975:0:(obd_support.h:456:obd_fail_check_set()) *** obd_fail_loc=314 *** May 17 14:08:11 client-5 kernel: LustreError: 28091:0:(obd_support.h:456:obd_fail_check_set()) *** obd_fail_loc=314 *** May 17 14:08:11 client-5 xinetd[3391]: START: shell pid=11343 from=192.168.4.5 May 17 14:08:11 client-5 rshd[11344]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:08:11 client-5 xinetd[3391]: EXIT: shell status=0 pid=11343 duration=0(sec) May 17 14:08:11 client-5 xinetd[3391]: START: shell pid=11515 from=192.168.4.5 May 17 14:08:11 client-5 rshd[11516]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 31b: voluntary OST cancel \/ blocking ast race========================================= 14:08:11 \(1305666491\)");echo XXRETCODE:$?' May 17 14:08:11 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 31b: voluntary OST cancel / blocking ast race========================================= 14:08:11 (1305666491) May 17 14:08:11 client-5 xinetd[3391]: EXIT: shell status=0 pid=11515 duration=0(sec) May 17 14:08:12 client-5 kernel: LustreError: 20975:0:(obd_support.h:456:obd_fail_check_set()) *** obd_fail_loc=314 *** May 17 14:08:12 client-5 kernel: LustreError: 20975:0:(obd_support.h:456:obd_fail_check_set()) Skipped 1 previous similar message May 17 14:08:12 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.129@o2ib. The ost_statfs operation failed with -107 May 17 14:08:12 client-5 kernel: Lustre: lustre-OST0000-osc-ffff8102f6091800: Connection to service lustre-OST0000 via nid 192.168.4.129@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 14:08:12 client-5 kernel: LustreError: 167-0: This client was evicted by lustre-OST0000; in progress operations using this service will fail. May 17 14:08:12 client-5 kernel: LustreError: 11623:0:(ldlm_resource.c:748:ldlm_resource_complain()) Namespace lustre-OST0000-osc-ffff8102f6091800 resource refcount nonzero (1) after lock cleanup; forcing cleanup. May 17 14:08:12 client-5 kernel: LustreError: 11623:0:(ldlm_resource.c:754:ldlm_resource_complain()) Resource: ffff81033a1acc00 (67165/0/0/0) (rc: 1) May 17 14:08:12 client-5 kernel: Lustre: lustre-OST0000-osc-ffff8102f6091800: Connection restored to service lustre-OST0000 using nid 192.168.4.129@o2ib. May 17 14:08:12 client-5 xinetd[3391]: START: shell pid=11679 from=192.168.4.5 May 17 14:08:12 client-5 rshd[11681]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:08:12 client-5 xinetd[3391]: EXIT: shell status=0 pid=11679 duration=0(sec) May 17 14:08:12 client-5 xinetd[3391]: START: shell pid=11850 from=192.168.4.5 May 17 14:08:12 client-5 rshd[11853]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 32a: lockless truncate =============================================================== 14:08:12 \(1305666492\)");echo XXRETCODE:$?' May 17 14:08:12 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 32a: lockless truncate =============================================================== 14:08:12 (1305666492) May 17 14:08:12 client-5 xinetd[3391]: EXIT: shell status=0 pid=11850 duration=0(sec) May 17 14:08:12 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 14:08:13 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 14:08:13 client-5 xinetd[3391]: START: shell pid=11976 from=192.168.4.5 May 17 14:08:13 client-5 rshd[11977]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark checking cached lockless truncate");echo XXRETCODE:$?' May 17 14:08:13 client-5 kernel: Lustre: DEBUG MARKER: checking cached lockless truncate May 17 14:08:13 client-5 xinetd[3391]: EXIT: shell status=0 pid=11976 duration=0(sec) May 17 14:08:13 client-5 xinetd[3391]: START: shell pid=12075 from=192.168.4.5 May 17 14:08:13 client-5 rshd[12076]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark checking not cached lockless truncate");echo XXRETCODE:$?' May 17 14:08:13 client-5 kernel: Lustre: DEBUG MARKER: checking not cached lockless truncate May 17 14:08:13 client-5 xinetd[3391]: EXIT: shell status=0 pid=12075 duration=0(sec) May 17 14:08:13 client-5 xinetd[3391]: START: shell pid=12174 from=192.168.4.5 May 17 14:08:13 client-5 rshd[12175]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark disabled lockless truncate");echo XXRETCODE:$?' May 17 14:08:13 client-5 kernel: Lustre: DEBUG MARKER: disabled lockless truncate May 17 14:08:13 client-5 xinetd[3391]: EXIT: shell status=0 pid=12174 duration=0(sec) May 17 14:08:13 client-5 xinetd[3391]: START: shell pid=12323 from=192.168.4.5 May 17 14:08:13 client-5 rshd[12324]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:08:13 client-5 xinetd[3391]: EXIT: shell status=0 pid=12323 duration=0(sec) May 17 14:08:14 client-5 xinetd[3391]: START: shell pid=12495 from=192.168.4.5 May 17 14:08:14 client-5 rshd[12496]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 32b: lockless i\/o ==================================================================== 14:08:14 \(1305666494\)");echo XXRETCODE:$?' May 17 14:08:14 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 32b: lockless i/o ==================================================================== 14:08:14 (1305666494) May 17 14:08:14 client-5 xinetd[3391]: EXIT: shell status=0 pid=12495 duration=0(sec) May 17 14:08:17 client-5 xinetd[3391]: START: shell pid=12880 from=192.168.4.5 May 17 14:08:17 client-5 rshd[12881]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:08:17 client-5 xinetd[3391]: EXIT: shell status=0 pid=12880 duration=0(sec) May 17 14:08:17 client-5 xinetd[3391]: START: shell pid=13052 from=192.168.4.5 May 17 14:08:17 client-5 rshd[13053]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 33a: commit on sharing, cross crete\/delete, 2 clients, benchmark ===================== 14:08:17 \(1305666497\)");echo XXRETCODE:$?' May 17 14:08:17 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 33a: commit on sharing, cross crete/delete, 2 clients, benchmark ===================== 14:08:17 (1305666497) May 17 14:08:17 client-5 xinetd[3391]: EXIT: shell status=0 pid=13052 duration=0(sec) May 17 14:08:17 client-5 xinetd[3391]: START: shell pid=13128 from=192.168.4.5 May 17 14:08:17 client-5 rshd[13129]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mkdir -p /mnt/lustre/d0.sanityn/d33-\$(hostname)-1");echo XXRETCODE:$?' May 17 14:08:17 client-5 xinetd[3391]: EXIT: shell status=0 pid=13128 duration=0(sec) May 17 14:08:18 client-5 xinetd[3391]: START: shell pid=13207 from=192.168.4.5 May 17 14:08:18 client-5 rshd[13208]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "createmany -o /mnt/lustre/d0.sanityn/d33-\$(hostname)-1/f- -r /mnt/lustre2/d0.sanityn/d33-\$(hostname)-1/f- 10000 > /dev/null 2>&1");echo XXRETCODE:$?' May 17 14:08:37 client-5 xinetd[3391]: EXIT: shell status=0 pid=13207 duration=19(sec) May 17 14:08:37 client-5 xinetd[3391]: START: shell pid=13292 from=192.168.4.5 May 17 14:08:38 client-5 rshd[13293]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mkdir -p /mnt/lustre/d0.sanityn/d33-\$(hostname)-2");echo XXRETCODE:$?' May 17 14:08:38 client-5 xinetd[3391]: EXIT: shell status=0 pid=13292 duration=1(sec) May 17 14:08:38 client-5 xinetd[3391]: START: shell pid=13371 from=192.168.4.5 May 17 14:08:38 client-5 rshd[13372]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "createmany -o /mnt/lustre/d0.sanityn/d33-\$(hostname)-2/f- -r /mnt/lustre2/d0.sanityn/d33-\$(hostname)-2/f- 10000 > /dev/null 2>&1");echo XXRETCODE:$?' May 17 14:08:56 client-5 xinetd[3391]: EXIT: shell status=0 pid=13371 duration=18(sec) May 17 14:08:57 client-5 xinetd[3391]: START: shell pid=13451 from=192.168.4.5 May 17 14:08:57 client-5 rshd[13453]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mkdir -p /mnt/lustre/d0.sanityn/d33-\$(hostname)-3");echo XXRETCODE:$?' May 17 14:08:57 client-5 xinetd[3391]: EXIT: shell status=0 pid=13451 duration=0(sec) May 17 14:08:58 client-5 xinetd[3391]: START: shell pid=13531 from=192.168.4.5 May 17 14:08:58 client-5 rshd[13532]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "createmany -o /mnt/lustre/d0.sanityn/d33-\$(hostname)-3/f- -r /mnt/lustre2/d0.sanityn/d33-\$(hostname)-3/f- 10000 > /dev/null 2>&1");echo XXRETCODE:$?' May 17 14:09:16 client-5 xinetd[3391]: EXIT: shell status=0 pid=13531 duration=18(sec) May 17 14:09:17 client-5 xinetd[3391]: START: shell pid=13620 from=192.168.4.5 May 17 14:09:17 client-5 rshd[13621]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mkdir -p /mnt/lustre/d0.sanityn/d33-\$(hostname)-1");echo XXRETCODE:$?' May 17 14:09:17 client-5 xinetd[3391]: EXIT: shell status=0 pid=13620 duration=0(sec) May 17 14:09:17 client-5 xinetd[3391]: START: shell pid=13699 from=192.168.4.5 May 17 14:09:17 client-5 rshd[13700]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "createmany -o /mnt/lustre/d0.sanityn/d33-\$(hostname)-1/f- -r /mnt/lustre2/d0.sanityn/d33-\$(hostname)-1/f- 10000 > /dev/null 2>&1");echo XXRETCODE:$?' May 17 14:09:54 client-5 xinetd[3391]: EXIT: shell status=0 pid=13699 duration=37(sec) May 17 14:09:55 client-5 xinetd[3391]: START: shell pid=13784 from=192.168.4.5 May 17 14:09:55 client-5 rshd[13785]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mkdir -p /mnt/lustre/d0.sanityn/d33-\$(hostname)-2");echo XXRETCODE:$?' May 17 14:09:55 client-5 xinetd[3391]: EXIT: shell status=0 pid=13784 duration=0(sec) May 17 14:09:55 client-5 xinetd[3391]: START: shell pid=13863 from=192.168.4.5 May 17 14:09:55 client-5 rshd[13864]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "createmany -o /mnt/lustre/d0.sanityn/d33-\$(hostname)-2/f- -r /mnt/lustre2/d0.sanityn/d33-\$(hostname)-2/f- 10000 > /dev/null 2>&1");echo XXRETCODE:$?' May 17 14:10:32 client-5 xinetd[3391]: EXIT: shell status=0 pid=13863 duration=37(sec) May 17 14:10:32 client-5 xinetd[3391]: START: shell pid=13944 from=192.168.4.5 May 17 14:10:32 client-5 rshd[13945]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mkdir -p /mnt/lustre/d0.sanityn/d33-\$(hostname)-3");echo XXRETCODE:$?' May 17 14:10:32 client-5 xinetd[3391]: EXIT: shell status=0 pid=13944 duration=0(sec) May 17 14:10:32 client-5 xinetd[3391]: START: shell pid=14023 from=192.168.4.5 May 17 14:10:32 client-5 rshd[14024]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "createmany -o /mnt/lustre/d0.sanityn/d33-\$(hostname)-3/f- -r /mnt/lustre2/d0.sanityn/d33-\$(hostname)-3/f- 10000 > /dev/null 2>&1");echo XXRETCODE:$?' May 17 14:11:07 client-5 xinetd[3391]: EXIT: shell status=0 pid=14023 duration=35(sec) May 17 14:11:07 client-5 xinetd[3391]: START: shell pid=14161 from=192.168.4.5 May 17 14:11:07 client-5 rshd[14164]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:11:07 client-5 xinetd[3391]: EXIT: shell status=0 pid=14161 duration=0(sec) May 17 14:11:08 client-5 xinetd[3391]: START: shell pid=14335 from=192.168.4.5 May 17 14:11:08 client-5 rshd[14336]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 34: no lock timeout under IO ========================================================= 14:11:08 \(1305666668\)");echo XXRETCODE:$?' May 17 14:11:08 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 34: no lock timeout under IO ========================================================= 14:11:08 (1305666668) May 17 14:11:08 client-5 xinetd[3391]: EXIT: shell status=0 pid=14335 duration=0(sec) May 17 14:11:26 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.129@o2ib. The ost_write operation failed with -107 May 17 14:11:26 client-5 kernel: Lustre: lustre-OST0003-osc-ffff810321d30800: Connection to service lustre-OST0003 via nid 192.168.4.129@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 14:11:26 client-5 kernel: LustreError: 167-0: This client was evicted by lustre-OST0003; in progress operations using this service will fail. May 17 14:11:26 client-5 kernel: LustreError: 14604:0:(ldlm_resource.c:748:ldlm_resource_complain()) Namespace lustre-OST0003-osc-ffff810321d30800 resource refcount nonzero (1) after lock cleanup; forcing cleanup. May 17 14:11:26 client-5 kernel: LustreError: 14604:0:(ldlm_resource.c:754:ldlm_resource_complain()) Resource: ffff8102fb3861c0 (87137/0/0/0) (rc: 0) May 17 14:11:26 client-5 kernel: Lustre: lustre-OST0003-osc-ffff810321d30800: Connection restored to service lustre-OST0003 using nid 192.168.4.129@o2ib. May 17 14:11:30 client-5 xinetd[3391]: START: shell pid=14702 from=192.168.4.5 May 17 14:11:30 client-5 rshd[14703]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:11:30 client-5 xinetd[3391]: EXIT: shell status=0 pid=14702 duration=0(sec) May 17 14:11:30 client-5 xinetd[3391]: START: shell pid=14865 from=192.168.4.5 May 17 14:11:30 client-5 rshd[14866]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark SKIP: sanityn test_35 skipping ALWAYS excluded test 35");echo XXRETCODE:$?' May 17 14:11:30 client-5 kernel: Lustre: DEBUG MARKER: SKIP: sanityn test_35 skipping ALWAYS excluded test 35 May 17 14:11:30 client-5 xinetd[3391]: EXIT: shell status=0 pid=14865 duration=0(sec) May 17 14:11:30 client-5 xinetd[3391]: START: shell pid=14970 from=192.168.4.5 May 17 14:11:30 client-5 rshd[14971]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 36: handle ESTALE\/open-unlink corectly =============================================== 14:11:30 \(1305666690\)");echo XXRETCODE:$?' May 17 14:11:30 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 36: handle ESTALE/open-unlink corectly =============================================== 14:11:30 (1305666690) May 17 14:11:30 client-5 xinetd[3391]: EXIT: shell status=0 pid=14970 duration=0(sec) May 17 14:11:30 client-5 kernel: Lustre: DEBUG MARKER: start test May 17 14:12:00 client-5 last message repeated 10 times May 17 14:12:03 client-5 xinetd[3391]: START: shell pid=15261 from=192.168.4.5 May 17 14:12:03 client-5 rshd[15262]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:12:03 client-5 xinetd[3391]: EXIT: shell status=0 pid=15261 duration=0(sec) May 17 14:12:03 client-5 xinetd[3391]: START: shell pid=15432 from=192.168.4.5 May 17 14:12:03 client-5 rshd[15434]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 37: check i_size is not updated for directory on close \(bug 18695\) =================== 14:12:03 \(1305666723\)");echo XXRETCODE:$?' May 17 14:12:03 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 37: check i_size is not updated for directory on close (bug 18695) =================== 14:12:03 (1305666723) May 17 14:12:03 client-5 xinetd[3391]: EXIT: shell status=0 pid=15432 duration=0(sec) May 17 14:12:12 client-5 xinetd[3391]: START: shell pid=15533 from=192.168.4.5 May 17 14:12:12 client-5 rshd[15534]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:12:12 client-5 xinetd[3391]: EXIT: shell status=0 pid=15533 duration=0(sec) May 17 14:12:12 client-5 xinetd[3391]: START: shell pid=15706 from=192.168.4.5 May 17 14:12:12 client-5 rshd[15707]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 39a: test from 11063 ================================================================= 14:12:12 \(1305666732\)");echo XXRETCODE:$?' May 17 14:12:12 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 39a: test from 11063 ================================================================= 14:12:12 (1305666732) May 17 14:12:12 client-5 xinetd[3391]: EXIT: shell status=0 pid=15706 duration=0(sec) May 17 14:12:13 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 14:12:13 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 14:12:13 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 14:12:13 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 14:12:13 client-5 xinetd[3391]: START: shell pid=15887 from=192.168.4.5 May 17 14:12:13 client-5 rshd[15888]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:12:13 client-5 xinetd[3391]: EXIT: shell status=0 pid=15887 duration=0(sec) May 17 14:12:13 client-5 xinetd[3391]: START: shell pid=16059 from=192.168.4.5 May 17 14:12:13 client-5 rshd[16060]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 39b: 11063 problem 1 ================================================================= 14:12:13 \(1305666733\)");echo XXRETCODE:$?' May 17 14:12:13 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 39b: 11063 problem 1 ================================================================= 14:12:13 (1305666733) May 17 14:12:13 client-5 xinetd[3391]: EXIT: shell status=0 pid=16059 duration=0(sec) May 17 14:12:14 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 14:12:14 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 14:12:15 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 14:12:15 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 14:12:15 client-5 xinetd[3391]: START: shell pid=16214 from=192.168.4.5 May 17 14:12:15 client-5 rshd[16215]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:12:15 client-5 xinetd[3391]: EXIT: shell status=0 pid=16214 duration=0(sec) May 17 14:12:15 client-5 xinetd[3391]: START: shell pid=16386 from=192.168.4.5 May 17 14:12:15 client-5 rshd[16387]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 39c: check truncate mtime update ===================================================== 14:12:15 \(1305666735\)");echo XXRETCODE:$?' May 17 14:12:15 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 39c: check truncate mtime update ===================================================== 14:12:15 (1305666735) May 17 14:12:15 client-5 xinetd[3391]: EXIT: shell status=0 pid=16386 duration=0(sec) May 17 14:12:16 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 14:12:16 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 14:12:16 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 14:12:16 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 14:12:17 client-5 xinetd[3391]: START: shell pid=16540 from=192.168.4.5 May 17 14:12:17 client-5 rshd[16541]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:12:17 client-5 xinetd[3391]: EXIT: shell status=0 pid=16540 duration=0(sec) May 17 14:12:17 client-5 xinetd[3391]: START: shell pid=16712 from=192.168.4.5 May 17 14:12:17 client-5 rshd[16713]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 40a: pdirops: create vs others ======================================================= 14:12:17 \(1305666737\)");echo XXRETCODE:$?' May 17 14:12:17 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 40a: pdirops: create vs others ======================================================= 14:12:17 (1305666737) May 17 14:12:17 client-5 xinetd[3391]: EXIT: shell status=0 pid=16712 duration=0(sec) May 17 14:12:34 client-5 xinetd[3391]: START: shell pid=16862 from=192.168.4.5 May 17 14:12:34 client-5 rshd[16863]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:12:34 client-5 xinetd[3391]: EXIT: shell status=0 pid=16862 duration=0(sec) May 17 14:12:35 client-5 xinetd[3391]: START: shell pid=17034 from=192.168.4.5 May 17 14:12:35 client-5 rshd[17035]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 40b: pdirops: open\|create and others ================================================= 14:12:34 \(1305666754\)");echo XXRETCODE:$?' May 17 14:12:35 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 40b: pdirops: open|create and others ================================================= 14:12:34 (1305666754) May 17 14:12:35 client-5 xinetd[3391]: EXIT: shell status=0 pid=17034 duration=0(sec) May 17 14:12:45 client-5 xinetd[3391]: START: shell pid=17164 from=192.168.4.5 May 17 14:12:45 client-5 rshd[17165]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:12:45 client-5 xinetd[3391]: EXIT: shell status=0 pid=17164 duration=0(sec) May 17 14:12:45 client-5 xinetd[3391]: START: shell pid=17336 from=192.168.4.5 May 17 14:12:45 client-5 rshd[17337]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 40c: pdirops: link and others ======================================================== 14:12:45 \(1305666765\)");echo XXRETCODE:$?' May 17 14:12:45 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 40c: pdirops: link and others ======================================================== 14:12:45 (1305666765) May 17 14:12:45 client-5 xinetd[3391]: EXIT: shell status=0 pid=17336 duration=0(sec) May 17 14:12:55 client-5 xinetd[3391]: START: shell pid=17467 from=192.168.4.5 May 17 14:12:55 client-5 rshd[17468]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:12:56 client-5 xinetd[3391]: EXIT: shell status=0 pid=17467 duration=1(sec) May 17 14:12:56 client-5 xinetd[3391]: START: shell pid=17639 from=192.168.4.5 May 17 14:12:56 client-5 rshd[17640]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 40d: pdirops: unlink and others ====================================================== 14:12:56 \(1305666776\)");echo XXRETCODE:$?' May 17 14:12:56 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 40d: pdirops: unlink and others ====================================================== 14:12:56 (1305666776) May 17 14:12:56 client-5 xinetd[3391]: EXIT: shell status=0 pid=17639 duration=0(sec) May 17 14:13:06 client-5 xinetd[3391]: START: shell pid=17769 from=192.168.4.5 May 17 14:13:06 client-5 rshd[17770]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:13:06 client-5 xinetd[3391]: EXIT: shell status=0 pid=17769 duration=0(sec) May 17 14:13:06 client-5 xinetd[3391]: START: shell pid=17941 from=192.168.4.5 May 17 14:13:06 client-5 rshd[17942]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 40e: pdirops: rename and others ====================================================== 14:13:06 \(1305666786\)");echo XXRETCODE:$?' May 17 14:13:06 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 40e: pdirops: rename and others ====================================================== 14:13:06 (1305666786) May 17 14:13:06 client-5 xinetd[3391]: EXIT: shell status=0 pid=17941 duration=0(sec) May 17 14:13:17 client-5 xinetd[3391]: START: shell pid=18068 from=192.168.4.5 May 17 14:13:17 client-5 rshd[18069]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:13:17 client-5 xinetd[3391]: EXIT: shell status=0 pid=18068 duration=0(sec) May 17 14:13:17 client-5 xinetd[3391]: START: shell pid=18240 from=192.168.4.5 May 17 14:13:17 client-5 rshd[18241]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 41a: pdirops: create vs mkdir ======================================================== 14:13:17 \(1305666797\)");echo XXRETCODE:$?' May 17 14:13:17 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 41a: pdirops: create vs mkdir ======================================================== 14:13:17 (1305666797) May 17 14:13:17 client-5 xinetd[3391]: EXIT: shell status=0 pid=18240 duration=0(sec) May 17 14:13:27 client-5 xinetd[3391]: START: shell pid=18346 from=192.168.4.5 May 17 14:13:27 client-5 rshd[18347]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:13:27 client-5 xinetd[3391]: EXIT: shell status=0 pid=18346 duration=0(sec) May 17 14:13:28 client-5 xinetd[3391]: START: shell pid=18518 from=192.168.4.5 May 17 14:13:28 client-5 rshd[18519]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 41b: pdirops: create vs create ======================================================= 14:13:28 \(1305666808\)");echo XXRETCODE:$?' May 17 14:13:28 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 41b: pdirops: create vs create ======================================================= 14:13:28 (1305666808) May 17 14:13:28 client-5 xinetd[3391]: EXIT: shell status=0 pid=18518 duration=0(sec) May 17 14:13:38 client-5 xinetd[3391]: START: shell pid=18627 from=192.168.4.5 May 17 14:13:38 client-5 rshd[18629]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:13:38 client-5 xinetd[3391]: EXIT: shell status=0 pid=18627 duration=0(sec) May 17 14:13:38 client-5 xinetd[3391]: START: shell pid=18800 from=192.168.4.5 May 17 14:13:38 client-5 rshd[18801]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 41c: pdirops: create vs link ========================================================= 14:13:38 \(1305666818\)");echo XXRETCODE:$?' May 17 14:13:38 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 41c: pdirops: create vs link ========================================================= 14:13:38 (1305666818) May 17 14:13:38 client-5 xinetd[3391]: EXIT: shell status=0 pid=18800 duration=0(sec) May 17 14:13:48 client-5 xinetd[3391]: START: shell pid=18907 from=192.168.4.5 May 17 14:13:48 client-5 rshd[18908]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:13:49 client-5 xinetd[3391]: EXIT: shell status=0 pid=18907 duration=1(sec) May 17 14:13:49 client-5 xinetd[3391]: START: shell pid=19079 from=192.168.4.5 May 17 14:13:49 client-5 rshd[19080]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 41d: pdirops: create vs unlink ======================================================= 14:13:49 \(1305666829\)");echo XXRETCODE:$?' May 17 14:13:49 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 41d: pdirops: create vs unlink ======================================================= 14:13:49 (1305666829) May 17 14:13:49 client-5 xinetd[3391]: EXIT: shell status=0 pid=19079 duration=0(sec) May 17 14:13:59 client-5 xinetd[3391]: START: shell pid=19185 from=192.168.4.5 May 17 14:13:59 client-5 rshd[19186]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:13:59 client-5 xinetd[3391]: EXIT: shell status=0 pid=19185 duration=0(sec) May 17 14:13:59 client-5 xinetd[3391]: START: shell pid=19356 from=192.168.4.5 May 17 14:13:59 client-5 rshd[19358]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 41e: pdirops: create and rename \(tgt\) ================================================ 14:13:59 \(1305666839\)");echo XXRETCODE:$?' May 17 14:14:00 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 41e: pdirops: create and rename (tgt) ================================================ 14:13:59 (1305666839) May 17 14:14:00 client-5 xinetd[3391]: EXIT: shell status=0 pid=19356 duration=1(sec) May 17 14:14:10 client-5 xinetd[3391]: START: shell pid=19462 from=192.168.4.5 May 17 14:14:10 client-5 rshd[19465]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:14:10 client-5 xinetd[3391]: EXIT: shell status=0 pid=19462 duration=0(sec) May 17 14:14:10 client-5 xinetd[3391]: START: shell pid=19636 from=192.168.4.5 May 17 14:14:10 client-5 rshd[19637]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 41f: pdirops: create and rename \(src\) ================================================ 14:14:10 \(1305666850\)");echo XXRETCODE:$?' May 17 14:14:10 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 41f: pdirops: create and rename (src) ================================================ 14:14:10 (1305666850) May 17 14:14:10 client-5 xinetd[3391]: EXIT: shell status=0 pid=19636 duration=0(sec) May 17 14:14:20 client-5 xinetd[3391]: START: shell pid=19742 from=192.168.4.5 May 17 14:14:20 client-5 rshd[19743]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:14:20 client-5 xinetd[3391]: EXIT: shell status=0 pid=19742 duration=0(sec) May 17 14:14:21 client-5 xinetd[3391]: START: shell pid=19914 from=192.168.4.5 May 17 14:14:21 client-5 rshd[19915]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 41g: pdirops: create vs getattr ====================================================== 14:14:21 \(1305666861\)");echo XXRETCODE:$?' May 17 14:14:21 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 41g: pdirops: create vs getattr ====================================================== 14:14:21 (1305666861) May 17 14:14:21 client-5 xinetd[3391]: EXIT: shell status=0 pid=19914 duration=0(sec) May 17 14:14:31 client-5 xinetd[3391]: START: shell pid=20020 from=192.168.4.5 May 17 14:14:31 client-5 rshd[20021]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:14:31 client-5 xinetd[3391]: EXIT: shell status=0 pid=20020 duration=0(sec) May 17 14:14:31 client-5 xinetd[3391]: START: shell pid=20223 from=192.168.4.5 May 17 14:14:31 client-5 rshd[20224]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 41h: pdirops: create vs readdir ====================================================== 14:14:31 \(1305666871\)");echo XXRETCODE:$?' May 17 14:14:31 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 41h: pdirops: create vs readdir ====================================================== 14:14:31 (1305666871) May 17 14:14:31 client-5 xinetd[3391]: EXIT: shell status=0 pid=20223 duration=0(sec) May 17 14:14:42 client-5 xinetd[3391]: START: shell pid=20330 from=192.168.4.5 May 17 14:14:42 client-5 rshd[20331]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:14:42 client-5 xinetd[3391]: EXIT: shell status=0 pid=20330 duration=0(sec) May 17 14:14:42 client-5 xinetd[3391]: START: shell pid=20501 from=192.168.4.5 May 17 14:14:42 client-5 rshd[20504]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 42a: pdirops: mkdir vs mkdir ========================================================= 14:14:42 \(1305666882\)");echo XXRETCODE:$?' May 17 14:14:42 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 42a: pdirops: mkdir vs mkdir ========================================================= 14:14:42 (1305666882) May 17 14:14:42 client-5 xinetd[3391]: EXIT: shell status=0 pid=20501 duration=0(sec) May 17 14:14:52 client-5 xinetd[3391]: START: shell pid=20613 from=192.168.4.5 May 17 14:14:52 client-5 rshd[20614]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:14:52 client-5 xinetd[3391]: EXIT: shell status=0 pid=20613 duration=0(sec) May 17 14:14:53 client-5 xinetd[3391]: START: shell pid=20785 from=192.168.4.5 May 17 14:14:53 client-5 rshd[20786]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 42b: pdirops: mkdir vs create ======================================================== 14:14:52 \(1305666892\)");echo XXRETCODE:$?' May 17 14:14:53 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 42b: pdirops: mkdir vs create ======================================================== 14:14:52 (1305666892) May 17 14:14:53 client-5 xinetd[3391]: EXIT: shell status=0 pid=20785 duration=0(sec) May 17 14:15:03 client-5 xinetd[3391]: START: shell pid=20889 from=192.168.4.5 May 17 14:15:03 client-5 rshd[20892]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:15:03 client-5 xinetd[3391]: EXIT: shell status=0 pid=20889 duration=0(sec) May 17 14:15:03 client-5 xinetd[3391]: START: shell pid=21072 from=192.168.4.5 May 17 14:15:03 client-5 rshd[21074]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 42c: pdirops: mkdir vs link ========================================================== 14:15:03 \(1305666903\)");echo XXRETCODE:$?' May 17 14:15:03 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 42c: pdirops: mkdir vs link ========================================================== 14:15:03 (1305666903) May 17 14:15:03 client-5 xinetd[3391]: EXIT: shell status=0 pid=21072 duration=0(sec) May 17 14:15:13 client-5 xinetd[3391]: START: shell pid=21180 from=192.168.4.5 May 17 14:15:13 client-5 rshd[21181]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:15:13 client-5 xinetd[3391]: EXIT: shell status=0 pid=21180 duration=0(sec) May 17 14:15:14 client-5 xinetd[3391]: START: shell pid=21352 from=192.168.4.5 May 17 14:15:14 client-5 rshd[21353]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 42d: pdirops: mkdir vs unlink ======================================================== 14:15:14 \(1305666914\)");echo XXRETCODE:$?' May 17 14:15:14 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 42d: pdirops: mkdir vs unlink ======================================================== 14:15:14 (1305666914) May 17 14:15:14 client-5 xinetd[3391]: EXIT: shell status=0 pid=21352 duration=0(sec) May 17 14:15:24 client-5 xinetd[3391]: START: shell pid=21458 from=192.168.4.5 May 17 14:15:24 client-5 rshd[21459]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:15:24 client-5 xinetd[3391]: EXIT: shell status=0 pid=21458 duration=0(sec) May 17 14:15:24 client-5 xinetd[3391]: START: shell pid=21631 from=192.168.4.5 May 17 14:15:24 client-5 rshd[21633]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 42e: pdirops: mkdir and rename \(tgt\) ================================================= 14:15:24 \(1305666924\)");echo XXRETCODE:$?' May 17 14:15:24 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 42e: pdirops: mkdir and rename (tgt) ================================================= 14:15:24 (1305666924) May 17 14:15:24 client-5 xinetd[3391]: EXIT: shell status=0 pid=21631 duration=0(sec) May 17 14:15:35 client-5 xinetd[3391]: START: shell pid=21739 from=192.168.4.5 May 17 14:15:35 client-5 rshd[21740]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:15:35 client-5 xinetd[3391]: EXIT: shell status=0 pid=21739 duration=0(sec) May 17 14:15:35 client-5 xinetd[3391]: START: shell pid=21911 from=192.168.4.5 May 17 14:15:35 client-5 rshd[21912]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 42f: pdirops: mkdir and rename \(src\) ================================================= 14:15:35 \(1305666935\)");echo XXRETCODE:$?' May 17 14:15:35 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 42f: pdirops: mkdir and rename (src) ================================================= 14:15:35 (1305666935) May 17 14:15:35 client-5 xinetd[3391]: EXIT: shell status=0 pid=21911 duration=0(sec) May 17 14:15:45 client-5 xinetd[3391]: START: shell pid=22017 from=192.168.4.5 May 17 14:15:45 client-5 rshd[22018]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:15:45 client-5 xinetd[3391]: EXIT: shell status=0 pid=22017 duration=0(sec) May 17 14:15:46 client-5 xinetd[3391]: START: shell pid=22189 from=192.168.4.5 May 17 14:15:46 client-5 rshd[22190]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 42g: pdirops: mkdir vs getattr ======================================================= 14:15:45 \(1305666945\)");echo XXRETCODE:$?' May 17 14:15:46 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 42g: pdirops: mkdir vs getattr ======================================================= 14:15:45 (1305666945) May 17 14:15:46 client-5 xinetd[3391]: EXIT: shell status=0 pid=22189 duration=0(sec) May 17 14:15:56 client-5 xinetd[3391]: START: shell pid=22295 from=192.168.4.5 May 17 14:15:56 client-5 rshd[22296]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:15:56 client-5 xinetd[3391]: EXIT: shell status=0 pid=22295 duration=0(sec) May 17 14:15:56 client-5 xinetd[3391]: START: shell pid=22467 from=192.168.4.5 May 17 14:15:56 client-5 rshd[22468]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 42h: pdirops: mkdir vs readdir ======================================================= 14:15:56 \(1305666956\)");echo XXRETCODE:$?' May 17 14:15:56 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 42h: pdirops: mkdir vs readdir ======================================================= 14:15:56 (1305666956) May 17 14:15:56 client-5 xinetd[3391]: EXIT: shell status=0 pid=22467 duration=0(sec) May 17 14:16:06 client-5 xinetd[3391]: START: shell pid=22578 from=192.168.4.5 May 17 14:16:06 client-5 rshd[22579]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:16:07 client-5 xinetd[3391]: EXIT: shell status=0 pid=22578 duration=1(sec) May 17 14:16:07 client-5 xinetd[3391]: START: shell pid=22750 from=192.168.4.5 May 17 14:16:07 client-5 rshd[22751]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 43a: pdirops: unlink vs mkdir ======================================================== 14:16:07 \(1305666967\)");echo XXRETCODE:$?' May 17 14:16:07 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 43a: pdirops: unlink vs mkdir ======================================================== 14:16:07 (1305666967) May 17 14:16:07 client-5 xinetd[3391]: EXIT: shell status=0 pid=22750 duration=0(sec) May 17 14:16:17 client-5 xinetd[3391]: START: shell pid=22857 from=192.168.4.5 May 17 14:16:17 client-5 rshd[22858]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:16:17 client-5 xinetd[3391]: EXIT: shell status=0 pid=22857 duration=0(sec) May 17 14:16:17 client-5 xinetd[3391]: START: shell pid=23029 from=192.168.4.5 May 17 14:16:17 client-5 rshd[23030]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 43b: pdirops: unlink vs create ======================================================= 14:16:17 \(1305666977\)");echo XXRETCODE:$?' May 17 14:16:17 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 43b: pdirops: unlink vs create ======================================================= 14:16:17 (1305666977) May 17 14:16:17 client-5 xinetd[3391]: EXIT: shell status=0 pid=23029 duration=0(sec) May 17 14:16:28 client-5 xinetd[3391]: START: shell pid=23136 from=192.168.4.5 May 17 14:16:28 client-5 rshd[23137]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:16:28 client-5 xinetd[3391]: EXIT: shell status=0 pid=23136 duration=0(sec) May 17 14:16:28 client-5 xinetd[3391]: START: shell pid=23308 from=192.168.4.5 May 17 14:16:28 client-5 rshd[23309]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 43c: pdirops: unlink vs link ========================================================= 14:16:28 \(1305666988\)");echo XXRETCODE:$?' May 17 14:16:28 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 43c: pdirops: unlink vs link ========================================================= 14:16:28 (1305666988) May 17 14:16:28 client-5 xinetd[3391]: EXIT: shell status=0 pid=23308 duration=0(sec) May 17 14:16:38 client-5 xinetd[3391]: START: shell pid=23416 from=192.168.4.5 May 17 14:16:38 client-5 rshd[23417]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:16:38 client-5 xinetd[3391]: EXIT: shell status=0 pid=23416 duration=0(sec) May 17 14:16:39 client-5 xinetd[3391]: START: shell pid=23588 from=192.168.4.5 May 17 14:16:39 client-5 rshd[23589]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 43d: pdirops: unlink vs unlink ======================================================= 14:16:39 \(1305666999\)");echo XXRETCODE:$?' May 17 14:16:39 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 43d: pdirops: unlink vs unlink ======================================================= 14:16:39 (1305666999) May 17 14:16:39 client-5 xinetd[3391]: EXIT: shell status=0 pid=23588 duration=0(sec) May 17 14:16:49 client-5 xinetd[3391]: START: shell pid=23695 from=192.168.4.5 May 17 14:16:49 client-5 rshd[23696]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:16:49 client-5 xinetd[3391]: EXIT: shell status=0 pid=23695 duration=0(sec) May 17 14:16:49 client-5 xinetd[3391]: START: shell pid=23867 from=192.168.4.5 May 17 14:16:49 client-5 rshd[23868]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 43e: pdirops: unlink and rename \(tgt\) ================================================ 14:16:49 \(1305667009\)");echo XXRETCODE:$?' May 17 14:16:49 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 43e: pdirops: unlink and rename (tgt) ================================================ 14:16:49 (1305667009) May 17 14:16:49 client-5 xinetd[3391]: EXIT: shell status=0 pid=23867 duration=0(sec) May 17 14:16:59 client-5 xinetd[3391]: START: shell pid=23975 from=192.168.4.5 May 17 14:16:59 client-5 rshd[23976]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:17:00 client-5 xinetd[3391]: EXIT: shell status=0 pid=23975 duration=1(sec) May 17 14:17:00 client-5 xinetd[3391]: START: shell pid=24147 from=192.168.4.5 May 17 14:17:00 client-5 rshd[24148]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 43f: pdirops: unlink and rename \(src\) ================================================ 14:17:00 \(1305667020\)");echo XXRETCODE:$?' May 17 14:17:00 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 43f: pdirops: unlink and rename (src) ================================================ 14:17:00 (1305667020) May 17 14:17:00 client-5 xinetd[3391]: EXIT: shell status=0 pid=24147 duration=0(sec) May 17 14:17:10 client-5 xinetd[3391]: START: shell pid=24254 from=192.168.4.5 May 17 14:17:10 client-5 rshd[24255]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:17:10 client-5 xinetd[3391]: EXIT: shell status=0 pid=24254 duration=0(sec) May 17 14:17:10 client-5 xinetd[3391]: START: shell pid=24424 from=192.168.4.5 May 17 14:17:10 client-5 rshd[24427]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 43g: pdirops: unlink vs getattr ====================================================== 14:17:10 \(1305667030\)");echo XXRETCODE:$?' May 17 14:17:11 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 43g: pdirops: unlink vs getattr ====================================================== 14:17:10 (1305667030) May 17 14:17:11 client-5 xinetd[3391]: EXIT: shell status=0 pid=24424 duration=1(sec) May 17 14:17:21 client-5 xinetd[3391]: START: shell pid=24535 from=192.168.4.5 May 17 14:17:21 client-5 rshd[24538]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:17:21 client-5 xinetd[3391]: EXIT: shell status=0 pid=24535 duration=0(sec) May 17 14:17:21 client-5 xinetd[3391]: START: shell pid=24709 from=192.168.4.5 May 17 14:17:21 client-5 rshd[24710]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 43h: pdirops: unlink vs readdir ====================================================== 14:17:21 \(1305667041\)");echo XXRETCODE:$?' May 17 14:17:21 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 43h: pdirops: unlink vs readdir ====================================================== 14:17:21 (1305667041) May 17 14:17:21 client-5 xinetd[3391]: EXIT: shell status=0 pid=24709 duration=0(sec) May 17 14:17:31 client-5 xinetd[3391]: START: shell pid=24817 from=192.168.4.5 May 17 14:17:31 client-5 rshd[24818]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:17:31 client-5 xinetd[3391]: EXIT: shell status=0 pid=24817 duration=0(sec) May 17 14:17:32 client-5 xinetd[3391]: START: shell pid=24989 from=192.168.4.5 May 17 14:17:32 client-5 rshd[24990]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 44a: pdirops: rename tgt vs mkdir ==================================================== 14:17:32 \(1305667052\)");echo XXRETCODE:$?' May 17 14:17:32 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 44a: pdirops: rename tgt vs mkdir ==================================================== 14:17:32 (1305667052) May 17 14:17:32 client-5 xinetd[3391]: EXIT: shell status=0 pid=24989 duration=0(sec) May 17 14:17:42 client-5 xinetd[3391]: START: shell pid=25096 from=192.168.4.5 May 17 14:17:42 client-5 rshd[25097]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:17:42 client-5 xinetd[3391]: EXIT: shell status=0 pid=25096 duration=0(sec) May 17 14:17:42 client-5 xinetd[3391]: START: shell pid=25268 from=192.168.4.5 May 17 14:17:42 client-5 rshd[25269]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 44b: pdirops: rename tgt vs create =================================================== 14:17:42 \(1305667062\)");echo XXRETCODE:$?' May 17 14:17:42 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 44b: pdirops: rename tgt vs create =================================================== 14:17:42 (1305667062) May 17 14:17:42 client-5 xinetd[3391]: EXIT: shell status=0 pid=25268 duration=0(sec) May 17 14:17:52 client-5 xinetd[3391]: START: shell pid=25375 from=192.168.4.5 May 17 14:17:53 client-5 rshd[25376]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:17:53 client-5 xinetd[3391]: EXIT: shell status=0 pid=25375 duration=1(sec) May 17 14:17:53 client-5 xinetd[3391]: START: shell pid=25546 from=192.168.4.5 May 17 14:17:53 client-5 rshd[25548]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 44c: pdirops: rename tgt vs link ===================================================== 14:17:53 \(1305667073\)");echo XXRETCODE:$?' May 17 14:17:53 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 44c: pdirops: rename tgt vs link ===================================================== 14:17:53 (1305667073) May 17 14:17:53 client-5 xinetd[3391]: EXIT: shell status=0 pid=25546 duration=0(sec) May 17 14:18:03 client-5 xinetd[3391]: START: shell pid=25655 from=192.168.4.5 May 17 14:18:03 client-5 rshd[25656]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:18:03 client-5 xinetd[3391]: EXIT: shell status=0 pid=25655 duration=0(sec) May 17 14:18:03 client-5 xinetd[3391]: START: shell pid=25827 from=192.168.4.5 May 17 14:18:03 client-5 rshd[25828]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 44d: pdirops: rename tgt vs unlink =================================================== 14:18:03 \(1305667083\)");echo XXRETCODE:$?' May 17 14:18:04 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 44d: pdirops: rename tgt vs unlink =================================================== 14:18:03 (1305667083) May 17 14:18:04 client-5 xinetd[3391]: EXIT: shell status=0 pid=25827 duration=1(sec) May 17 14:18:14 client-5 xinetd[3391]: START: shell pid=25934 from=192.168.4.5 May 17 14:18:14 client-5 rshd[25935]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:18:14 client-5 xinetd[3391]: EXIT: shell status=0 pid=25934 duration=0(sec) May 17 14:18:14 client-5 xinetd[3391]: START: shell pid=26104 from=192.168.4.5 May 17 14:18:14 client-5 rshd[26107]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 44e: pdirops: rename tgt and rename \(tgt\) ============================================ 14:18:14 \(1305667094\)");echo XXRETCODE:$?' May 17 14:18:14 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 44e: pdirops: rename tgt and rename (tgt) ============================================ 14:18:14 (1305667094) May 17 14:18:14 client-5 xinetd[3391]: EXIT: shell status=0 pid=26104 duration=0(sec) May 17 14:18:24 client-5 xinetd[3391]: START: shell pid=26213 from=192.168.4.5 May 17 14:18:24 client-5 rshd[26216]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:18:24 client-5 xinetd[3391]: EXIT: shell status=0 pid=26213 duration=0(sec) May 17 14:18:25 client-5 xinetd[3391]: START: shell pid=26388 from=192.168.4.5 May 17 14:18:25 client-5 rshd[26389]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 44f: pdirops: rename tgt and rename \(src\) ============================================ 14:18:25 \(1305667105\)");echo XXRETCODE:$?' May 17 14:18:25 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 44f: pdirops: rename tgt and rename (src) ============================================ 14:18:25 (1305667105) May 17 14:18:25 client-5 xinetd[3391]: EXIT: shell status=0 pid=26388 duration=0(sec) May 17 14:18:35 client-5 xinetd[3391]: START: shell pid=26496 from=192.168.4.5 May 17 14:18:35 client-5 rshd[26497]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:18:35 client-5 xinetd[3391]: EXIT: shell status=0 pid=26496 duration=0(sec) May 17 14:18:35 client-5 xinetd[3391]: START: shell pid=26671 from=192.168.4.5 May 17 14:18:35 client-5 rshd[26673]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 44g: pdirops: rename tgt vs getattr ================================================== 14:18:35 \(1305667115\)");echo XXRETCODE:$?' May 17 14:18:35 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 44g: pdirops: rename tgt vs getattr ================================================== 14:18:35 (1305667115) May 17 14:18:35 client-5 xinetd[3391]: EXIT: shell status=0 pid=26671 duration=0(sec) May 17 14:18:46 client-5 xinetd[3391]: START: shell pid=26779 from=192.168.4.5 May 17 14:18:46 client-5 rshd[26780]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:18:46 client-5 xinetd[3391]: EXIT: shell status=0 pid=26779 duration=0(sec) May 17 14:18:46 client-5 xinetd[3391]: START: shell pid=26951 from=192.168.4.5 May 17 14:18:46 client-5 rshd[26952]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 44h: pdirops: rename tgt vs readdir ================================================== 14:18:46 \(1305667126\)");echo XXRETCODE:$?' May 17 14:18:46 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 44h: pdirops: rename tgt vs readdir ================================================== 14:18:46 (1305667126) May 17 14:18:46 client-5 xinetd[3391]: EXIT: shell status=0 pid=26951 duration=0(sec) May 17 14:18:56 client-5 xinetd[3391]: START: shell pid=27058 from=192.168.4.5 May 17 14:18:56 client-5 rshd[27060]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:18:56 client-5 xinetd[3391]: EXIT: shell status=0 pid=27058 duration=0(sec) May 17 14:18:57 client-5 xinetd[3391]: START: shell pid=27231 from=192.168.4.5 May 17 14:18:57 client-5 rshd[27232]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 45a: pdirops: rename src vs mkdir ==================================================== 14:18:57 \(1305667137\)");echo XXRETCODE:$?' May 17 14:18:57 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 45a: pdirops: rename src vs mkdir ==================================================== 14:18:57 (1305667137) May 17 14:18:57 client-5 xinetd[3391]: EXIT: shell status=0 pid=27231 duration=0(sec) May 17 14:19:07 client-5 xinetd[3391]: START: shell pid=27337 from=192.168.4.5 May 17 14:19:07 client-5 rshd[27339]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:19:07 client-5 xinetd[3391]: EXIT: shell status=0 pid=27337 duration=0(sec) May 17 14:19:07 client-5 xinetd[3391]: START: shell pid=27509 from=192.168.4.5 May 17 14:19:07 client-5 rshd[27511]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 45b: pdirops: rename src vs create =================================================== 14:19:07 \(1305667147\)");echo XXRETCODE:$?' May 17 14:19:07 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 45b: pdirops: rename src vs create =================================================== 14:19:07 (1305667147) May 17 14:19:07 client-5 xinetd[3391]: EXIT: shell status=0 pid=27509 duration=0(sec) May 17 14:19:17 client-5 xinetd[3391]: START: shell pid=27616 from=192.168.4.5 May 17 14:19:18 client-5 rshd[27618]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:19:18 client-5 xinetd[3391]: EXIT: shell status=0 pid=27616 duration=1(sec) May 17 14:19:18 client-5 xinetd[3391]: START: shell pid=27789 from=192.168.4.5 May 17 14:19:18 client-5 rshd[27790]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 45c: pdirops: rename src vs link ===================================================== 14:19:18 \(1305667158\)");echo XXRETCODE:$?' May 17 14:19:18 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 45c: pdirops: rename src vs link ===================================================== 14:19:18 (1305667158) May 17 14:19:18 client-5 xinetd[3391]: EXIT: shell status=0 pid=27789 duration=0(sec) May 17 14:19:28 client-5 xinetd[3391]: START: shell pid=27898 from=192.168.4.5 May 17 14:19:28 client-5 rshd[27899]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:19:28 client-5 xinetd[3391]: EXIT: shell status=0 pid=27898 duration=0(sec) May 17 14:19:28 client-5 xinetd[3391]: START: shell pid=28074 from=192.168.4.5 May 17 14:19:28 client-5 rshd[28075]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 45d: pdirops: rename src vs unlink =================================================== 14:19:28 \(1305667168\)");echo XXRETCODE:$?' May 17 14:19:29 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 45d: pdirops: rename src vs unlink =================================================== 14:19:28 (1305667168) May 17 14:19:29 client-5 xinetd[3391]: EXIT: shell status=0 pid=28074 duration=1(sec) May 17 14:19:39 client-5 xinetd[3391]: START: shell pid=28181 from=192.168.4.5 May 17 14:19:39 client-5 rshd[28184]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:19:39 client-5 xinetd[3391]: EXIT: shell status=0 pid=28181 duration=0(sec) May 17 14:19:39 client-5 xinetd[3391]: START: shell pid=28355 from=192.168.4.5 May 17 14:19:39 client-5 rshd[28356]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 45e: pdirops: rename src and rename \(tgt\) ============================================ 14:19:39 \(1305667179\)");echo XXRETCODE:$?' May 17 14:19:39 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 45e: pdirops: rename src and rename (tgt) ============================================ 14:19:39 (1305667179) May 17 14:19:39 client-5 xinetd[3391]: EXIT: shell status=0 pid=28355 duration=0(sec) May 17 14:19:49 client-5 xinetd[3391]: START: shell pid=28463 from=192.168.4.5 May 17 14:19:49 client-5 rshd[28464]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:19:49 client-5 xinetd[3391]: EXIT: shell status=0 pid=28463 duration=0(sec) May 17 14:19:50 client-5 xinetd[3391]: START: shell pid=28635 from=192.168.4.5 May 17 14:19:50 client-5 rshd[28636]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 45f: pdirops: rename src and rename \(src\) ============================================ 14:19:50 \(1305667190\)");echo XXRETCODE:$?' May 17 14:19:50 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 45f: pdirops: rename src and rename (src) ============================================ 14:19:50 (1305667190) May 17 14:19:50 client-5 xinetd[3391]: EXIT: shell status=0 pid=28635 duration=0(sec) May 17 14:20:00 client-5 xinetd[3391]: START: shell pid=28746 from=192.168.4.5 May 17 14:20:00 client-5 rshd[28747]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:20:00 client-5 xinetd[3391]: EXIT: shell status=0 pid=28746 duration=0(sec) May 17 14:20:00 client-5 xinetd[3391]: START: shell pid=28918 from=192.168.4.5 May 17 14:20:00 client-5 rshd[28919]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 45g: pdirops: rename src vs getattr ================================================== 14:20:00 \(1305667200\)");echo XXRETCODE:$?' May 17 14:20:00 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 45g: pdirops: rename src vs getattr ================================================== 14:20:00 (1305667200) May 17 14:20:00 client-5 xinetd[3391]: EXIT: shell status=0 pid=28918 duration=0(sec) May 17 14:20:11 client-5 xinetd[3391]: START: shell pid=29025 from=192.168.4.5 May 17 14:20:11 client-5 rshd[29026]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:20:11 client-5 xinetd[3391]: EXIT: shell status=0 pid=29025 duration=0(sec) May 17 14:20:11 client-5 xinetd[3391]: START: shell pid=29197 from=192.168.4.5 May 17 14:20:11 client-5 rshd[29198]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 45h: pdirops: unlink vs readdir ====================================================== 14:20:11 \(1305667211\)");echo XXRETCODE:$?' May 17 14:20:11 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 45h: pdirops: unlink vs readdir ====================================================== 14:20:11 (1305667211) May 17 14:20:11 client-5 xinetd[3391]: EXIT: shell status=0 pid=29197 duration=0(sec) May 17 14:20:21 client-5 xinetd[3391]: START: shell pid=29305 from=192.168.4.5 May 17 14:20:21 client-5 rshd[29306]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:20:21 client-5 xinetd[3391]: EXIT: shell status=0 pid=29305 duration=0(sec) May 17 14:20:22 client-5 xinetd[3391]: START: shell pid=29477 from=192.168.4.5 May 17 14:20:22 client-5 rshd[29478]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 46a: pdirops: link vs mkdir ========================================================== 14:20:21 \(1305667221\)");echo XXRETCODE:$?' May 17 14:20:22 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 46a: pdirops: link vs mkdir ========================================================== 14:20:21 (1305667221) May 17 14:20:22 client-5 xinetd[3391]: EXIT: shell status=0 pid=29477 duration=0(sec) May 17 14:20:32 client-5 xinetd[3391]: START: shell pid=29584 from=192.168.4.5 May 17 14:20:32 client-5 rshd[29585]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:20:32 client-5 xinetd[3391]: EXIT: shell status=0 pid=29584 duration=0(sec) May 17 14:20:32 client-5 xinetd[3391]: START: shell pid=29756 from=192.168.4.5 May 17 14:20:32 client-5 rshd[29757]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 46b: pdirops: link vs create ========================================================= 14:20:32 \(1305667232\)");echo XXRETCODE:$?' May 17 14:20:32 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 46b: pdirops: link vs create ========================================================= 14:20:32 (1305667232) May 17 14:20:32 client-5 xinetd[3391]: EXIT: shell status=0 pid=29756 duration=0(sec) May 17 14:20:42 client-5 xinetd[3391]: START: shell pid=29862 from=192.168.4.5 May 17 14:20:42 client-5 rshd[29864]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:20:43 client-5 xinetd[3391]: EXIT: shell status=0 pid=29862 duration=1(sec) May 17 14:20:43 client-5 xinetd[3391]: START: shell pid=30035 from=192.168.4.5 May 17 14:20:43 client-5 rshd[30036]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 46c: pdirops: link vs link =========================================================== 14:20:43 \(1305667243\)");echo XXRETCODE:$?' May 17 14:20:43 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 46c: pdirops: link vs link =========================================================== 14:20:43 (1305667243) May 17 14:20:43 client-5 xinetd[3391]: EXIT: shell status=0 pid=30035 duration=0(sec) May 17 14:20:53 client-5 xinetd[3391]: START: shell pid=30141 from=192.168.4.5 May 17 14:20:53 client-5 rshd[30143]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:20:53 client-5 xinetd[3391]: EXIT: shell status=0 pid=30141 duration=0(sec) May 17 14:20:53 client-5 xinetd[3391]: START: shell pid=30314 from=192.168.4.5 May 17 14:20:53 client-5 rshd[30315]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 46d: pdirops: link vs unlink ========================================================= 14:20:53 \(1305667253\)");echo XXRETCODE:$?' May 17 14:20:53 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 46d: pdirops: link vs unlink ========================================================= 14:20:53 (1305667253) May 17 14:20:53 client-5 xinetd[3391]: EXIT: shell status=0 pid=30314 duration=0(sec) May 17 14:21:04 client-5 xinetd[3391]: START: shell pid=30421 from=192.168.4.5 May 17 14:21:04 client-5 rshd[30422]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:21:04 client-5 xinetd[3391]: EXIT: shell status=0 pid=30421 duration=0(sec) May 17 14:21:04 client-5 xinetd[3391]: START: shell pid=30592 from=192.168.4.5 May 17 14:21:04 client-5 rshd[30594]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 46e: pdirops: link and rename \(tgt\) ================================================== 14:21:04 \(1305667264\)");echo XXRETCODE:$?' May 17 14:21:04 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 46e: pdirops: link and rename (tgt) ================================================== 14:21:04 (1305667264) May 17 14:21:04 client-5 xinetd[3391]: EXIT: shell status=0 pid=30592 duration=0(sec) May 17 14:21:14 client-5 xinetd[3391]: START: shell pid=30706 from=192.168.4.5 May 17 14:21:14 client-5 rshd[30707]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:21:14 client-5 xinetd[3391]: EXIT: shell status=0 pid=30706 duration=0(sec) May 17 14:21:15 client-5 xinetd[3391]: START: shell pid=30879 from=192.168.4.5 May 17 14:21:15 client-5 rshd[30880]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 46f: pdirops: link and rename \(src\) ================================================== 14:21:15 \(1305667275\)");echo XXRETCODE:$?' May 17 14:21:15 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 46f: pdirops: link and rename (src) ================================================== 14:21:15 (1305667275) May 17 14:21:15 client-5 xinetd[3391]: EXIT: shell status=0 pid=30879 duration=0(sec) May 17 14:21:25 client-5 xinetd[3391]: START: shell pid=30986 from=192.168.4.5 May 17 14:21:25 client-5 rshd[30988]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:21:25 client-5 xinetd[3391]: EXIT: shell status=0 pid=30986 duration=0(sec) May 17 14:21:25 client-5 xinetd[3391]: START: shell pid=31159 from=192.168.4.5 May 17 14:21:25 client-5 rshd[31160]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 46g: pdirops: link vs getattr ======================================================== 14:21:25 \(1305667285\)");echo XXRETCODE:$?' May 17 14:21:25 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 46g: pdirops: link vs getattr ======================================================== 14:21:25 (1305667285) May 17 14:21:25 client-5 xinetd[3391]: EXIT: shell status=0 pid=31159 duration=0(sec) May 17 14:21:36 client-5 xinetd[3391]: START: shell pid=31266 from=192.168.4.5 May 17 14:21:36 client-5 rshd[31267]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:21:36 client-5 xinetd[3391]: EXIT: shell status=0 pid=31266 duration=0(sec) May 17 14:21:36 client-5 xinetd[3391]: START: shell pid=31438 from=192.168.4.5 May 17 14:21:36 client-5 rshd[31439]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 46h: pdirops: link vs readdir ======================================================== 14:21:36 \(1305667296\)");echo XXRETCODE:$?' May 17 14:21:36 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 46h: pdirops: link vs readdir ======================================================== 14:21:36 (1305667296) May 17 14:21:36 client-5 xinetd[3391]: EXIT: shell status=0 pid=31438 duration=0(sec) May 17 14:21:46 client-5 xinetd[3391]: START: shell pid=31546 from=192.168.4.5 May 17 14:21:46 client-5 rshd[31547]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:21:46 client-5 xinetd[3391]: EXIT: shell status=0 pid=31546 duration=0(sec) May 17 14:21:46 client-5 xinetd[3391]: START: shell pid=31717 from=192.168.4.5 May 17 14:21:47 client-5 rshd[31719]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ================================================ 14:21:46 \(1305667306\)");echo XXRETCODE:$?' May 17 14:21:47 client-5 kernel: Lustre: DEBUG MARKER: == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ================================================ 14:21:46 (1305667306) May 17 14:21:47 client-5 xinetd[3391]: EXIT: shell status=0 pid=31717 duration=1(sec) May 17 14:21:47 client-5 kernel: LustreError: 20197:0:(lvfs_lib.c:150:__obd_fail_timeout_set()) obd_fail_timeout id 410 sleeping for 1000ms May 17 14:21:47 client-5 kernel: LustreError: 20975:0:(lvfs_lib.c:150:__obd_fail_timeout_set()) obd_fail_timeout id 410 sleeping for 2000ms May 17 14:21:47 client-5 kernel: LustreError: 20197:0:(lvfs_lib.c:154:__obd_fail_timeout_set()) obd_fail_timeout id 410 awake May 17 14:21:49 client-5 kernel: LustreError: 20975:0:(lvfs_lib.c:154:__obd_fail_timeout_set()) obd_fail_timeout id 410 awake May 17 14:21:50 client-5 xinetd[3391]: START: shell pid=31825 from=192.168.4.5 May 17 14:21:50 client-5 rshd[31826]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:21:50 client-5 xinetd[3391]: EXIT: shell status=0 pid=31825 duration=0(sec) May 17 14:21:50 client-5 xinetd[3391]: START: shell pid=31987 from=192.168.4.5 May 17 14:21:50 client-5 rshd[31988]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark cleanup: ======================================================");echo XXRETCODE:$?' May 17 14:21:50 client-5 kernel: Lustre: DEBUG MARKER: cleanup: ====================================================== May 17 14:21:50 client-5 xinetd[3391]: EXIT: shell status=0 pid=31987 duration=0(sec) May 17 14:21:51 client-5 kernel: LustreError: 32025:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 14:21:51 client-5 kernel: LustreError: 32025:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 14:21:51 client-5 kernel: LustreError: 32025:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 14:21:51 client-5 kernel: LustreError: 32025:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 14:21:51 client-5 kernel: Lustre: client ffff8102f6091800 umount complete May 17 14:21:51 client-5 xinetd[3391]: START: shell pid=32087 from=192.168.4.5 May 17 14:21:51 client-5 rshd[32088]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanityn sanityn.sh test complete, duration 1135 sec =============================================== 14:21:51 \(1305667311\)");echo XXRETCODE:$?' May 17 14:21:51 client-5 kernel: Lustre: DEBUG MARKER: == sanityn sanityn.sh test complete, duration 1135 sec =============================================== 14:21:51 (1305667311) May 17 14:21:51 client-5 xinetd[3391]: EXIT: shell status=0 pid=32087 duration=0(sec) May 17 14:21:51 client-5 xinetd[3391]: START: shell pid=32135 from=192.168.4.5 May 17 14:21:51 client-5 rshd[32136]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:; lsof -t /mnt/lustre2 || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre2 | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre2 is still busy, wait one second && sleep 1; umount /mnt/lustre2; fi fi");echo XXRETCODE:$?' May 17 14:21:51 client-5 xinetd[3391]: EXIT: shell status=0 pid=32135 duration=0(sec) May 17 14:21:51 client-5 xinetd[3391]: START: shell pid=32239 from=192.168.4.5 May 17 14:21:51 client-5 rshd[32242]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark -----============= acceptance-small: lfsck ============----- Tue May 17 14:21:51 PDT 2011");echo XXRETCODE:$?' May 17 14:21:51 client-5 kernel: Lustre: DEBUG MARKER: -----============= acceptance-small: lfsck ============----- Tue May 17 14:21:51 PDT 2011 May 17 14:21:51 client-5 xinetd[3391]: EXIT: shell status=0 pid=32239 duration=0(sec) May 17 14:21:51 client-5 xinetd[3391]: START: shell pid=32332 from=192.168.4.5 May 17 14:21:51 client-5 rshd[32333]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/tests:/usr/lib64/openmpi/1.4-gcc/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/usr/lib64/lustre/utils:/usr/lib64/lustre/utils/gss:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey::/usr/lib64/lustre/tests/mpi NAME=sarah sh rpc.sh check_config_client /mnt/lustre ");echo XXRETCODE:$?' May 17 14:21:52 client-5 xinetd[3391]: EXIT: shell status=0 pid=32332 duration=1(sec) May 17 14:21:53 client-5 xinetd[3391]: START: shell pid=32532 from=192.168.4.5 May 17 14:21:53 client-5 rshd[32533]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); mpts=\$(mount | grep -w -c /mnt/lustre); if [ \$running -ne \$mpts ]; then echo \$(hostname) env are INSANE!; exit 1; fi");echo XXRETCODE:$?' May 17 14:21:53 client-5 xinetd[3391]: EXIT: shell status=0 pid=32532 duration=0(sec) May 17 14:21:53 client-5 xinetd[3391]: START: shell pid=32578 from=192.168.4.5 May 17 14:21:53 client-5 rshd[32580]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); mpts=\$(mount | grep -w -c /mnt/lustre2); if [ \$running -ne \$mpts ]; then echo \$(hostname) env are INSANE!; exit 1; fi");echo XXRETCODE:$?' May 17 14:21:53 client-5 xinetd[3391]: EXIT: shell status=0 pid=32578 duration=0(sec) May 17 14:21:55 client-5 xinetd[3391]: START: shell pid=483 from=192.168.4.5 May 17 14:21:55 client-5 rshd[484]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark Using TIMEOUT=20");echo XXRETCODE:$?' May 17 14:21:55 client-5 kernel: Lustre: DEBUG MARKER: Using TIMEOUT=20 May 17 14:21:55 client-5 xinetd[3391]: EXIT: shell status=0 pid=483 duration=0(sec) May 17 14:21:55 client-5 kernel: LustreError: 537:0:(quota_ctl.c:328:client_quota_ctl()) ptlrpc_queue_wait failed, rc: -114 May 17 14:21:55 client-5 xinetd[3391]: START: shell pid=595 from=192.168.4.5 May 17 14:21:55 client-5 rshd[596]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param debug=\"0x33f0406\"; lctl set_param subsystem_debug=\"0xffb7e3ff\"; lctl set_param debug_mb=32; sync");echo XXRETCODE:$?' May 17 14:21:55 client-5 kernel: Lustre: 642:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 14:21:55 client-5 kernel: Lustre: 642:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message May 17 14:21:55 client-5 xinetd[3391]: EXIT: shell status=0 pid=595 duration=0(sec) May 17 14:21:55 client-5 xinetd[3391]: START: shell pid=660 from=192.168.4.5 May 17 14:21:55 client-5 rshd[661]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "cat /proc/mounts");echo XXRETCODE:$?' May 17 14:21:55 client-5 xinetd[3391]: EXIT: shell status=0 pid=660 duration=0(sec) May 17 14:21:57 client-5 kernel: FS-Cache: Loaded May 17 14:21:57 client-5 xinetd[3391]: START: shell pid=974 from=192.168.4.5 May 17 14:21:57 client-5 rshd[976]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "sync; sleep 2; sync");echo XXRETCODE:$?' May 17 14:21:59 client-5 xinetd[3391]: EXIT: shell status=0 pid=974 duration=2(sec) May 17 14:21:59 client-5 xinetd[3391]: START: shell pid=1019 from=192.168.4.5 May 17 14:21:59 client-5 rshd[1023]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "ls /scratch/tmp.BAJFvFh912");echo XXRETCODE:$?' May 17 14:21:59 client-5 xinetd[3391]: EXIT: shell status=0 pid=1019 duration=0(sec) May 17 14:22:07 client-5 xinetd[3391]: START: shell pid=1198 from=192.168.4.5 May 17 14:22:07 client-5 rshd[1199]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == lfsck lfsck.sh test complete, duration 16 sec ===================================================== 14:22:07 \(1305667327\)");echo XXRETCODE:$?' May 17 14:22:07 client-5 kernel: Lustre: DEBUG MARKER: == lfsck lfsck.sh test complete, duration 16 sec ===================================================== 14:22:07 (1305667327) May 17 14:22:07 client-5 xinetd[3391]: EXIT: shell status=0 pid=1198 duration=0(sec) May 17 14:22:07 client-5 xinetd[3391]: START: shell pid=1307 from=192.168.4.5 May 17 14:22:07 client-5 rshd[1308]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark -----============= acceptance-small: runracer ============----- Tue May 17 14:22:07 PDT 2011");echo XXRETCODE:$?' May 17 14:22:08 client-5 kernel: Lustre: DEBUG MARKER: -----============= acceptance-small: runracer ============----- Tue May 17 14:22:07 PDT 2011 May 17 14:22:08 client-5 xinetd[3391]: EXIT: shell status=0 pid=1307 duration=1(sec) May 17 14:22:08 client-5 xinetd[3391]: START: shell pid=1442 from=192.168.4.5 May 17 14:22:08 client-5 rshd[1444]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark excepting tests: ");echo XXRETCODE:$?' May 17 14:22:08 client-5 kernel: Lustre: DEBUG MARKER: excepting tests: May 17 14:22:08 client-5 xinetd[3391]: EXIT: shell status=0 pid=1442 duration=0(sec) May 17 14:22:08 client-5 xinetd[3391]: START: shell pid=1494 from=192.168.4.5 May 17 14:22:08 client-5 rshd[1495]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/tests:/usr/lib64/openmpi/1.4-gcc/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/usr/lib64/lustre/utils:/usr/lib64/lustre/utils/gss:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey::/usr/lib64/lustre/tests/mpi NAME=sarah sh rpc.sh check_config_client /mnt/lustre ");echo XXRETCODE:$?' May 17 14:22:08 client-5 xinetd[3391]: EXIT: shell status=0 pid=1494 duration=0(sec) May 17 14:22:09 client-5 xinetd[3391]: START: shell pid=1696 from=192.168.4.5 May 17 14:22:09 client-5 rshd[1697]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); mpts=\$(mount | grep -w -c /mnt/lustre); if [ \$running -ne \$mpts ]; then echo \$(hostname) env are INSANE!; exit 1; fi");echo XXRETCODE:$?' May 17 14:22:09 client-5 xinetd[3391]: EXIT: shell status=0 pid=1696 duration=0(sec) May 17 14:22:09 client-5 xinetd[3391]: START: shell pid=1745 from=192.168.4.5 May 17 14:22:09 client-5 rshd[1746]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); mpts=\$(mount | grep -w -c /mnt/lustre2); if [ \$running -ne \$mpts ]; then echo \$(hostname) env are INSANE!; exit 1; fi");echo XXRETCODE:$?' May 17 14:22:09 client-5 xinetd[3391]: EXIT: shell status=0 pid=1745 duration=0(sec) May 17 14:22:11 client-5 xinetd[3391]: START: shell pid=2113 from=192.168.4.5 May 17 14:22:11 client-5 rshd[2115]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark Using TIMEOUT=20");echo XXRETCODE:$?' May 17 14:22:11 client-5 kernel: Lustre: DEBUG MARKER: Using TIMEOUT=20 May 17 14:22:11 client-5 xinetd[3391]: EXIT: shell status=0 pid=2113 duration=0(sec) May 17 14:22:11 client-5 kernel: LustreError: 2169:0:(quota_ctl.c:328:client_quota_ctl()) ptlrpc_queue_wait failed, rc: -114 May 17 14:22:11 client-5 xinetd[3391]: START: shell pid=2230 from=192.168.4.5 May 17 14:22:11 client-5 rshd[2231]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param debug=\"0x33f0406\"; lctl set_param subsystem_debug=\"0xffb7e3ff\"; lctl set_param debug_mb=32; sync");echo XXRETCODE:$?' May 17 14:22:11 client-5 kernel: Lustre: 2273:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 14:22:11 client-5 kernel: Lustre: 2273:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message May 17 14:22:11 client-5 xinetd[3391]: EXIT: shell status=0 pid=2230 duration=0(sec) May 17 14:22:11 client-5 xinetd[3391]: START: shell pid=2290 from=192.168.4.5 May 17 14:22:11 client-5 rshd[2291]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "cat /proc/mounts");echo XXRETCODE:$?' May 17 14:22:12 client-5 xinetd[3391]: EXIT: shell status=0 pid=2290 duration=1(sec) May 17 14:22:12 client-5 xinetd[3391]: START: shell pid=2577 from=192.168.4.5 May 17 14:22:13 client-5 rshd[2578]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == runracer test 1: racer on clients: client-5-ib,client-18-ib DURATION=120 ========================== 14:22:12 \(1305667332\)");echo XXRETCODE:$?' May 17 14:22:13 client-5 kernel: Lustre: DEBUG MARKER: == runracer test 1: racer on clients: client-5-ib,client-18-ib DURATION=120 ========================== 14:22:12 (1305667332) May 17 14:22:13 client-5 xinetd[3391]: EXIT: shell status=0 pid=2577 duration=1(sec) May 17 14:22:13 client-5 xinetd[3391]: START: shell pid=2622 from=192.168.4.5 May 17 14:22:13 client-5 rshd[2623]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "DURATION=120 /usr/lib64/lustre/tests/racer/racer.sh /mnt/lustre/racer ");echo XXRETCODE:$?' May 17 14:23:29 client-5 kernel: LustreError: 5984:0:(file.c:2161:ll_inode_revalidate_fini()) failure -2 inode 144115305935871905 May 17 14:24:14 client-5 xinetd[3391]: EXIT: shell status=0 pid=2622 duration=121(sec) May 17 14:24:14 client-5 xinetd[3391]: START: shell pid=30671 from=192.168.4.5 May 17 14:24:14 client-5 rshd[30672]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "ps uax | grep /mnt/lustre/racer ");echo XXRETCODE:$?' May 17 14:24:14 client-5 xinetd[3391]: EXIT: shell status=0 pid=30671 duration=0(sec) May 17 14:24:24 client-5 xinetd[3391]: START: shell pid=30719 from=192.168.4.5 May 17 14:24:24 client-5 rshd[30720]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "ps uax | grep /mnt/lustre/racer ");echo XXRETCODE:$?' May 17 14:24:24 client-5 xinetd[3391]: EXIT: shell status=0 pid=30719 duration=0(sec) May 17 14:24:44 client-5 xinetd[3391]: START: shell pid=30767 from=192.168.4.5 May 17 14:24:44 client-5 rshd[30768]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "ps uax | grep /mnt/lustre/racer ");echo XXRETCODE:$?' May 17 14:24:44 client-5 xinetd[3391]: EXIT: shell status=0 pid=30767 duration=0(sec) May 17 14:25:24 client-5 xinetd[3391]: START: shell pid=30819 from=192.168.4.5 May 17 14:25:24 client-5 rshd[30820]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "ps uax | grep /mnt/lustre/racer ");echo XXRETCODE:$?' May 17 14:25:24 client-5 xinetd[3391]: EXIT: shell status=0 pid=30819 duration=0(sec) May 17 14:26:04 client-5 xinetd[3391]: START: shell pid=30867 from=192.168.4.5 May 17 14:26:04 client-5 rshd[30868]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "ps uax | grep /mnt/lustre/racer ");echo XXRETCODE:$?' May 17 14:26:04 client-5 xinetd[3391]: EXIT: shell status=0 pid=30867 duration=0(sec) May 17 14:26:44 client-5 xinetd[3391]: START: shell pid=30919 from=192.168.4.5 May 17 14:26:45 client-5 rshd[30920]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "ps uax | grep /mnt/lustre/racer ");echo XXRETCODE:$?' May 17 14:26:45 client-5 xinetd[3391]: EXIT: shell status=0 pid=30919 duration=1(sec) May 17 14:27:25 client-5 xinetd[3391]: START: shell pid=30974 from=192.168.4.5 May 17 14:27:25 client-5 rshd[30975]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "ps uax | grep /mnt/lustre/racer ");echo XXRETCODE:$?' May 17 14:27:25 client-5 xinetd[3391]: EXIT: shell status=0 pid=30974 duration=0(sec) May 17 14:28:05 client-5 xinetd[3391]: START: shell pid=31022 from=192.168.4.5 May 17 14:28:05 client-5 rshd[31023]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "ps uax | grep /mnt/lustre/racer ");echo XXRETCODE:$?' May 17 14:28:05 client-5 xinetd[3391]: EXIT: shell status=0 pid=31022 duration=0(sec) May 17 14:28:45 client-5 xinetd[3391]: START: shell pid=31074 from=192.168.4.5 May 17 14:28:45 client-5 rshd[31075]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "ps uax | grep /mnt/lustre/racer ");echo XXRETCODE:$?' May 17 14:28:45 client-5 xinetd[3391]: EXIT: shell status=0 pid=31074 duration=0(sec) May 17 14:29:25 client-5 xinetd[3391]: START: shell pid=31122 from=192.168.4.5 May 17 14:29:25 client-5 rshd[31123]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "ps uax | grep /mnt/lustre/racer ");echo XXRETCODE:$?' May 17 14:29:25 client-5 xinetd[3391]: EXIT: shell status=0 pid=31122 duration=0(sec) May 17 14:30:05 client-5 xinetd[3391]: START: shell pid=31173 from=192.168.4.5 May 17 14:30:05 client-5 rshd[31175]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "ps uax | grep /mnt/lustre/racer ");echo XXRETCODE:$?' May 17 14:30:05 client-5 xinetd[3391]: EXIT: shell status=0 pid=31173 duration=0(sec) May 17 14:30:45 client-5 xinetd[3391]: START: shell pid=31222 from=192.168.4.5 May 17 14:30:45 client-5 rshd[31223]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "ps uax | grep /mnt/lustre/racer ");echo XXRETCODE:$?' May 17 14:30:45 client-5 xinetd[3391]: EXIT: shell status=0 pid=31222 duration=0(sec) May 17 14:31:25 client-5 xinetd[3391]: START: shell pid=31274 from=192.168.4.5 May 17 14:31:25 client-5 rshd[31275]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "ps uax | grep /mnt/lustre/racer ");echo XXRETCODE:$?' May 17 14:31:25 client-5 xinetd[3391]: EXIT: shell status=0 pid=31274 duration=0(sec) May 17 14:32:05 client-5 xinetd[3391]: START: shell pid=31322 from=192.168.4.5 May 17 14:32:05 client-5 rshd[31323]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "ps uax | grep /mnt/lustre/racer ");echo XXRETCODE:$?' May 17 14:32:06 client-5 xinetd[3391]: EXIT: shell status=0 pid=31322 duration=1(sec) May 17 14:32:46 client-5 xinetd[3391]: START: shell pid=31374 from=192.168.4.5 May 17 14:32:46 client-5 rshd[31375]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "ps uax | grep /mnt/lustre/racer ");echo XXRETCODE:$?' May 17 14:32:46 client-5 xinetd[3391]: EXIT: shell status=0 pid=31374 duration=0(sec) May 17 14:33:26 client-5 xinetd[3391]: START: shell pid=31422 from=192.168.4.5 May 17 14:33:26 client-5 rshd[31423]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "ps uax | grep /mnt/lustre/racer ");echo XXRETCODE:$?' May 17 14:33:26 client-5 xinetd[3391]: EXIT: shell status=0 pid=31422 duration=0(sec) May 17 14:34:06 client-5 xinetd[3391]: START: shell pid=31474 from=192.168.4.5 May 17 14:34:06 client-5 rshd[31475]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "ps uax | grep /mnt/lustre/racer ");echo XXRETCODE:$?' May 17 14:34:06 client-5 xinetd[3391]: EXIT: shell status=0 pid=31474 duration=0(sec) May 17 14:35:06 client-5 xinetd[3391]: START: shell pid=31608 from=192.168.4.5 May 17 14:35:06 client-5 rshd[31609]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:35:06 client-5 xinetd[3391]: EXIT: shell status=0 pid=31608 duration=0(sec) May 17 14:35:07 client-5 xinetd[3391]: START: shell pid=31773 from=192.168.4.5 May 17 14:35:07 client-5 rshd[31774]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == runracer runracer test complete, duration 778 sec ================================================= 14:35:06 \(1305668106\)");echo XXRETCODE:$?' May 17 14:35:07 client-5 kernel: Lustre: DEBUG MARKER: == runracer runracer test complete, duration 778 sec ================================================= 14:35:06 (1305668106) May 17 14:35:07 client-5 xinetd[3391]: EXIT: shell status=0 pid=31773 duration=0(sec) May 17 14:35:07 client-5 xinetd[3391]: START: shell pid=31883 from=192.168.4.5 May 17 14:35:07 client-5 rshd[31884]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark -----============= acceptance-small: replay-single ============----- Tue May 17 14:35:07 PDT 2011");echo XXRETCODE:$?' May 17 14:35:07 client-5 kernel: Lustre: DEBUG MARKER: -----============= acceptance-small: replay-single ============----- Tue May 17 14:35:07 PDT 2011 May 17 14:35:07 client-5 xinetd[3391]: EXIT: shell status=0 pid=31883 duration=0(sec) May 17 14:35:07 client-5 xinetd[3391]: START: shell pid=32024 from=192.168.4.5 May 17 14:35:07 client-5 rshd[32027]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark excepting tests: 61d 33a 33b");echo XXRETCODE:$?' May 17 14:35:07 client-5 kernel: Lustre: DEBUG MARKER: excepting tests: 61d 33a 33b May 17 14:35:07 client-5 xinetd[3391]: EXIT: shell status=0 pid=32024 duration=0(sec) May 17 14:35:07 client-5 xinetd[3391]: START: shell pid=32077 from=192.168.4.5 May 17 14:35:07 client-5 rshd[32078]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/tests:/usr/lib64/openmpi/1.4-gcc/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/usr/lib64/lustre/utils:/usr/lib64/lustre/utils/gss:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey::/usr/lib64/lustre/tests/mpi NAME=sarah sh rpc.sh check_config_client /mnt/lustre ");echo XXRETCODE:$?' May 17 14:35:07 client-5 xinetd[3391]: EXIT: shell status=0 pid=32077 duration=0(sec) May 17 14:35:08 client-5 xinetd[3391]: START: shell pid=32277 from=192.168.4.5 May 17 14:35:08 client-5 rshd[32278]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); mpts=\$(mount | grep -w -c /mnt/lustre); if [ \$running -ne \$mpts ]; then echo \$(hostname) env are INSANE!; exit 1; fi");echo XXRETCODE:$?' May 17 14:35:08 client-5 xinetd[3391]: EXIT: shell status=0 pid=32277 duration=0(sec) May 17 14:35:08 client-5 xinetd[3391]: START: shell pid=32324 from=192.168.4.5 May 17 14:35:08 client-5 rshd[32325]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); mpts=\$(mount | grep -w -c /mnt/lustre2); if [ \$running -ne \$mpts ]; then echo \$(hostname) env are INSANE!; exit 1; fi");echo XXRETCODE:$?' May 17 14:35:08 client-5 xinetd[3391]: EXIT: shell status=0 pid=32324 duration=0(sec) May 17 14:35:10 client-5 xinetd[3391]: START: shell pid=32686 from=192.168.4.5 May 17 14:35:10 client-5 rshd[32687]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark Using TIMEOUT=20");echo XXRETCODE:$?' May 17 14:35:10 client-5 kernel: Lustre: DEBUG MARKER: Using TIMEOUT=20 May 17 14:35:10 client-5 xinetd[3391]: EXIT: shell status=0 pid=32686 duration=0(sec) May 17 14:35:11 client-5 kernel: LustreError: 32740:0:(quota_ctl.c:328:client_quota_ctl()) ptlrpc_queue_wait failed, rc: -114 May 17 14:35:11 client-5 xinetd[3391]: START: shell pid=331 from=192.168.4.5 May 17 14:35:11 client-5 rshd[332]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param debug=\"0x33f0406\"; lctl set_param subsystem_debug=\"0xffb7e3ff\"; lctl set_param debug_mb=32; sync");echo XXRETCODE:$?' May 17 14:35:11 client-5 kernel: Lustre: 366:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 14:35:11 client-5 kernel: Lustre: 366:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message May 17 14:35:11 client-5 xinetd[3391]: EXIT: shell status=0 pid=331 duration=0(sec) May 17 14:35:11 client-5 xinetd[3391]: START: shell pid=383 from=192.168.4.5 May 17 14:35:11 client-5 rshd[384]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "cat /proc/mounts");echo XXRETCODE:$?' May 17 14:35:11 client-5 xinetd[3391]: EXIT: shell status=0 pid=383 duration=0(sec) May 17 14:35:12 client-5 xinetd[3391]: START: shell pid=678 from=192.168.4.5 May 17 14:35:12 client-5 rshd[679]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 0a: empty replay =============================================================== 14:35:12 \(1305668112\)");echo XXRETCODE:$?' May 17 14:35:12 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 0a: empty replay =============================================================== 14:35:12 (1305668112) May 17 14:35:12 client-5 xinetd[3391]: EXIT: shell status=0 pid=678 duration=0(sec) May 17 14:35:22 client-5 xinetd[3391]: START: shell pid=732 from=192.168.4.5 May 17 14:35:22 client-5 rshd[733]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 14:35:22 client-5 xinetd[3391]: EXIT: shell status=0 pid=732 duration=0(sec) May 17 14:35:23 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 14:35:34 client-5 kernel: Lustre: 20197:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369045627419698 sent from MGC192.168.4.128@o2ib to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305668127] [real_sent 1305668127] [current 1305668134] [deadline 7s] [delay 0s] req@ffff81033a56ac00 x1369045627419698/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 192/192 e 0 to 1 dl 1305668134 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 14:35:34 client-5 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail. May 17 14:35:39 client-5 kernel: Lustre: 20197:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369045627419700 sent from lustre-MDT0000-mdc-ffff810321d30800 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305668132] [real_sent 1305668132] [current 1305668139] [deadline 7s] [delay 0s] req@ffff8102f640d400 x1369045627419700/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 192/192 e 0 to 1 dl 1305668139 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 14:35:39 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff810321d30800: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 14:35:40 client-5 kernel: Lustre: 20198:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369045627419707 sent from MGC192.168.4.128@o2ib to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305668134] [real_sent 1305668134] [current 1305668140] [deadline 6s] [delay 0s] req@ffff8101682a7000 x1369045627419707/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 368/392 e 0 to 1 dl 1305668140 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 14:35:41 client-5 xinetd[3391]: START: shell pid=914 from=192.168.4.5 May 17 14:35:42 client-5 rshd[915]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 14:35:44 client-5 kernel: Lustre: 20199:0:(import.c:529:import_select_connection()) MGC192.168.4.128@o2ib: tried all connections, increasing latency to 6s May 17 14:35:44 client-5 kernel: Lustre: 20198:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d6cda9907 to 0x62e5a8d74c86adf May 17 14:35:44 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 14:35:44 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Connection restored to service MGS using nid 192.168.4.128@o2ib. May 17 14:35:45 client-5 kernel: Lustre: 20198:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369045627419715 sent from lustre-MDT0000-mdc-ffff810321d30800 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305668139] [real_sent 1305668139] [current 1305668145] [deadline 6s] [delay 0s] req@ffff810317561800 x1369045627419715/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 368/392 e 0 to 1 dl 1305668145 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 14:35:49 client-5 kernel: Lustre: 20199:0:(import.c:529:import_select_connection()) lustre-MDT0000-mdc-ffff810321d30800: tried all connections, increasing latency to 6s May 17 14:35:49 client-5 kernel: LustreError: 20198:0:(mdc_request.c:642:mdc_replay_open()) @@@ Can't properly replay without open data. req@ffff8102c493cc00 x1369045626928997/t17185302343(17185302343) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 528/544 e 0 to 0 dl 1305668161 ref 2 fl Interpret:RP/ffffffff/ffffffff rc 301/-1 May 17 14:35:49 client-5 kernel: LustreError: 20198:0:(mdc_request.c:642:mdc_replay_open()) @@@ Can't properly replay without open data. req@ffff8103165b3c00 x1369045626947546/t17185319191(17185319191) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 528/544 e 0 to 0 dl 1305668161 ref 2 fl Interpret:RP/ffffffff/ffffffff rc 301/-1 May 17 14:35:49 client-5 kernel: LustreError: 20198:0:(client.c:2570:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff810250339800 x1369045627324300/t17185712083(17185712083) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 544/544 e 0 to 0 dl 1305668161 ref 2 fl Interpret:RP/ffffffff/ffffffff rc 301/-1 May 17 14:35:49 client-5 kernel: LustreError: 20198:0:(client.c:2570:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff810317bb9800 x1369045627350643/t17185745189(17185745189) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 544/544 e 0 to 0 dl 1305668161 ref 2 fl Interpret:RP/ffffffff/ffffffff rc 301/-1 May 17 14:35:49 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff810321d30800: Connection restored to service lustre-MDT0000 using nid 192.168.4.128@o2ib. May 17 14:35:49 client-5 xinetd[3391]: EXIT: shell status=0 pid=914 duration=8(sec) May 17 14:35:49 client-5 xinetd[3391]: START: shell pid=1010 from=192.168.4.5 May 17 14:35:49 client-5 rshd[1011]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:35:49 client-5 xinetd[3391]: EXIT: shell status=0 pid=1010 duration=0(sec) May 17 14:35:50 client-5 xinetd[3391]: START: shell pid=1180 from=192.168.4.5 May 17 14:35:50 client-5 rshd[1183]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 0b: ensure object created after recover exists. \(3284\) ========================= 14:35:50 \(1305668150\)");echo XXRETCODE:$?' May 17 14:35:50 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 0b: ensure object created after recover exists. (3284) ========================= 14:35:50 (1305668150) May 17 14:35:50 client-5 xinetd[3391]: EXIT: shell status=0 pid=1180 duration=0(sec) May 17 14:35:59 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.129@o2ib. The obd_ping operation failed with -107 May 17 14:35:59 client-5 kernel: Lustre: lustre-OST0000-osc-ffff810321d30800: Connection to service lustre-OST0000 via nid 192.168.4.129@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 14:36:04 client-5 xinetd[3391]: START: shell pid=1360 from=192.168.4.5 May 17 14:36:04 client-5 rshd[1361]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 14:36:09 client-5 kernel: Lustre: 20199:0:(import.c:529:import_select_connection()) lustre-OST0000-osc-ffff810321d30800: tried all connections, increasing latency to 6s May 17 14:36:09 client-5 kernel: Lustre: lustre-OST0000-osc-ffff810321d30800: Connection restored to service lustre-OST0000 using nid 192.168.4.129@o2ib. May 17 14:36:09 client-5 xinetd[3391]: EXIT: shell status=0 pid=1360 duration=5(sec) May 17 14:36:09 client-5 xinetd[3391]: START: shell pid=1456 from=192.168.4.5 May 17 14:36:09 client-5 rshd[1457]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:36:09 client-5 xinetd[3391]: EXIT: shell status=0 pid=1456 duration=0(sec) May 17 14:36:09 client-5 xinetd[3391]: START: shell pid=1629 from=192.168.4.5 May 17 14:36:09 client-5 rshd[1630]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 0d: expired recovery with no clients =========================================== 14:36:09 \(1305668169\)");echo XXRETCODE:$?' May 17 14:36:09 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 0d: expired recovery with no clients =========================================== 14:36:09 (1305668169) May 17 14:36:09 client-5 xinetd[3391]: EXIT: shell status=0 pid=1629 duration=0(sec) May 17 14:36:09 client-5 xinetd[3391]: START: shell pid=1681 from=192.168.4.5 May 17 14:36:10 client-5 rshd[1682]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 14:36:10 client-5 xinetd[3391]: EXIT: shell status=0 pid=1681 duration=1(sec) May 17 14:36:10 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 14:36:11 client-5 kernel: LustreError: 1747:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 14:36:11 client-5 kernel: LustreError: 1747:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Skipped 1 previous similar message May 17 14:36:11 client-5 kernel: LustreError: 1747:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 14:36:11 client-5 kernel: LustreError: 1747:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) Skipped 1 previous similar message May 17 14:36:11 client-5 kernel: Lustre: client ffff810321d30800 umount complete May 17 14:36:29 client-5 kernel: Lustre: 1869:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 14:36:29 client-5 kernel: Lustre: 1869:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 6 previous similar messages May 17 14:36:29 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 14:36:29 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 14:36:29 client-5 kernel: Lustre: Increasing default stripe size to min 1048576 May 17 14:36:29 client-5 kernel: Lustre: 1869:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff8102c72f6400->192.168.4.128@o2ib netid 50000: select flavor null May 17 14:36:29 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -16 May 17 14:36:29 client-5 kernel: LustreError: Skipped 1 previous similar message May 17 14:36:34 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -16 May 17 14:36:59 client-5 last message repeated 4 times May 17 14:36:59 client-5 kernel: LustreError: Skipped 1 previous similar message May 17 14:37:19 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -16 May 17 14:37:19 client-5 kernel: LustreError: Skipped 3 previous similar messages May 17 14:37:39 client-5 kernel: Lustre: 1869:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-OST0000-osc-ffff8102c72f6400->192.168.4.129@o2ib netid 50000: select flavor null May 17 14:37:39 client-5 kernel: Lustre: Client lustre-client has started May 17 14:37:39 client-5 kernel: Lustre: 1895:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 14:37:39 client-5 kernel: Lustre: 1895:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message May 17 14:37:40 client-5 xinetd[3391]: START: shell pid=1956 from=192.168.4.5 May 17 14:37:40 client-5 rshd[1957]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:37:40 client-5 xinetd[3391]: EXIT: shell status=0 pid=1956 duration=0(sec) May 17 14:37:40 client-5 xinetd[3391]: START: shell pid=2130 from=192.168.4.5 May 17 14:37:40 client-5 rshd[2131]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 1: simple create =============================================================== 14:37:40 \(1305668260\)");echo XXRETCODE:$?' May 17 14:37:40 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 1: simple create =============================================================== 14:37:40 (1305668260) May 17 14:37:40 client-5 xinetd[3391]: EXIT: shell status=0 pid=2130 duration=0(sec) May 17 14:37:40 client-5 xinetd[3391]: START: shell pid=2183 from=192.168.4.5 May 17 14:37:40 client-5 rshd[2184]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 14:37:40 client-5 xinetd[3391]: EXIT: shell status=0 pid=2183 duration=0(sec) May 17 14:37:41 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 14:37:51 client-5 kernel: Lustre: 20197:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369045627420149 sent from MGC192.168.4.128@o2ib to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305668264] [real_sent 1305668264] [current 1305668271] [deadline 7s] [delay 0s] req@ffff81031ff89000 x1369045627420149/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 192/192 e 0 to 1 dl 1305668271 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 14:37:51 client-5 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail. May 17 14:37:57 client-5 kernel: Lustre: 20198:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369045627420158 sent from MGC192.168.4.128@o2ib to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305668271] [real_sent 1305668271] [current 1305668277] [deadline 6s] [delay 0s] req@ffff810319d9a800 x1369045627420158/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 368/392 e 0 to 1 dl 1305668277 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 14:38:00 client-5 xinetd[3391]: START: shell pid=2377 from=192.168.4.5 May 17 14:38:00 client-5 rshd[2378]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 14:38:00 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_statfs operation failed with -107 May 17 14:38:00 client-5 kernel: LustreError: Skipped 3 previous similar messages May 17 14:38:00 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff8102c72f6400: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 14:38:00 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff8102c72f6400: Connection restored to service lustre-MDT0000 using nid 192.168.4.128@o2ib. May 17 14:38:00 client-5 xinetd[3391]: EXIT: shell status=0 pid=2377 duration=0(sec) May 17 14:38:00 client-5 xinetd[3391]: START: shell pid=2481 from=192.168.4.5 May 17 14:38:00 client-5 rshd[2482]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:38:00 client-5 xinetd[3391]: EXIT: shell status=0 pid=2481 duration=0(sec) May 17 14:38:00 client-5 xinetd[3391]: START: shell pid=2656 from=192.168.4.5 May 17 14:38:00 client-5 rshd[2659]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 2a: touch ====================================================================== 14:38:00 \(1305668280\)");echo XXRETCODE:$?' May 17 14:38:00 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 2a: touch ====================================================================== 14:38:00 (1305668280) May 17 14:38:00 client-5 xinetd[3391]: EXIT: shell status=0 pid=2656 duration=0(sec) May 17 14:38:00 client-5 xinetd[3391]: START: shell pid=2709 from=192.168.4.5 May 17 14:38:01 client-5 rshd[2710]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 14:38:01 client-5 xinetd[3391]: EXIT: shell status=0 pid=2709 duration=1(sec) May 17 14:38:01 client-5 kernel: Lustre: 20199:0:(import.c:529:import_select_connection()) MGC192.168.4.128@o2ib: tried all connections, increasing latency to 6s May 17 14:38:01 client-5 kernel: Lustre: 20198:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74c878df to 0x62e5a8d74c881c9 May 17 14:38:01 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 14:38:01 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 14:38:15 client-5 kernel: Lustre: 20197:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369045627420151 sent from lustre-MDT0000-mdc-ffff8102c72f6400 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305668269] [real_sent 1305668269] [current 1305668295] [deadline 26s] [delay 0s] req@ffff81031ecba400 x1369045627420151/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 192/192 e 0 to 1 dl 1305668295 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 14:38:17 client-5 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail. May 17 14:38:17 client-5 kernel: LustreError: 2765:0:(client.c:1057:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@ffff810329f38400 x1369045627420211/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:/ffffffff/ffffffff rc 0/-1 May 17 14:38:17 client-5 kernel: LustreError: 2765:0:(client.c:1057:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@ffff810329f38400 x1369045627420213/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:/ffffffff/ffffffff rc 0/-1 May 17 14:38:20 client-5 xinetd[3391]: START: shell pid=2894 from=192.168.4.5 May 17 14:38:20 client-5 rshd[2895]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 14:38:20 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff8102c72f6400: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 14:38:20 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff8102c72f6400: Connection restored to service lustre-MDT0000 using nid 192.168.4.128@o2ib. May 17 14:38:20 client-5 kernel: Lustre: Skipped 1 previous similar message May 17 14:38:20 client-5 xinetd[3391]: EXIT: shell status=0 pid=2894 duration=0(sec) May 17 14:38:20 client-5 xinetd[3391]: START: shell pid=2986 from=192.168.4.5 May 17 14:38:20 client-5 rshd[2987]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:38:20 client-5 xinetd[3391]: EXIT: shell status=0 pid=2986 duration=0(sec) May 17 14:38:21 client-5 xinetd[3391]: START: shell pid=3177 from=192.168.4.5 May 17 14:38:21 client-5 rshd[3178]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 2b: touch ====================================================================== 14:38:20 \(1305668300\)");echo XXRETCODE:$?' May 17 14:38:21 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 2b: touch ====================================================================== 14:38:20 (1305668300) May 17 14:38:21 client-5 xinetd[3391]: EXIT: shell status=0 pid=3177 duration=0(sec) May 17 14:38:21 client-5 xinetd[3391]: START: shell pid=3232 from=192.168.4.5 May 17 14:38:21 client-5 rshd[3233]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 14:38:21 client-5 xinetd[3391]: EXIT: shell status=0 pid=3232 duration=0(sec) May 17 14:38:21 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 14:38:22 client-5 kernel: LustreError: 2765:0:(client.c:1057:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@ffff8102f8f1dc00 x1369045627420238/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:/ffffffff/ffffffff rc 0/-1 May 17 14:38:26 client-5 kernel: LustreError: 2765:0:(client.c:1057:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@ffff8102f8f1dc00 x1369045627420241/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:/ffffffff/ffffffff rc 0/-1 May 17 14:38:26 client-5 kernel: LustreError: 2765:0:(client.c:1057:ptlrpc_import_delay_req()) Skipped 2 previous similar messages May 17 14:38:30 client-5 kernel: LustreError: 2765:0:(client.c:1057:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@ffff8103260b1000 x1369045627420251/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:/ffffffff/ffffffff rc 0/-1 May 17 14:38:30 client-5 kernel: LustreError: 2765:0:(client.c:1057:ptlrpc_import_delay_req()) Skipped 2 previous similar messages May 17 14:38:32 client-5 kernel: Lustre: 20199:0:(import.c:529:import_select_connection()) MGC192.168.4.128@o2ib: tried all connections, increasing latency to 11s May 17 14:38:35 client-5 kernel: LustreError: 2765:0:(client.c:1057:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@ffff810329f38400 x1369045627420262/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:/ffffffff/ffffffff rc 0/-1 May 17 14:38:35 client-5 kernel: LustreError: 2765:0:(client.c:1057:ptlrpc_import_delay_req()) Skipped 2 previous similar messages May 17 14:38:37 client-5 kernel: Lustre: 20197:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369045627420196 sent from lustre-MDT0000-mdc-ffff8102c72f6400 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305668291] [real_sent 1305668291] [current 1305668317] [deadline 26s] [delay 0s] req@ffff810324eed000 x1369045627420196/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 192/192 e 0 to 1 dl 1305668317 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 14:38:37 client-5 kernel: Lustre: 20197:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 3 previous similar messages May 17 14:38:40 client-5 xinetd[3391]: START: shell pid=3443 from=192.168.4.5 May 17 14:38:40 client-5 rshd[3445]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 14:38:40 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff8102c72f6400: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 14:38:40 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff8102c72f6400: Connection restored to service lustre-MDT0000 using nid 192.168.4.128@o2ib. May 17 14:38:40 client-5 xinetd[3391]: EXIT: shell status=0 pid=3443 duration=0(sec) May 17 14:38:41 client-5 xinetd[3391]: START: shell pid=3547 from=192.168.4.5 May 17 14:38:41 client-5 rshd[3548]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:38:41 client-5 xinetd[3391]: EXIT: shell status=0 pid=3547 duration=0(sec) May 17 14:38:41 client-5 xinetd[3391]: START: shell pid=3725 from=192.168.4.5 May 17 14:38:41 client-5 rshd[3726]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 3a: replay failed open\(O_DIRECTORY\) ============================================ 14:38:41 \(1305668321\)");echo XXRETCODE:$?' May 17 14:38:41 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 3a: replay failed open(O_DIRECTORY) ============================================ 14:38:41 (1305668321) May 17 14:38:41 client-5 xinetd[3391]: EXIT: shell status=0 pid=3725 duration=0(sec) May 17 14:38:41 client-5 xinetd[3391]: START: shell pid=3776 from=192.168.4.5 May 17 14:38:41 client-5 rshd[3777]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 14:38:41 client-5 xinetd[3391]: EXIT: shell status=0 pid=3776 duration=0(sec) May 17 14:38:42 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 14:38:45 client-5 kernel: LustreError: 2765:0:(client.c:1057:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@ffff810325ecd000 x1369045627420290/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:/ffffffff/ffffffff rc 0/-1 May 17 14:38:45 client-5 kernel: LustreError: 2765:0:(client.c:1057:ptlrpc_import_delay_req()) Skipped 5 previous similar messages May 17 14:38:52 client-5 kernel: Lustre: 20199:0:(import.c:529:import_select_connection()) MGC192.168.4.128@o2ib: tried all connections, increasing latency to 16s May 17 14:39:00 client-5 xinetd[3391]: START: shell pid=3960 from=192.168.4.5 May 17 14:39:00 client-5 rshd[3961]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 14:39:01 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff8102c72f6400: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 14:39:01 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff8102c72f6400: Connection restored to service lustre-MDT0000 using nid 192.168.4.128@o2ib. May 17 14:39:01 client-5 xinetd[3391]: EXIT: shell status=0 pid=3960 duration=1(sec) May 17 14:39:01 client-5 kernel: LustreError: 2765:0:(client.c:1057:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@ffff81031095ec00 x1369045627420329/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:/ffffffff/ffffffff rc 0/-1 May 17 14:39:01 client-5 kernel: LustreError: 2765:0:(client.c:1057:ptlrpc_import_delay_req()) Skipped 11 previous similar messages May 17 14:39:01 client-5 xinetd[3391]: START: shell pid=4052 from=192.168.4.5 May 17 14:39:01 client-5 rshd[4053]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:39:01 client-5 xinetd[3391]: EXIT: shell status=0 pid=4052 duration=0(sec) May 17 14:39:01 client-5 xinetd[3391]: START: shell pid=4223 from=192.168.4.5 May 17 14:39:01 client-5 rshd[4225]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 3b: replay failed open -ENOMEM ================================================= 14:39:01 \(1305668341\)");echo XXRETCODE:$?' May 17 14:39:01 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 3b: replay failed open -ENOMEM ================================================= 14:39:01 (1305668341) May 17 14:39:01 client-5 xinetd[3391]: EXIT: shell status=0 pid=4223 duration=0(sec) May 17 14:39:01 client-5 xinetd[3391]: START: shell pid=4275 from=192.168.4.5 May 17 14:39:01 client-5 rshd[4276]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 14:39:02 client-5 xinetd[3391]: EXIT: shell status=0 pid=4275 duration=1(sec) May 17 14:39:02 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 14:39:13 client-5 kernel: Lustre: 20198:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369045627420303 sent from MGC192.168.4.128@o2ib to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305668332] [real_sent 1305668332] [current 1305668353] [deadline 21s] [delay 0s] req@ffff810329f38c00 x1369045627420303/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 368/392 e 0 to 1 dl 1305668353 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 14:39:13 client-5 kernel: Lustre: 20198:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 4 previous similar messages May 17 14:39:17 client-5 kernel: Lustre: 20199:0:(import.c:529:import_select_connection()) MGC192.168.4.128@o2ib: tried all connections, increasing latency to 21s May 17 14:39:21 client-5 xinetd[3391]: START: shell pid=4474 from=192.168.4.5 May 17 14:39:21 client-5 rshd[4475]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 14:39:21 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_statfs operation failed with -107 May 17 14:39:21 client-5 kernel: LustreError: Skipped 3 previous similar messages May 17 14:39:21 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff8102c72f6400: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 14:39:21 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff8102c72f6400: Connection restored to service lustre-MDT0000 using nid 192.168.4.128@o2ib. May 17 14:39:21 client-5 xinetd[3391]: EXIT: shell status=0 pid=4474 duration=0(sec) May 17 14:39:21 client-5 xinetd[3391]: START: shell pid=4564 from=192.168.4.5 May 17 14:39:21 client-5 rshd[4566]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:39:21 client-5 xinetd[3391]: EXIT: shell status=0 pid=4564 duration=0(sec) May 17 14:39:22 client-5 xinetd[3391]: START: shell pid=4737 from=192.168.4.5 May 17 14:39:22 client-5 rshd[4738]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 3c: replay failed open -ENOMEM ================================================= 14:39:22 \(1305668362\)");echo XXRETCODE:$?' May 17 14:39:22 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 3c: replay failed open -ENOMEM ================================================= 14:39:22 (1305668362) May 17 14:39:22 client-5 xinetd[3391]: EXIT: shell status=0 pid=4737 duration=0(sec) May 17 14:39:22 client-5 xinetd[3391]: START: shell pid=4788 from=192.168.4.5 May 17 14:39:22 client-5 rshd[4789]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 14:39:22 client-5 xinetd[3391]: EXIT: shell status=0 pid=4788 duration=0(sec) May 17 14:39:22 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 14:39:34 client-5 kernel: LustreError: 2765:0:(client.c:1057:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@ffff8103051cd800 x1369045627420413/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:/ffffffff/ffffffff rc 0/-1 May 17 14:39:34 client-5 kernel: LustreError: 2765:0:(client.c:1057:ptlrpc_import_delay_req()) Skipped 23 previous similar messages May 17 14:39:41 client-5 xinetd[3391]: START: shell pid=4987 from=192.168.4.5 May 17 14:39:41 client-5 rshd[4988]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 14:39:41 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff8102c72f6400: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 14:39:42 client-5 xinetd[3391]: EXIT: shell status=0 pid=4987 duration=1(sec) May 17 14:39:42 client-5 xinetd[3391]: START: shell pid=5078 from=192.168.4.5 May 17 14:39:42 client-5 rshd[5079]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:39:42 client-5 xinetd[3391]: EXIT: shell status=0 pid=5078 duration=0(sec) May 17 14:39:42 client-5 xinetd[3391]: START: shell pid=5250 from=192.168.4.5 May 17 14:39:42 client-5 rshd[5251]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 4a: \|x\| 10 open\(O_CREAT\)s ====================================================== 14:39:42 \(1305668382\)");echo XXRETCODE:$?' May 17 14:39:42 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 4a: |x| 10 open(O_CREAT)s ====================================================== 14:39:42 (1305668382) May 17 14:39:42 client-5 xinetd[3391]: EXIT: shell status=0 pid=5250 duration=0(sec) May 17 14:39:42 client-5 xinetd[3391]: START: shell pid=5300 from=192.168.4.5 May 17 14:39:42 client-5 rshd[5302]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 14:39:42 client-5 xinetd[3391]: EXIT: shell status=0 pid=5300 duration=0(sec) May 17 14:39:43 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 14:39:47 client-5 kernel: Lustre: 20199:0:(import.c:529:import_select_connection()) MGC192.168.4.128@o2ib: tried all connections, increasing latency to 21s May 17 14:40:01 client-5 xinetd[3391]: START: shell pid=5488 from=192.168.4.5 May 17 14:40:01 client-5 rshd[5489]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 14:40:02 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff8102c72f6400: Connection restored to service lustre-MDT0000 using nid 192.168.4.128@o2ib. May 17 14:40:02 client-5 kernel: Lustre: Skipped 1 previous similar message May 17 14:40:02 client-5 xinetd[3391]: EXIT: shell status=0 pid=5488 duration=1(sec) May 17 14:40:02 client-5 xinetd[3391]: START: shell pid=5589 from=192.168.4.5 May 17 14:40:02 client-5 rshd[5590]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:40:02 client-5 xinetd[3391]: EXIT: shell status=0 pid=5589 duration=0(sec) May 17 14:40:02 client-5 xinetd[3391]: START: shell pid=5761 from=192.168.4.5 May 17 14:40:02 client-5 rshd[5762]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 4b: \|x\| rm 10 files ============================================================ 14:40:02 \(1305668402\)");echo XXRETCODE:$?' May 17 14:40:02 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 4b: |x| rm 10 files ============================================================ 14:40:02 (1305668402) May 17 14:40:02 client-5 xinetd[3391]: EXIT: shell status=0 pid=5761 duration=0(sec) May 17 14:40:02 client-5 xinetd[3391]: START: shell pid=5812 from=192.168.4.5 May 17 14:40:02 client-5 rshd[5813]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 14:40:02 client-5 xinetd[3391]: EXIT: shell status=0 pid=5812 duration=0(sec) May 17 14:40:03 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 14:40:18 client-5 kernel: Lustre: 20197:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369045627420482 sent from lustre-MDT0000-mdc-ffff8102c72f6400 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305668392] [real_sent 1305668392] [current 1305668418] [deadline 26s] [delay 0s] req@ffff81033b1e1400 x1369045627420482/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 192/192 e 0 to 1 dl 1305668418 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 14:40:18 client-5 kernel: Lustre: 20197:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 9 previous similar messages May 17 14:40:22 client-5 xinetd[3391]: START: shell pid=5994 from=192.168.4.5 May 17 14:40:22 client-5 rshd[5996]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 14:40:22 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff8102c72f6400: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 14:40:22 client-5 kernel: Lustre: Skipped 1 previous similar message May 17 14:40:22 client-5 kernel: LustreError: 20198:0:(client.c:2570:ptlrpc_replay_interpret()) @@@ status -116, old was 0 req@ffff8102a9c65800 x1369045627420471/t51539607576(51539607576) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:23/10 lens 360/424 e 0 to 0 dl 1305668429 ref 2 fl Interpret:R/ffffffff/ffffffff rc -116/-1 May 17 14:40:22 client-5 kernel: LustreError: 20198:0:(client.c:2570:ptlrpc_replay_interpret()) Skipped 2 previous similar messages May 17 14:40:22 client-5 xinetd[3391]: EXIT: shell status=0 pid=5994 duration=0(sec) May 17 14:40:22 client-5 xinetd[3391]: START: shell pid=6086 from=192.168.4.5 May 17 14:40:22 client-5 rshd[6087]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:40:22 client-5 xinetd[3391]: EXIT: shell status=0 pid=6086 duration=0(sec) May 17 14:40:22 client-5 xinetd[3391]: START: shell pid=6257 from=192.168.4.5 May 17 14:40:22 client-5 rshd[6259]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 5: \|x\| 220 open\(O_CREAT\) ======================================================= 14:40:22 \(1305668422\)");echo XXRETCODE:$?' May 17 14:40:22 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 5: |x| 220 open(O_CREAT) ======================================================= 14:40:22 (1305668422) May 17 14:40:22 client-5 xinetd[3391]: EXIT: shell status=0 pid=6257 duration=0(sec) May 17 14:40:23 client-5 xinetd[3391]: START: shell pid=6309 from=192.168.4.5 May 17 14:40:23 client-5 rshd[6310]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 14:40:23 client-5 xinetd[3391]: EXIT: shell status=0 pid=6309 duration=0(sec) May 17 14:40:23 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 14:40:42 client-5 kernel: LustreError: 2765:0:(client.c:1057:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@ffff8102eb5a4000 x1369045627421291/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:/ffffffff/ffffffff rc 0/-1 May 17 14:40:42 client-5 kernel: LustreError: 2765:0:(client.c:1057:ptlrpc_import_delay_req()) Skipped 47 previous similar messages May 17 14:40:42 client-5 xinetd[3391]: START: shell pid=6491 from=192.168.4.5 May 17 14:40:42 client-5 rshd[6493]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 14:40:42 client-5 xinetd[3391]: EXIT: shell status=0 pid=6491 duration=0(sec) May 17 14:40:46 client-5 xinetd[3391]: START: shell pid=6805 from=192.168.4.5 May 17 14:40:46 client-5 rshd[6806]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:40:46 client-5 xinetd[3391]: EXIT: shell status=0 pid=6805 duration=0(sec) May 17 14:40:47 client-5 xinetd[3391]: START: shell pid=6977 from=192.168.4.5 May 17 14:40:47 client-5 rshd[6978]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 6a: mkdir + contained create =================================================== 14:40:47 \(1305668447\)");echo XXRETCODE:$?' May 17 14:40:47 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 6a: mkdir + contained create =================================================== 14:40:47 (1305668447) May 17 14:40:47 client-5 xinetd[3391]: EXIT: shell status=0 pid=6977 duration=0(sec) May 17 14:40:47 client-5 xinetd[3391]: START: shell pid=7028 from=192.168.4.5 May 17 14:40:47 client-5 rshd[7030]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 14:40:47 client-5 xinetd[3391]: EXIT: shell status=0 pid=7028 duration=0(sec) May 17 14:40:47 client-5 kernel: Lustre: 20199:0:(import.c:529:import_select_connection()) MGC192.168.4.128@o2ib: tried all connections, increasing latency to 21s May 17 14:40:47 client-5 kernel: Lustre: 20199:0:(import.c:529:import_select_connection()) Skipped 1 previous similar message May 17 14:40:47 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 14:40:47 client-5 kernel: Lustre: 20198:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74c881c9 to 0x62e5a8d74c92156 May 17 14:40:47 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 14:41:06 client-5 xinetd[3391]: START: shell pid=7219 from=192.168.4.5 May 17 14:41:06 client-5 rshd[7220]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 14:41:06 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff8102c72f6400: Connection restored to service lustre-MDT0000 using nid 192.168.4.128@o2ib. May 17 14:41:06 client-5 kernel: Lustre: Skipped 3 previous similar messages May 17 14:41:06 client-5 xinetd[3391]: EXIT: shell status=0 pid=7219 duration=0(sec) May 17 14:41:07 client-5 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail. May 17 14:41:07 client-5 kernel: Lustre: 20198:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74c92156 to 0x62e5a8d74c925d9 May 17 14:41:07 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 14:41:08 client-5 xinetd[3391]: START: shell pid=7313 from=192.168.4.5 May 17 14:41:08 client-5 rshd[7314]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:41:08 client-5 xinetd[3391]: EXIT: shell status=0 pid=7313 duration=0(sec) May 17 14:41:09 client-5 xinetd[3391]: START: shell pid=7485 from=192.168.4.5 May 17 14:41:09 client-5 rshd[7486]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 6b: \|X\| rmdir ================================================================== 14:41:09 \(1305668469\)");echo XXRETCODE:$?' May 17 14:41:09 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 6b: |X| rmdir ================================================================== 14:41:09 (1305668469) May 17 14:41:09 client-5 xinetd[3391]: EXIT: shell status=0 pid=7485 duration=0(sec) May 17 14:41:09 client-5 xinetd[3391]: START: shell pid=7537 from=192.168.4.5 May 17 14:41:09 client-5 rshd[7538]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 14:41:09 client-5 xinetd[3391]: EXIT: shell status=0 pid=7537 duration=0(sec) May 17 14:41:09 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 14:41:27 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff8102c72f6400: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 14:41:27 client-5 kernel: Lustre: Skipped 2 previous similar messages May 17 14:41:27 client-5 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail. May 17 14:41:27 client-5 kernel: Lustre: 20198:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74c925d9 to 0x62e5a8d74c927d1 May 17 14:41:27 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 14:41:28 client-5 xinetd[3391]: START: shell pid=7721 from=192.168.4.5 May 17 14:41:28 client-5 rshd[7722]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 14:41:28 client-5 kernel: LustreError: 20198:0:(client.c:2570:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff81031ff89000 x1369045627422487/t68719476741(68719476741) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 544/544 e 0 to 0 dl 1305668514 ref 2 fl Interpret:R/ffffffff/ffffffff rc 301/-1 May 17 14:41:28 client-5 xinetd[3391]: EXIT: shell status=0 pid=7721 duration=0(sec) May 17 14:41:29 client-5 xinetd[3391]: START: shell pid=7812 from=192.168.4.5 May 17 14:41:29 client-5 rshd[7813]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:41:29 client-5 xinetd[3391]: EXIT: shell status=0 pid=7812 duration=0(sec) May 17 14:41:29 client-5 xinetd[3391]: START: shell pid=7984 from=192.168.4.5 May 17 14:41:29 client-5 rshd[7985]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 7: mkdir \|X\| contained create ================================================== 14:41:29 \(1305668489\)");echo XXRETCODE:$?' May 17 14:41:29 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 7: mkdir |X| contained create ================================================== 14:41:29 (1305668489) May 17 14:41:29 client-5 xinetd[3391]: EXIT: shell status=0 pid=7984 duration=0(sec) May 17 14:41:29 client-5 xinetd[3391]: START: shell pid=8036 from=192.168.4.5 May 17 14:41:29 client-5 rshd[8037]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 14:41:29 client-5 xinetd[3391]: EXIT: shell status=0 pid=8036 duration=0(sec) May 17 14:41:29 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 14:41:47 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The obd_ping operation failed with -107 May 17 14:41:47 client-5 kernel: LustreError: Skipped 8 previous similar messages May 17 14:41:47 client-5 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail. May 17 14:41:47 client-5 kernel: Lustre: 20198:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74c927d1 to 0x62e5a8d74c92d57 May 17 14:41:47 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 14:41:48 client-5 xinetd[3391]: START: shell pid=8220 from=192.168.4.5 May 17 14:41:48 client-5 rshd[8221]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 14:41:48 client-5 xinetd[3391]: EXIT: shell status=0 pid=8220 duration=0(sec) May 17 14:41:48 client-5 xinetd[3391]: START: shell pid=8313 from=192.168.4.5 May 17 14:41:48 client-5 rshd[8314]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:41:48 client-5 xinetd[3391]: EXIT: shell status=0 pid=8313 duration=0(sec) May 17 14:41:49 client-5 xinetd[3391]: START: shell pid=8484 from=192.168.4.5 May 17 14:41:49 client-5 rshd[8486]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 8: creat open \|X\| close ======================================================== 14:41:49 \(1305668509\)");echo XXRETCODE:$?' May 17 14:41:49 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 8: creat open |X| close ======================================================== 14:41:49 (1305668509) May 17 14:41:49 client-5 xinetd[3391]: EXIT: shell status=0 pid=8484 duration=0(sec) May 17 14:41:49 client-5 xinetd[3391]: START: shell pid=8537 from=192.168.4.5 May 17 14:41:49 client-5 rshd[8538]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 14:41:49 client-5 xinetd[3391]: EXIT: shell status=0 pid=8537 duration=0(sec) May 17 14:41:49 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 14:42:07 client-5 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail. May 17 14:42:07 client-5 kernel: Lustre: 20198:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74c92d57 to 0x62e5a8d74c932e4 May 17 14:42:07 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 14:42:08 client-5 xinetd[3391]: START: shell pid=8723 from=192.168.4.5 May 17 14:42:08 client-5 rshd[8724]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 14:42:08 client-5 xinetd[3391]: EXIT: shell status=0 pid=8723 duration=0(sec) May 17 14:42:08 client-5 xinetd[3391]: START: shell pid=8815 from=192.168.4.5 May 17 14:42:08 client-5 rshd[8817]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:42:08 client-5 xinetd[3391]: EXIT: shell status=0 pid=8815 duration=0(sec) May 17 14:42:09 client-5 xinetd[3391]: START: shell pid=8988 from=192.168.4.5 May 17 14:42:09 client-5 rshd[8989]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 9: \|X\| create \(same inum\/gen\) ================================================== 14:42:08 \(1305668528\)");echo XXRETCODE:$?' May 17 14:42:09 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 9: |X| create (same inum/gen) ================================================== 14:42:08 (1305668528) May 17 14:42:09 client-5 xinetd[3391]: EXIT: shell status=0 pid=8988 duration=0(sec) May 17 14:42:09 client-5 xinetd[3391]: START: shell pid=9039 from=192.168.4.5 May 17 14:42:09 client-5 rshd[9040]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 14:42:09 client-5 xinetd[3391]: EXIT: shell status=0 pid=9039 duration=0(sec) May 17 14:42:09 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 14:42:27 client-5 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail. May 17 14:42:27 client-5 kernel: Lustre: 20198:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74c932e4 to 0x62e5a8d74c93816 May 17 14:42:28 client-5 xinetd[3391]: START: shell pid=9230 from=192.168.4.5 May 17 14:42:28 client-5 rshd[9231]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 14:42:28 client-5 xinetd[3391]: EXIT: shell status=0 pid=9230 duration=0(sec) May 17 14:42:28 client-5 xinetd[3391]: START: shell pid=9324 from=192.168.4.5 May 17 14:42:28 client-5 rshd[9325]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:42:28 client-5 xinetd[3391]: EXIT: shell status=0 pid=9324 duration=0(sec) May 17 14:42:28 client-5 kernel: Lustre: 20197:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369045627422605 sent from lustre-MDT0000-mdc-ffff8102c72f6400 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305668522] [real_sent 1305668522] [current 1305668548] [deadline 26s] [delay 0s] req@ffff810319d9a400 x1369045627422605/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 192/192 e 0 to 1 dl 1305668548 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 14:42:28 client-5 kernel: Lustre: 20197:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 13 previous similar messages May 17 14:42:29 client-5 xinetd[3391]: START: shell pid=9495 from=192.168.4.5 May 17 14:42:29 client-5 rshd[9497]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 10: create \|X\| rename unlink =================================================== 14:42:28 \(1305668548\)");echo XXRETCODE:$?' May 17 14:42:29 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 10: create |X| rename unlink =================================================== 14:42:28 (1305668548) May 17 14:42:29 client-5 xinetd[3391]: EXIT: shell status=0 pid=9495 duration=0(sec) May 17 14:42:29 client-5 xinetd[3391]: START: shell pid=9548 from=192.168.4.5 May 17 14:42:29 client-5 rshd[9549]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 14:42:29 client-5 xinetd[3391]: EXIT: shell status=0 pid=9548 duration=0(sec) May 17 14:42:29 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 14:42:47 client-5 kernel: Lustre: 20198:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74c93816 to 0x62e5a8d74c93db1 May 17 14:42:47 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 14:42:47 client-5 kernel: Lustre: Skipped 1 previous similar message May 17 14:42:48 client-5 xinetd[3391]: START: shell pid=9733 from=192.168.4.5 May 17 14:42:48 client-5 rshd[9734]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 14:42:48 client-5 xinetd[3391]: EXIT: shell status=0 pid=9733 duration=0(sec) May 17 14:42:48 client-5 xinetd[3391]: START: shell pid=9826 from=192.168.4.5 May 17 14:42:48 client-5 rshd[9827]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:42:48 client-5 xinetd[3391]: EXIT: shell status=0 pid=9826 duration=0(sec) May 17 14:42:48 client-5 xinetd[3391]: START: shell pid=9996 from=192.168.4.5 May 17 14:42:48 client-5 rshd[9999]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 11: create open write rename \|X\| create-old-name read ========================== 14:42:48 \(1305668568\)");echo XXRETCODE:$?' May 17 14:42:48 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 11: create open write rename |X| create-old-name read ========================== 14:42:48 (1305668568) May 17 14:42:48 client-5 xinetd[3391]: EXIT: shell status=0 pid=9996 duration=0(sec) May 17 14:42:49 client-5 xinetd[3391]: START: shell pid=10051 from=192.168.4.5 May 17 14:42:49 client-5 rshd[10052]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 14:42:49 client-5 xinetd[3391]: EXIT: shell status=0 pid=10051 duration=0(sec) May 17 14:42:49 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 14:43:07 client-5 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail. May 17 14:43:07 client-5 kernel: LustreError: Skipped 1 previous similar message May 17 14:43:07 client-5 kernel: LustreError: 2765:0:(client.c:1057:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@ffff81031c46fc00 x1369045627422788/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:/ffffffff/ffffffff rc 0/-1 May 17 14:43:07 client-5 kernel: LustreError: 2765:0:(client.c:1057:ptlrpc_import_delay_req()) Skipped 12 previous similar messages May 17 14:43:07 client-5 kernel: Lustre: 20198:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74c93db1 to 0x62e5a8d74c94410 May 17 14:43:08 client-5 xinetd[3391]: START: shell pid=10236 from=192.168.4.5 May 17 14:43:08 client-5 rshd[10237]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 14:43:08 client-5 xinetd[3391]: EXIT: shell status=0 pid=10236 duration=0(sec) May 17 14:43:08 client-5 xinetd[3391]: START: shell pid=10328 from=192.168.4.5 May 17 14:43:08 client-5 rshd[10329]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:43:08 client-5 xinetd[3391]: EXIT: shell status=0 pid=10328 duration=0(sec) May 17 14:43:08 client-5 xinetd[3391]: START: shell pid=10500 from=192.168.4.5 May 17 14:43:08 client-5 rshd[10501]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 12: open, unlink \|X\| close ===================================================== 14:43:08 \(1305668588\)");echo XXRETCODE:$?' May 17 14:43:08 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 12: open, unlink |X| close ===================================================== 14:43:08 (1305668588) May 17 14:43:08 client-5 xinetd[3391]: EXIT: shell status=0 pid=10500 duration=0(sec) May 17 14:43:08 client-5 xinetd[3391]: START: shell pid=10556 from=192.168.4.5 May 17 14:43:08 client-5 rshd[10557]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 14:43:09 client-5 xinetd[3391]: EXIT: shell status=0 pid=10556 duration=1(sec) May 17 14:43:09 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 14:43:27 client-5 kernel: Lustre: 20198:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74c94410 to 0x62e5a8d74c949e3 May 17 14:43:27 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Connection restored to service MGS using nid 192.168.4.128@o2ib. May 17 14:43:27 client-5 kernel: Lustre: Skipped 13 previous similar messages May 17 14:43:28 client-5 kernel: LustreError: 20198:0:(client.c:2570:ptlrpc_replay_interpret()) @@@ status -116, old was 0 req@ffff8102f2c5fc00 x1369045627422817/t94489280525(94489280525) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:23/10 lens 360/424 e 0 to 0 dl 1305668614 ref 2 fl Interpret:R/ffffffff/ffffffff rc -116/-1 May 17 14:43:28 client-5 xinetd[3391]: START: shell pid=10739 from=192.168.4.5 May 17 14:43:28 client-5 rshd[10740]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 14:43:28 client-5 xinetd[3391]: EXIT: shell status=0 pid=10739 duration=0(sec) May 17 14:43:28 client-5 xinetd[3391]: START: shell pid=10829 from=192.168.4.5 May 17 14:43:28 client-5 rshd[10830]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:43:28 client-5 xinetd[3391]: EXIT: shell status=0 pid=10829 duration=0(sec) May 17 14:43:28 client-5 xinetd[3391]: START: shell pid=11001 from=192.168.4.5 May 17 14:43:28 client-5 rshd[11002]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 13: open chmod 0 \|x\| write close =============================================== 14:43:28 \(1305668608\)");echo XXRETCODE:$?' May 17 14:43:28 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 13: open chmod 0 |x| write close =============================================== 14:43:28 (1305668608) May 17 14:43:28 client-5 xinetd[3391]: EXIT: shell status=0 pid=11001 duration=0(sec) May 17 14:43:28 client-5 xinetd[3391]: START: shell pid=11058 from=192.168.4.5 May 17 14:43:28 client-5 rshd[11059]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 14:43:29 client-5 xinetd[3391]: EXIT: shell status=0 pid=11058 duration=1(sec) May 17 14:43:29 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 14:43:47 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff8102c72f6400: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 14:43:47 client-5 kernel: Lustre: Skipped 6 previous similar messages May 17 14:43:47 client-5 kernel: Lustre: 20198:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74c949e3 to 0x62e5a8d74c94f2a May 17 14:43:48 client-5 xinetd[3391]: START: shell pid=11245 from=192.168.4.5 May 17 14:43:48 client-5 rshd[11246]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 14:43:48 client-5 xinetd[3391]: EXIT: shell status=0 pid=11245 duration=0(sec) May 17 14:43:48 client-5 xinetd[3391]: START: shell pid=11337 from=192.168.4.5 May 17 14:43:48 client-5 rshd[11338]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:43:48 client-5 xinetd[3391]: EXIT: shell status=0 pid=11337 duration=0(sec) May 17 14:43:48 client-5 xinetd[3391]: START: shell pid=11509 from=192.168.4.5 May 17 14:43:48 client-5 rshd[11510]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 14: open\(O_CREAT\), unlink \|X\| close ============================================ 14:43:48 \(1305668628\)");echo XXRETCODE:$?' May 17 14:43:48 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 14: open(O_CREAT), unlink |X| close ============================================ 14:43:48 (1305668628) May 17 14:43:48 client-5 xinetd[3391]: EXIT: shell status=0 pid=11509 duration=0(sec) May 17 14:43:48 client-5 xinetd[3391]: START: shell pid=11564 from=192.168.4.5 May 17 14:43:48 client-5 rshd[11565]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 14:43:49 client-5 xinetd[3391]: EXIT: shell status=0 pid=11564 duration=1(sec) May 17 14:43:49 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 14:44:07 client-5 kernel: Lustre: 20198:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74c94f2a to 0x62e5a8d74c9547f May 17 14:44:07 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 14:44:07 client-5 kernel: Lustre: Skipped 3 previous similar messages May 17 14:44:07 client-5 kernel: LustreError: 20198:0:(client.c:2570:ptlrpc_replay_interpret()) @@@ status -116, old was 0 req@ffff8102cd8e5800 x1369045627422923/t103079215114(103079215114) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:23/10 lens 360/424 e 0 to 0 dl 1305668654 ref 2 fl Interpret:R/ffffffff/ffffffff rc -116/-1 May 17 14:44:08 client-5 xinetd[3391]: START: shell pid=11746 from=192.168.4.5 May 17 14:44:08 client-5 rshd[11748]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 14:44:08 client-5 xinetd[3391]: EXIT: shell status=0 pid=11746 duration=0(sec) May 17 14:44:08 client-5 xinetd[3391]: START: shell pid=11837 from=192.168.4.5 May 17 14:44:08 client-5 rshd[11838]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:44:08 client-5 xinetd[3391]: EXIT: shell status=0 pid=11837 duration=0(sec) May 17 14:44:08 client-5 xinetd[3391]: START: shell pid=12009 from=192.168.4.5 May 17 14:44:08 client-5 rshd[12010]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 15: open\(O_CREAT\), unlink \|X\| touch new, close ================================ 14:44:08 \(1305668648\)");echo XXRETCODE:$?' May 17 14:44:08 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 15: open(O_CREAT), unlink |X| touch new, close ================================ 14:44:08 (1305668648) May 17 14:44:08 client-5 xinetd[3391]: EXIT: shell status=0 pid=12009 duration=0(sec) May 17 14:44:08 client-5 xinetd[3391]: START: shell pid=12064 from=192.168.4.5 May 17 14:44:08 client-5 rshd[12065]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 14:44:08 client-5 xinetd[3391]: EXIT: shell status=0 pid=12064 duration=0(sec) May 17 14:44:09 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 14:44:27 client-5 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail. May 17 14:44:27 client-5 kernel: LustreError: Skipped 3 previous similar messages May 17 14:44:27 client-5 kernel: Lustre: 20198:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74c9547f to 0x62e5a8d74c959db May 17 14:44:27 client-5 xinetd[3391]: START: shell pid=12248 from=192.168.4.5 May 17 14:44:27 client-5 kernel: LustreError: 20198:0:(client.c:2570:ptlrpc_replay_interpret()) @@@ status -116, old was 0 req@ffff8102fceb0000 x1369045627422979/t107374182411(107374182411) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:23/10 lens 360/424 e 0 to 0 dl 1305668685 ref 2 fl Interpret:R/ffffffff/ffffffff rc -116/-1 May 17 14:44:27 client-5 rshd[12249]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 14:44:28 client-5 xinetd[3391]: EXIT: shell status=0 pid=12248 duration=1(sec) May 17 14:44:28 client-5 xinetd[3391]: START: shell pid=12339 from=192.168.4.5 May 17 14:44:28 client-5 rshd[12340]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:44:28 client-5 xinetd[3391]: EXIT: shell status=0 pid=12339 duration=0(sec) May 17 14:44:28 client-5 xinetd[3391]: START: shell pid=12509 from=192.168.4.5 May 17 14:44:28 client-5 rshd[12512]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 16: \|X\| open\(O_CREAT\), unlink, touch new, unlink new ========================== 14:44:28 \(1305668668\)");echo XXRETCODE:$?' May 17 14:44:28 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 16: |X| open(O_CREAT), unlink, touch new, unlink new ========================== 14:44:28 (1305668668) May 17 14:44:28 client-5 xinetd[3391]: EXIT: shell status=0 pid=12509 duration=0(sec) May 17 14:44:28 client-5 xinetd[3391]: START: shell pid=12562 from=192.168.4.5 May 17 14:44:28 client-5 rshd[12563]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 14:44:28 client-5 xinetd[3391]: EXIT: shell status=0 pid=12562 duration=0(sec) May 17 14:44:29 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 14:44:47 client-5 kernel: Lustre: 20198:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74c959db to 0x62e5a8d74c95f45 May 17 14:44:47 client-5 xinetd[3391]: START: shell pid=12748 from=192.168.4.5 May 17 14:44:47 client-5 rshd[12749]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 14:44:48 client-5 kernel: LustreError: 20198:0:(client.c:2570:ptlrpc_replay_interpret()) @@@ status -116, old was 0 req@ffff81031c46f400 x1369045627423024/t111669149699(111669149699) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:23/10 lens 360/424 e 0 to 0 dl 1305668694 ref 2 fl Interpret:R/ffffffff/ffffffff rc -116/-1 May 17 14:44:48 client-5 xinetd[3391]: EXIT: shell status=0 pid=12748 duration=1(sec) May 17 14:44:48 client-5 xinetd[3391]: START: shell pid=12839 from=192.168.4.5 May 17 14:44:48 client-5 rshd[12840]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:44:48 client-5 xinetd[3391]: EXIT: shell status=0 pid=12839 duration=0(sec) May 17 14:44:48 client-5 xinetd[3391]: START: shell pid=13011 from=192.168.4.5 May 17 14:44:48 client-5 rshd[13012]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 17: \|X\| open\(O_CREAT\), \|replay\| close ========================================== 14:44:48 \(1305668688\)");echo XXRETCODE:$?' May 17 14:44:48 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 17: |X| open(O_CREAT), |replay| close ========================================== 14:44:48 (1305668688) May 17 14:44:48 client-5 xinetd[3391]: EXIT: shell status=0 pid=13011 duration=0(sec) May 17 14:44:48 client-5 xinetd[3391]: START: shell pid=13062 from=192.168.4.5 May 17 14:44:48 client-5 rshd[13063]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 14:44:48 client-5 xinetd[3391]: EXIT: shell status=0 pid=13062 duration=0(sec) May 17 14:44:49 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 14:45:07 client-5 kernel: Lustre: 20198:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74c95f45 to 0x62e5a8d74c96493 May 17 14:45:07 client-5 xinetd[3391]: START: shell pid=13252 from=192.168.4.5 May 17 14:45:07 client-5 rshd[13254]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 14:45:08 client-5 xinetd[3391]: EXIT: shell status=0 pid=13252 duration=1(sec) May 17 14:45:08 client-5 xinetd[3391]: START: shell pid=13345 from=192.168.4.5 May 17 14:45:08 client-5 rshd[13346]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:45:08 client-5 xinetd[3391]: EXIT: shell status=0 pid=13345 duration=0(sec) May 17 14:45:08 client-5 xinetd[3391]: START: shell pid=13517 from=192.168.4.5 May 17 14:45:08 client-5 rshd[13518]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 18: \|X\| open\(O_CREAT\), unlink, touch new, close, touch, unlink ================= 14:45:08 \(1305668708\)");echo XXRETCODE:$?' May 17 14:45:08 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 18: |X| open(O_CREAT), unlink, touch new, close, touch, unlink ================= 14:45:08 (1305668708) May 17 14:45:08 client-5 xinetd[3391]: EXIT: shell status=0 pid=13517 duration=0(sec) May 17 14:45:08 client-5 xinetd[3391]: START: shell pid=13568 from=192.168.4.5 May 17 14:45:08 client-5 rshd[13569]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 14:45:08 client-5 xinetd[3391]: EXIT: shell status=0 pid=13568 duration=0(sec) May 17 14:45:09 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 14:45:27 client-5 kernel: Lustre: 20198:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74c96493 to 0x62e5a8d74c96a12 May 17 14:45:27 client-5 xinetd[3391]: START: shell pid=13756 from=192.168.4.5 May 17 14:45:27 client-5 rshd[13757]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 14:45:28 client-5 xinetd[3391]: EXIT: shell status=0 pid=13756 duration=1(sec) May 17 14:45:28 client-5 xinetd[3391]: START: shell pid=13849 from=192.168.4.5 May 17 14:45:28 client-5 rshd[13850]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:45:28 client-5 xinetd[3391]: EXIT: shell status=0 pid=13849 duration=0(sec) May 17 14:45:28 client-5 xinetd[3391]: START: shell pid=14021 from=192.168.4.5 May 17 14:45:28 client-5 rshd[14022]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 19: \|X\| mcreate, open, write, rename =========================================== 14:45:28 \(1305668728\)");echo XXRETCODE:$?' May 17 14:45:28 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 19: |X| mcreate, open, write, rename =========================================== 14:45:28 (1305668728) May 17 14:45:28 client-5 xinetd[3391]: EXIT: shell status=0 pid=14021 duration=0(sec) May 17 14:45:28 client-5 xinetd[3391]: START: shell pid=14072 from=192.168.4.5 May 17 14:45:28 client-5 rshd[14073]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 14:45:28 client-5 xinetd[3391]: EXIT: shell status=0 pid=14072 duration=0(sec) May 17 14:45:29 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 14:45:47 client-5 kernel: Lustre: 20198:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74c96a12 to 0x62e5a8d74c9709b May 17 14:45:47 client-5 xinetd[3391]: START: shell pid=14258 from=192.168.4.5 May 17 14:45:47 client-5 rshd[14259]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 14:45:48 client-5 xinetd[3391]: EXIT: shell status=0 pid=14258 duration=1(sec) May 17 14:45:48 client-5 xinetd[3391]: START: shell pid=14349 from=192.168.4.5 May 17 14:45:48 client-5 rshd[14350]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:45:48 client-5 xinetd[3391]: EXIT: shell status=0 pid=14349 duration=0(sec) May 17 14:45:48 client-5 xinetd[3391]: START: shell pid=14520 from=192.168.4.5 May 17 14:45:48 client-5 rshd[14522]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 20a: \|X\| open\(O_CREAT\), unlink, replay, close \(test mds_cleanup_orphans\) ======= 14:45:48 \(1305668748\)");echo XXRETCODE:$?' May 17 14:45:48 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 20a: |X| open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ======= 14:45:48 (1305668748) May 17 14:45:48 client-5 xinetd[3391]: EXIT: shell status=0 pid=14520 duration=0(sec) May 17 14:45:48 client-5 xinetd[3391]: START: shell pid=14572 from=192.168.4.5 May 17 14:45:48 client-5 rshd[14573]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 14:45:48 client-5 xinetd[3391]: EXIT: shell status=0 pid=14572 duration=0(sec) May 17 14:45:49 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 14:46:07 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The obd_ping operation failed with -107 May 17 14:46:07 client-5 kernel: LustreError: Skipped 25 previous similar messages May 17 14:46:07 client-5 kernel: Lustre: 20198:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74c9709b to 0x62e5a8d74c97628 May 17 14:46:07 client-5 xinetd[3391]: START: shell pid=14762 from=192.168.4.5 May 17 14:46:07 client-5 rshd[14764]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 14:46:08 client-5 xinetd[3391]: EXIT: shell status=0 pid=14762 duration=1(sec) May 17 14:46:08 client-5 xinetd[3391]: START: shell pid=14853 from=192.168.4.5 May 17 14:46:08 client-5 rshd[14854]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:46:08 client-5 xinetd[3391]: EXIT: shell status=0 pid=14853 duration=0(sec) May 17 14:46:08 client-5 xinetd[3391]: START: shell pid=15025 from=192.168.4.5 May 17 14:46:08 client-5 rshd[15026]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 20b: write, unlink, eviction, replay, \(test mds_cleanup_orphans\) =============== 14:46:08 \(1305668768\)");echo XXRETCODE:$?' May 17 14:46:08 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 20b: write, unlink, eviction, replay, (test mds_cleanup_orphans) =============== 14:46:08 (1305668768) May 17 14:46:08 client-5 xinetd[3391]: EXIT: shell status=0 pid=15025 duration=0(sec) May 17 14:46:08 client-5 kernel: LustreError: 15064:0:(vvp_io.c:987:vvp_io_commit_write()) Write page 1298 of inode ffff81032a6a6bd0 failed -2 May 17 14:46:08 client-5 kernel: LustreError: 167-0: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. May 17 14:46:08 client-5 kernel: LustreError: 15064:0:(file.c:155:ll_close_inode_openhandle()) inode 144115339473453346 mdc close failed: rc = -4 May 17 14:46:27 client-5 kernel: Lustre: 20198:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74c97628 to 0x62e5a8d74c97b45 May 17 14:46:27 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 14:46:27 client-5 kernel: Lustre: Skipped 6 previous similar messages May 17 14:46:28 client-5 xinetd[3391]: START: shell pid=15199 from=192.168.4.5 May 17 14:46:28 client-5 rshd[15200]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 14:46:28 client-5 xinetd[3391]: EXIT: shell status=0 pid=15199 duration=0(sec) May 17 14:46:33 client-5 xinetd[3391]: START: shell pid=15389 from=192.168.4.5 May 17 14:46:33 client-5 rshd[15390]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark before 3024924, after 3024924");echo XXRETCODE:$?' May 17 14:46:33 client-5 kernel: Lustre: DEBUG MARKER: before 3024924, after 3024924 May 17 14:46:33 client-5 xinetd[3391]: EXIT: shell status=0 pid=15389 duration=0(sec) May 17 14:46:33 client-5 xinetd[3391]: START: shell pid=15478 from=192.168.4.5 May 17 14:46:33 client-5 rshd[15481]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:46:33 client-5 xinetd[3391]: EXIT: shell status=0 pid=15478 duration=0(sec) May 17 14:46:33 client-5 xinetd[3391]: START: shell pid=15652 from=192.168.4.5 May 17 14:46:33 client-5 rshd[15653]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 20c: check that client eviction does not affect file content =================== 14:46:33 \(1305668793\)");echo XXRETCODE:$?' May 17 14:46:33 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 20c: check that client eviction does not affect file content =================== 14:46:33 (1305668793) May 17 14:46:33 client-5 xinetd[3391]: EXIT: shell status=0 pid=15652 duration=0(sec) May 17 14:46:34 client-5 kernel: LustreError: 167-0: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. May 17 14:46:34 client-5 kernel: LustreError: 15701:0:(lmv_obd.c:1198:lmv_statfs()) can't stat MDS #0 (lustre-MDT0000-mdc-ffff8102c72f6400), error -4 May 17 14:46:34 client-5 kernel: LustreError: 15701:0:(llite_lib.c:1370:ll_statfs_internal()) md_statfs fails: rc = -4 May 17 14:46:36 client-5 xinetd[3391]: START: shell pid=15761 from=192.168.4.5 May 17 14:46:36 client-5 rshd[15762]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:46:36 client-5 xinetd[3391]: EXIT: shell status=0 pid=15761 duration=0(sec) May 17 14:46:36 client-5 xinetd[3391]: START: shell pid=15933 from=192.168.4.5 May 17 14:46:36 client-5 rshd[15934]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 21: \|X\| open\(O_CREAT\), unlink touch new, replay, close \(test mds_cleanup_orphans\) ====================================================================================================== 14:46:36 \(1305668796\)");echo XXRETCODE:$?' May 17 14:46:36 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 21: |X| open(O_CREAT), unlink touch new, replay, close (test mds_cleanup_orphans) ====================================================================================================== 14:46:36 (1305668796) May 17 14:46:36 client-5 xinetd[3391]: EXIT: shell status=0 pid=15933 duration=0(sec) May 17 14:46:36 client-5 xinetd[3391]: START: shell pid=15984 from=192.168.4.5 May 17 14:46:36 client-5 rshd[15985]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 14:46:36 client-5 xinetd[3391]: EXIT: shell status=0 pid=15984 duration=0(sec) May 17 14:46:37 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 14:46:55 client-5 xinetd[3391]: START: shell pid=16171 from=192.168.4.5 May 17 14:46:55 client-5 rshd[16172]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 14:46:56 client-5 xinetd[3391]: EXIT: shell status=0 pid=16171 duration=1(sec) May 17 14:46:56 client-5 xinetd[3391]: START: shell pid=16262 from=192.168.4.5 May 17 14:46:56 client-5 rshd[16263]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:46:56 client-5 xinetd[3391]: EXIT: shell status=0 pid=16262 duration=0(sec) May 17 14:46:56 client-5 xinetd[3391]: START: shell pid=16434 from=192.168.4.5 May 17 14:46:56 client-5 rshd[16435]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 22: open\(O_CREAT\), \|X\| unlink, replay, close \(test mds_cleanup_orphans\) ======== 14:46:56 \(1305668816\)");echo XXRETCODE:$?' May 17 14:46:56 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 22: open(O_CREAT), |X| unlink, replay, close (test mds_cleanup_orphans) ======== 14:46:56 (1305668816) May 17 14:46:56 client-5 xinetd[3391]: EXIT: shell status=0 pid=16434 duration=0(sec) May 17 14:46:56 client-5 xinetd[3391]: START: shell pid=16488 from=192.168.4.5 May 17 14:46:56 client-5 rshd[16489]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 14:46:56 client-5 xinetd[3391]: EXIT: shell status=0 pid=16488 duration=0(sec) May 17 14:46:57 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 14:46:57 client-5 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail. May 17 14:46:57 client-5 kernel: LustreError: Skipped 6 previous similar messages May 17 14:46:57 client-5 kernel: Lustre: 20198:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74c97b45 to 0x62e5a8d74c98586 May 17 14:47:08 client-5 kernel: Lustre: 20197:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369045627423414 sent from lustre-MDT0000-mdc-ffff8102c72f6400 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305668802] [real_sent 1305668802] [current 1305668828] [deadline 26s] [delay 0s] req@ffff8102effad400 x1369045627423414/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 192/192 e 0 to 1 dl 1305668828 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 14:47:08 client-5 kernel: Lustre: 20197:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 21 previous similar messages May 17 14:47:15 client-5 xinetd[3391]: START: shell pid=16671 from=192.168.4.5 May 17 14:47:15 client-5 rshd[16673]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 14:47:16 client-5 xinetd[3391]: EXIT: shell status=0 pid=16671 duration=1(sec) May 17 14:47:16 client-5 xinetd[3391]: START: shell pid=16762 from=192.168.4.5 May 17 14:47:16 client-5 rshd[16763]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:47:16 client-5 xinetd[3391]: EXIT: shell status=0 pid=16762 duration=0(sec) May 17 14:47:16 client-5 xinetd[3391]: START: shell pid=16932 from=192.168.4.5 May 17 14:47:16 client-5 rshd[16935]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 23: open\(O_CREAT\), \|X\| unlink touch new, replay, close \(test mds_cleanup_orphans\) ====================================================================================================== 14:47:16 \(1305668836\)");echo XXRETCODE:$?' May 17 14:47:16 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 23: open(O_CREAT), |X| unlink touch new, replay, close (test mds_cleanup_orphans) ====================================================================================================== 14:47:16 (1305668836) May 17 14:47:16 client-5 xinetd[3391]: EXIT: shell status=0 pid=16932 duration=0(sec) May 17 14:47:16 client-5 xinetd[3391]: START: shell pid=16988 from=192.168.4.5 May 17 14:47:16 client-5 rshd[16989]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 14:47:16 client-5 xinetd[3391]: EXIT: shell status=0 pid=16988 duration=0(sec) May 17 14:47:17 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 14:47:27 client-5 kernel: LustreError: 2765:0:(client.c:1057:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@ffff81032b300000 x1369045627423539/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:/ffffffff/ffffffff rc 0/-1 May 17 14:47:27 client-5 kernel: LustreError: 2765:0:(client.c:1057:ptlrpc_import_delay_req()) Skipped 21 previous similar messages May 17 14:47:35 client-5 xinetd[3391]: START: shell pid=17176 from=192.168.4.5 May 17 14:47:35 client-5 rshd[17177]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 14:47:36 client-5 xinetd[3391]: EXIT: shell status=0 pid=17176 duration=1(sec) May 17 14:47:36 client-5 xinetd[3391]: START: shell pid=17267 from=192.168.4.5 May 17 14:47:36 client-5 rshd[17268]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:47:36 client-5 xinetd[3391]: EXIT: shell status=0 pid=17267 duration=0(sec) May 17 14:47:36 client-5 xinetd[3391]: START: shell pid=17439 from=192.168.4.5 May 17 14:47:36 client-5 rshd[17440]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 24: open\(O_CREAT\), replay, unlink, close \(test mds_cleanup_orphans\) ============ 14:47:36 \(1305668856\)");echo XXRETCODE:$?' May 17 14:47:36 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 24: open(O_CREAT), replay, unlink, close (test mds_cleanup_orphans) ============ 14:47:36 (1305668856) May 17 14:47:36 client-5 xinetd[3391]: EXIT: shell status=0 pid=17439 duration=0(sec) May 17 14:47:36 client-5 xinetd[3391]: START: shell pid=17493 from=192.168.4.5 May 17 14:47:36 client-5 rshd[17494]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 14:47:36 client-5 xinetd[3391]: EXIT: shell status=0 pid=17493 duration=0(sec) May 17 14:47:37 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 14:47:56 client-5 xinetd[3391]: START: shell pid=17675 from=192.168.4.5 May 17 14:47:56 client-5 rshd[17676]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 14:47:56 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff8102c72f6400: Connection restored to service lustre-MDT0000 using nid 192.168.4.128@o2ib. May 17 14:47:56 client-5 kernel: Lustre: Skipped 25 previous similar messages May 17 14:47:56 client-5 xinetd[3391]: EXIT: shell status=0 pid=17675 duration=0(sec) May 17 14:47:56 client-5 xinetd[3391]: START: shell pid=17764 from=192.168.4.5 May 17 14:47:56 client-5 rshd[17767]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:47:56 client-5 xinetd[3391]: EXIT: shell status=0 pid=17764 duration=0(sec) May 17 14:47:56 client-5 xinetd[3391]: START: shell pid=17938 from=192.168.4.5 May 17 14:47:56 client-5 rshd[17939]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 25: open\(O_CREAT\), unlink, replay, close \(test mds_cleanup_orphans\) ============ 14:47:56 \(1305668876\)");echo XXRETCODE:$?' May 17 14:47:56 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 25: open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ============ 14:47:56 (1305668876) May 17 14:47:56 client-5 xinetd[3391]: EXIT: shell status=0 pid=17938 duration=0(sec) May 17 14:47:56 client-5 xinetd[3391]: START: shell pid=17993 from=192.168.4.5 May 17 14:47:56 client-5 rshd[17994]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 14:47:57 client-5 xinetd[3391]: EXIT: shell status=0 pid=17993 duration=1(sec) May 17 14:47:57 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 14:47:57 client-5 kernel: Lustre: 20199:0:(import.c:529:import_select_connection()) MGC192.168.4.128@o2ib: tried all connections, increasing latency to 21s May 17 14:47:57 client-5 kernel: Lustre: 20198:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74c98586 to 0x62e5a8d74c99577 May 17 14:48:16 client-5 xinetd[3391]: START: shell pid=18176 from=192.168.4.5 May 17 14:48:16 client-5 rshd[18177]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 14:48:16 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff8102c72f6400: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 14:48:16 client-5 kernel: Lustre: Skipped 14 previous similar messages May 17 14:48:16 client-5 xinetd[3391]: EXIT: shell status=0 pid=18176 duration=0(sec) May 17 14:48:16 client-5 xinetd[3391]: START: shell pid=18266 from=192.168.4.5 May 17 14:48:16 client-5 rshd[18267]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:48:16 client-5 xinetd[3391]: EXIT: shell status=0 pid=18266 duration=0(sec) May 17 14:48:16 client-5 xinetd[3391]: START: shell pid=18438 from=192.168.4.5 May 17 14:48:17 client-5 rshd[18439]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 26: \|X\| open\(O_CREAT\), unlink two, close one, replay, close one \(test mds_cleanup_orphans\) ====================================================================================================== 14:48:16 \(1305668896\)");echo XXRETCODE:$?' May 17 14:48:17 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 26: |X| open(O_CREAT), unlink two, close one, replay, close one (test mds_cleanup_orphans) ====================================================================================================== 14:48:16 (1305668896) May 17 14:48:17 client-5 xinetd[3391]: EXIT: shell status=0 pid=18438 duration=1(sec) May 17 14:48:17 client-5 xinetd[3391]: START: shell pid=18489 from=192.168.4.5 May 17 14:48:17 client-5 rshd[18490]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 14:48:17 client-5 xinetd[3391]: EXIT: shell status=0 pid=18489 duration=0(sec) May 17 14:48:17 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 14:48:17 client-5 kernel: Lustre: 20198:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74c99577 to 0x62e5a8d74c99aef May 17 14:48:36 client-5 xinetd[3391]: START: shell pid=18684 from=192.168.4.5 May 17 14:48:36 client-5 rshd[18685]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 14:48:36 client-5 kernel: LustreError: 20198:0:(client.c:2570:ptlrpc_replay_interpret()) @@@ status -116, old was 0 req@ffff81033ba46c00 x1369045627423678/t158913789954(158913789954) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:23/10 lens 360/424 e 0 to 0 dl 1305668923 ref 2 fl Interpret:R/ffffffff/ffffffff rc -116/-1 May 17 14:48:36 client-5 xinetd[3391]: EXIT: shell status=0 pid=18684 duration=0(sec) May 17 14:48:36 client-5 xinetd[3391]: START: shell pid=18774 from=192.168.4.5 May 17 14:48:36 client-5 rshd[18775]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:48:36 client-5 xinetd[3391]: EXIT: shell status=0 pid=18774 duration=0(sec) May 17 14:48:37 client-5 xinetd[3391]: START: shell pid=18945 from=192.168.4.5 May 17 14:48:37 client-5 rshd[18947]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 27: \|X\| open\(O_CREAT\), unlink two, replay, close two \(test mds_cleanup_orphans\) ====================================================================================================== 14:48:37 \(1305668917\)");echo XXRETCODE:$?' May 17 14:48:37 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 27: |X| open(O_CREAT), unlink two, replay, close two (test mds_cleanup_orphans) ====================================================================================================== 14:48:37 (1305668917) May 17 14:48:37 client-5 xinetd[3391]: EXIT: shell status=0 pid=18945 duration=0(sec) May 17 14:48:37 client-5 xinetd[3391]: START: shell pid=18997 from=192.168.4.5 May 17 14:48:37 client-5 rshd[18998]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 14:48:37 client-5 xinetd[3391]: EXIT: shell status=0 pid=18997 duration=0(sec) May 17 14:48:37 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 14:48:37 client-5 kernel: Lustre: 20198:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74c99aef to 0x62e5a8d74c9a0de May 17 14:48:56 client-5 xinetd[3391]: START: shell pid=19188 from=192.168.4.5 May 17 14:48:56 client-5 rshd[19189]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 14:48:56 client-5 xinetd[3391]: EXIT: shell status=0 pid=19188 duration=0(sec) May 17 14:48:56 client-5 xinetd[3391]: START: shell pid=19278 from=192.168.4.5 May 17 14:48:56 client-5 rshd[19279]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:48:57 client-5 xinetd[3391]: EXIT: shell status=0 pid=19278 duration=1(sec) May 17 14:48:57 client-5 xinetd[3391]: START: shell pid=19450 from=192.168.4.5 May 17 14:48:57 client-5 rshd[19451]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 28: open\(O_CREAT\), \|X\| unlink two, close one, replay, close one \(test mds_cleanup_orphans\) ====================================================================================================== 14:48:57 \(1305668937\)");echo XXRETCODE:$?' May 17 14:48:57 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 28: open(O_CREAT), |X| unlink two, close one, replay, close one (test mds_cleanup_orphans) ====================================================================================================== 14:48:57 (1305668937) May 17 14:48:57 client-5 xinetd[3391]: EXIT: shell status=0 pid=19450 duration=0(sec) May 17 14:48:57 client-5 xinetd[3391]: START: shell pid=19507 from=192.168.4.5 May 17 14:48:57 client-5 rshd[19508]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 14:48:57 client-5 xinetd[3391]: EXIT: shell status=0 pid=19507 duration=0(sec) May 17 14:48:57 client-5 kernel: Lustre: 20198:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74c9a0de to 0x62e5a8d74c9a66b May 17 14:48:57 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 14:49:16 client-5 xinetd[3391]: START: shell pid=19691 from=192.168.4.5 May 17 14:49:16 client-5 rshd[19693]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 14:49:16 client-5 xinetd[3391]: EXIT: shell status=0 pid=19691 duration=0(sec) May 17 14:49:17 client-5 xinetd[3391]: START: shell pid=19780 from=192.168.4.5 May 17 14:49:17 client-5 rshd[19783]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:49:17 client-5 xinetd[3391]: EXIT: shell status=0 pid=19780 duration=0(sec) May 17 14:49:17 client-5 xinetd[3391]: START: shell pid=19954 from=192.168.4.5 May 17 14:49:17 client-5 rshd[19955]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 29: open\(O_CREAT\), \|X\| unlink two, replay, close two \(test mds_cleanup_orphans\) ====================================================================================================== 14:49:17 \(1305668957\)");echo XXRETCODE:$?' May 17 14:49:17 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 29: open(O_CREAT), |X| unlink two, replay, close two (test mds_cleanup_orphans) ====================================================================================================== 14:49:17 (1305668957) May 17 14:49:17 client-5 xinetd[3391]: EXIT: shell status=0 pid=19954 duration=0(sec) May 17 14:49:17 client-5 xinetd[3391]: START: shell pid=20011 from=192.168.4.5 May 17 14:49:17 client-5 rshd[20012]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 14:49:17 client-5 xinetd[3391]: EXIT: shell status=0 pid=20011 duration=0(sec) May 17 14:49:17 client-5 kernel: Lustre: 20198:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74c9a66b to 0x62e5a8d74c9ac37 May 17 14:49:18 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 14:49:36 client-5 xinetd[3391]: START: shell pid=20227 from=192.168.4.5 May 17 14:49:36 client-5 rshd[20228]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 14:49:37 client-5 xinetd[3391]: EXIT: shell status=0 pid=20227 duration=1(sec) May 17 14:49:37 client-5 xinetd[3391]: START: shell pid=20317 from=192.168.4.5 May 17 14:49:37 client-5 rshd[20318]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:49:37 client-5 xinetd[3391]: EXIT: shell status=0 pid=20317 duration=0(sec) May 17 14:49:37 client-5 xinetd[3391]: START: shell pid=20490 from=192.168.4.5 May 17 14:49:37 client-5 rshd[20491]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 30: open\(O_CREAT\) two, unlink two, replay, close two \(test mds_cleanup_orphans\) ====================================================================================================== 14:49:37 \(1305668977\)");echo XXRETCODE:$?' May 17 14:49:37 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 30: open(O_CREAT) two, unlink two, replay, close two (test mds_cleanup_orphans) ====================================================================================================== 14:49:37 (1305668977) May 17 14:49:37 client-5 xinetd[3391]: EXIT: shell status=0 pid=20490 duration=0(sec) May 17 14:49:37 client-5 xinetd[3391]: START: shell pid=20549 from=192.168.4.5 May 17 14:49:37 client-5 rshd[20550]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 14:49:37 client-5 xinetd[3391]: EXIT: shell status=0 pid=20549 duration=0(sec) May 17 14:49:38 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 14:49:56 client-5 xinetd[3391]: START: shell pid=20735 from=192.168.4.5 May 17 14:49:57 client-5 rshd[20736]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 14:50:00 client-5 xinetd[3391]: EXIT: shell status=0 pid=20735 duration=4(sec) May 17 14:50:00 client-5 xinetd[3391]: START: shell pid=20825 from=192.168.4.5 May 17 14:50:00 client-5 rshd[20826]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:50:01 client-5 xinetd[3391]: EXIT: shell status=0 pid=20825 duration=1(sec) May 17 14:50:01 client-5 xinetd[3391]: START: shell pid=20997 from=192.168.4.5 May 17 14:50:01 client-5 rshd[20998]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 31: open\(O_CREAT\) two, unlink one, \|X\| unlink one, close two \(test mds_cleanup_orphans\) ====================================================================================================== 14:50:01 \(1305669001\)");echo XXRETCODE:$?' May 17 14:50:01 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 31: open(O_CREAT) two, unlink one, |X| unlink one, close two (test mds_cleanup_orphans) ====================================================================================================== 14:50:01 (1305669001) May 17 14:50:01 client-5 xinetd[3391]: EXIT: shell status=0 pid=20997 duration=0(sec) May 17 14:50:01 client-5 xinetd[3391]: START: shell pid=21055 from=192.168.4.5 May 17 14:50:01 client-5 rshd[21056]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 14:50:01 client-5 xinetd[3391]: EXIT: shell status=0 pid=21055 duration=0(sec) May 17 14:50:01 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 14:50:14 client-5 kernel: Lustre: 20199:0:(import.c:529:import_select_connection()) MGC192.168.4.128@o2ib: tried all connections, increasing latency to 16s May 17 14:50:14 client-5 kernel: Lustre: 20199:0:(import.c:529:import_select_connection()) Skipped 2 previous similar messages May 17 14:50:20 client-5 xinetd[3391]: START: shell pid=21238 from=192.168.4.5 May 17 14:50:20 client-5 rshd[21239]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 14:50:21 client-5 xinetd[3391]: EXIT: shell status=0 pid=21238 duration=1(sec) May 17 14:50:21 client-5 xinetd[3391]: START: shell pid=21328 from=192.168.4.5 May 17 14:50:21 client-5 rshd[21329]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:50:21 client-5 xinetd[3391]: EXIT: shell status=0 pid=21328 duration=0(sec) May 17 14:50:21 client-5 xinetd[3391]: START: shell pid=21500 from=192.168.4.5 May 17 14:50:21 client-5 rshd[21501]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 32: close\(\) notices client eviction\; close\(\) after client eviction ============= 14:50:21 \(1305669021\)");echo XXRETCODE:$?' May 17 14:50:21 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 32: close() notices client eviction; close() after client eviction ============= 14:50:21 (1305669021) May 17 14:50:21 client-5 xinetd[3391]: EXIT: shell status=0 pid=21500 duration=0(sec) May 17 14:50:22 client-5 kernel: LustreError: 167-0: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. May 17 14:50:22 client-5 kernel: LustreError: 21551:0:(lmv_obd.c:1198:lmv_statfs()) can't stat MDS #0 (lustre-MDT0000-mdc-ffff8102c72f6400), error -4 May 17 14:50:22 client-5 kernel: LustreError: 21551:0:(llite_lib.c:1370:ll_statfs_internal()) md_statfs fails: rc = -4 May 17 14:50:23 client-5 xinetd[3391]: START: shell pid=21609 from=192.168.4.5 May 17 14:50:23 client-5 rshd[21612]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:50:23 client-5 xinetd[3391]: EXIT: shell status=0 pid=21609 duration=0(sec) May 17 14:50:24 client-5 xinetd[3391]: START: shell pid=21774 from=192.168.4.5 May 17 14:50:24 client-5 rshd[21775]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark SKIP: replay-single test_33a skipping ALWAYS excluded test 33a");echo XXRETCODE:$?' May 17 14:50:24 client-5 kernel: Lustre: DEBUG MARKER: SKIP: replay-single test_33a skipping ALWAYS excluded test 33a May 17 14:50:24 client-5 xinetd[3391]: EXIT: shell status=0 pid=21774 duration=0(sec) May 17 14:50:24 client-5 xinetd[3391]: START: shell pid=21870 from=192.168.4.5 May 17 14:50:24 client-5 rshd[21871]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark SKIP: replay-single test_33b skipping ALWAYS excluded test 33b");echo XXRETCODE:$?' May 17 14:50:24 client-5 kernel: Lustre: DEBUG MARKER: SKIP: replay-single test_33b skipping ALWAYS excluded test 33b May 17 14:50:24 client-5 xinetd[3391]: EXIT: shell status=0 pid=21870 duration=0(sec) May 17 14:50:24 client-5 xinetd[3391]: START: shell pid=21975 from=192.168.4.5 May 17 14:50:24 client-5 rshd[21976]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 34: abort recovery before client does replay \(test mds_cleanup_orphans\) ======== 14:50:24 \(1305669024\)");echo XXRETCODE:$?' May 17 14:50:24 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 34: abort recovery before client does replay (test mds_cleanup_orphans) ======== 14:50:24 (1305669024) May 17 14:50:24 client-5 xinetd[3391]: EXIT: shell status=0 pid=21975 duration=0(sec) May 17 14:50:24 client-5 xinetd[3391]: START: shell pid=22030 from=192.168.4.5 May 17 14:50:24 client-5 rshd[22031]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 14:50:24 client-5 xinetd[3391]: EXIT: shell status=0 pid=22030 duration=0(sec) May 17 14:50:25 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 14:50:34 client-5 kernel: LustreError: 167-0: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. May 17 14:50:35 client-5 xinetd[3391]: START: shell pid=22173 from=192.168.4.5 May 17 14:50:35 client-5 rshd[22174]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 14:50:35 client-5 xinetd[3391]: EXIT: shell status=0 pid=22173 duration=0(sec) May 17 14:50:36 client-5 xinetd[3391]: START: shell pid=22213 from=192.168.4.5 May 17 14:50:36 client-5 rshd[22214]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 14:50:36 client-5 xinetd[3391]: EXIT: shell status=0 pid=22213 duration=0(sec) May 17 14:50:36 client-5 xinetd[3391]: START: shell pid=22303 from=192.168.4.5 May 17 14:50:36 client-5 rshd[22305]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:50:36 client-5 xinetd[3391]: EXIT: shell status=0 pid=22303 duration=0(sec) May 17 14:50:36 client-5 xinetd[3391]: START: shell pid=22476 from=192.168.4.5 May 17 14:50:36 client-5 rshd[22477]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 35: test recovery from llog for unlink op ====================================== 14:50:36 \(1305669036\)");echo XXRETCODE:$?' May 17 14:50:37 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 35: test recovery from llog for unlink op ====================================== 14:50:36 (1305669036) May 17 14:50:37 client-5 xinetd[3391]: EXIT: shell status=0 pid=22476 duration=1(sec) May 17 14:50:48 client-5 xinetd[3391]: START: shell pid=22602 from=192.168.4.5 May 17 14:50:48 client-5 rshd[22603]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 14:50:54 client-5 kernel: LustreError: 167-0: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. May 17 14:50:54 client-5 kernel: LustreError: 22635:0:(lmv_obd.c:1198:lmv_statfs()) can't stat MDS #0 (lustre-MDT0000-mdc-ffff8102c72f6400), error -4 May 17 14:50:54 client-5 kernel: LustreError: 22635:0:(llite_lib.c:1370:ll_statfs_internal()) md_statfs fails: rc = -4 May 17 14:50:54 client-5 xinetd[3391]: EXIT: shell status=0 pid=22602 duration=6(sec) May 17 14:50:55 client-5 xinetd[3391]: START: shell pid=22643 from=192.168.4.5 May 17 14:50:55 client-5 rshd[22644]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 14:50:55 client-5 xinetd[3391]: EXIT: shell status=0 pid=22643 duration=0(sec) May 17 14:50:55 client-5 xinetd[3391]: START: shell pid=22734 from=192.168.4.5 May 17 14:50:55 client-5 rshd[22735]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:50:55 client-5 xinetd[3391]: EXIT: shell status=0 pid=22734 duration=0(sec) May 17 14:50:55 client-5 xinetd[3391]: START: shell pid=22905 from=192.168.4.5 May 17 14:50:55 client-5 rshd[22907]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 36: don\'t resend cancel ======================================================== 14:50:55 \(1305669055\)");echo XXRETCODE:$?' May 17 14:50:55 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 36: don't resend cancel ======================================================== 14:50:55 (1305669055) May 17 14:50:55 client-5 xinetd[3391]: EXIT: shell status=0 pid=22905 duration=0(sec) May 17 14:50:55 client-5 xinetd[3391]: START: shell pid=22957 from=192.168.4.5 May 17 14:50:55 client-5 rshd[22958]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 14:50:56 client-5 xinetd[3391]: EXIT: shell status=0 pid=22957 duration=1(sec) May 17 14:50:56 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 14:51:14 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc start May 17 14:51:14 client-5 kernel: LustreError: 1870:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -11 from cancel RPC: canceling anyway May 17 14:51:14 client-5 kernel: LustreError: 1870:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Skipped 5 previous similar messages May 17 14:51:14 client-5 kernel: LustreError: 1870:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -11 May 17 14:51:14 client-5 kernel: LustreError: 1870:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) Skipped 5 previous similar messages May 17 14:51:14 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc stop May 17 14:51:14 client-5 xinetd[3391]: START: shell pid=23206 from=192.168.4.5 May 17 14:51:14 client-5 rshd[23207]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:51:14 client-5 xinetd[3391]: EXIT: shell status=0 pid=23206 duration=0(sec) May 17 14:51:14 client-5 xinetd[3391]: START: shell pid=23368 from=192.168.4.5 May 17 14:51:14 client-5 rshd[23369]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl set_param debug_mb=150");echo XXRETCODE:$?' May 17 14:51:14 client-5 xinetd[3391]: EXIT: shell status=0 pid=23368 duration=0(sec) May 17 14:51:14 client-5 xinetd[3391]: START: shell pid=23459 from=192.168.4.5 May 17 14:51:14 client-5 rshd[23460]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl set_param debug=-1;");echo XXRETCODE:$?' May 17 14:51:14 client-5 kernel: Lustre: 23493:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 14:51:14 client-5 kernel: Lustre: 23493:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message May 17 14:51:14 client-5 xinetd[3391]: EXIT: shell status=0 pid=23459 duration=0(sec) May 17 14:51:14 client-5 xinetd[3391]: START: shell pid=23562 from=192.168.4.5 May 17 14:51:14 client-5 rshd[23563]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 37: abort recovery before client does replay \(test mds_cleanup_orphans for directories\) ====================================================================================================== 14:51:14 \(1305669074\)");echo XXRETCODE:$?' May 17 14:51:15 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 37: abort recovery before client does replay (test mds_cleanup_orphans for directories) ====================================================================================================== 14:51:14 (1305669074) May 17 14:51:15 client-5 xinetd[3391]: EXIT: shell status=0 pid=23562 duration=1(sec) May 17 14:51:22 client-5 xinetd[3391]: START: shell pid=23617 from=192.168.4.5 May 17 14:51:22 client-5 rshd[23619]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 14:51:23 client-5 xinetd[3391]: EXIT: shell status=0 pid=23617 duration=1(sec) May 17 14:51:23 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 14:51:32 client-5 xinetd[3391]: START: shell pid=23761 from=192.168.4.5 May 17 14:51:32 client-5 rshd[23762]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 14:51:32 client-5 kernel: LustreError: 167-0: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. May 17 14:51:33 client-5 kernel: LustreError: 23794:0:(lmv_obd.c:1198:lmv_statfs()) can't stat MDS #0 (lustre-MDT0000-mdc-ffff8102c72f6400), error -4 May 17 14:51:33 client-5 kernel: LustreError: 23794:0:(llite_lib.c:1370:ll_statfs_internal()) md_statfs fails: rc = -4 May 17 14:51:33 client-5 xinetd[3391]: EXIT: shell status=0 pid=23761 duration=1(sec) May 17 14:51:34 client-5 xinetd[3391]: START: shell pid=23802 from=192.168.4.5 May 17 14:51:34 client-5 rshd[23803]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 14:51:34 client-5 xinetd[3391]: EXIT: shell status=0 pid=23802 duration=0(sec) May 17 14:51:34 client-5 xinetd[3391]: START: shell pid=23895 from=192.168.4.5 May 17 14:51:34 client-5 rshd[23896]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:51:34 client-5 xinetd[3391]: EXIT: shell status=0 pid=23895 duration=0(sec) May 17 14:51:34 client-5 xinetd[3391]: START: shell pid=24055 from=192.168.4.5 May 17 14:51:34 client-5 rshd[24056]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl set_param debug_mb=24");echo XXRETCODE:$?' May 17 14:51:34 client-5 xinetd[3391]: EXIT: shell status=0 pid=24055 duration=0(sec) May 17 14:51:34 client-5 xinetd[3391]: START: shell pid=24145 from=192.168.4.5 May 17 14:51:34 client-5 rshd[24147]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl set_param debug=\"inode super warning dlmtrace error emerg ha rpctrace vfstrace config console\";");echo XXRETCODE:$?' May 17 14:51:34 client-5 xinetd[3391]: EXIT: shell status=0 pid=24145 duration=0(sec) May 17 14:51:34 client-5 xinetd[3391]: START: shell pid=24249 from=192.168.4.5 May 17 14:51:34 client-5 rshd[24250]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 38: test recovery from unlink llog \(test llog_gen_rec\) ========================= 14:51:34 \(1305669094\)");echo XXRETCODE:$?' May 17 14:51:34 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 38: test recovery from unlink llog (test llog_gen_rec) ========================= 14:51:34 (1305669094) May 17 14:51:34 client-5 xinetd[3391]: EXIT: shell status=0 pid=24249 duration=0(sec) May 17 14:51:36 client-5 xinetd[3391]: START: shell pid=24302 from=192.168.4.5 May 17 14:51:36 client-5 rshd[24303]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 14:51:36 client-5 xinetd[3391]: EXIT: shell status=0 pid=24302 duration=0(sec) May 17 14:51:36 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 14:51:55 client-5 xinetd[3391]: START: shell pid=24484 from=192.168.4.5 May 17 14:51:55 client-5 rshd[24485]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 14:51:59 client-5 xinetd[3391]: EXIT: shell status=0 pid=24484 duration=4(sec) May 17 14:52:01 client-5 xinetd[3391]: START: shell pid=24577 from=192.168.4.5 May 17 14:52:02 client-5 rshd[24578]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:52:02 client-5 xinetd[3391]: EXIT: shell status=0 pid=24577 duration=1(sec) May 17 14:52:02 client-5 xinetd[3391]: START: shell pid=24749 from=192.168.4.5 May 17 14:52:02 client-5 rshd[24750]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 39: test recovery from unlink llog \(test llog_gen_rec\) ========================= 14:52:02 \(1305669122\)");echo XXRETCODE:$?' May 17 14:52:02 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 39: test recovery from unlink llog (test llog_gen_rec) ========================= 14:52:02 (1305669122) May 17 14:52:02 client-5 xinetd[3391]: EXIT: shell status=0 pid=24749 duration=0(sec) May 17 14:52:03 client-5 xinetd[3391]: START: shell pid=24801 from=192.168.4.5 May 17 14:52:03 client-5 rshd[24802]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 14:52:03 client-5 xinetd[3391]: EXIT: shell status=0 pid=24801 duration=0(sec) May 17 14:52:03 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 14:52:23 client-5 xinetd[3391]: START: shell pid=24987 from=192.168.4.5 May 17 14:52:23 client-5 rshd[24989]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 14:52:23 client-5 kernel: LustreError: 20198:0:(client.c:2570:ptlrpc_replay_interpret()) @@@ status -116, old was 0 req@ffff8102c1186000 x1369045627429970/t206158432208(206158432208) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:23/10 lens 360/424 e 0 to 0 dl 1305669155 ref 2 fl Interpret:R/ffffffff/ffffffff rc -116/-1 May 17 14:52:23 client-5 kernel: LustreError: 20198:0:(client.c:2570:ptlrpc_replay_interpret()) Skipped 2 previous similar messages May 17 14:52:23 client-5 xinetd[3391]: EXIT: shell status=0 pid=24987 duration=0(sec) May 17 14:52:25 client-5 xinetd[3391]: START: shell pid=25081 from=192.168.4.5 May 17 14:52:26 client-5 rshd[25082]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:52:26 client-5 xinetd[3391]: EXIT: shell status=0 pid=25081 duration=1(sec) May 17 14:52:26 client-5 xinetd[3391]: START: shell pid=25253 from=192.168.4.5 May 17 14:52:26 client-5 rshd[25254]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 40: cause recovery in ptlrpc, ensure IO continues ============================== 14:52:26 \(1305669146\)");echo XXRETCODE:$?' May 17 14:52:26 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 40: cause recovery in ptlrpc, ensure IO continues ============================== 14:52:26 (1305669146) May 17 14:52:26 client-5 kernel: Lustre: DEBUG MARKER: multiop /mnt/lustre/f40 OS_c May 17 14:52:26 client-5 xinetd[3391]: EXIT: shell status=0 pid=25253 duration=0(sec) May 17 14:53:05 client-5 xinetd[3391]: START: shell pid=25487 from=192.168.4.5 May 17 14:53:05 client-5 rshd[25488]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:53:05 client-5 xinetd[3391]: EXIT: shell status=0 pid=25487 duration=0(sec) May 17 14:53:05 client-5 xinetd[3391]: START: shell pid=25659 from=192.168.4.5 May 17 14:53:05 client-5 rshd[25660]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 41: read from a valid osc while other oscs are invalid ========================= 14:53:05 \(1305669185\)");echo XXRETCODE:$?' May 17 14:53:05 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 41: read from a valid osc while other oscs are invalid ========================= 14:53:05 (1305669185) May 17 14:53:05 client-5 xinetd[3391]: EXIT: shell status=0 pid=25659 duration=0(sec) May 17 14:53:05 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 14:53:05 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 14:53:06 client-5 xinetd[3391]: START: shell pid=25874 from=192.168.4.5 May 17 14:53:06 client-5 rshd[25875]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:53:06 client-5 xinetd[3391]: EXIT: shell status=0 pid=25874 duration=0(sec) May 17 14:53:07 client-5 xinetd[3391]: START: shell pid=26046 from=192.168.4.5 May 17 14:53:07 client-5 rshd[26047]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 42: recovery after ost failure ================================================= 14:53:07 \(1305669187\)");echo XXRETCODE:$?' May 17 14:53:07 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 42: recovery after ost failure ================================================= 14:53:07 (1305669187) May 17 14:53:07 client-5 xinetd[3391]: EXIT: shell status=0 pid=26046 duration=0(sec) May 17 14:53:08 client-5 xinetd[3391]: START: shell pid=26102 from=192.168.4.5 May 17 14:53:08 client-5 rshd[26103]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 14:53:08 client-5 xinetd[3391]: EXIT: shell status=0 pid=26102 duration=0(sec) May 17 14:53:08 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-OST0000 May 17 14:53:09 client-5 kernel: Lustre: 26168:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 14:53:09 client-5 kernel: Lustre: 20198:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74c9ac37 to 0x62e5a8d74cc73f2 May 17 14:53:09 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 14:53:09 client-5 kernel: Lustre: Skipped 6 previous similar messages May 17 14:54:01 client-5 xinetd[3391]: START: shell pid=26345 from=192.168.4.5 May 17 14:54:01 client-5 rshd[26346]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl set_param debug=\"inode super warning dlmtrace error emerg ha rpctrace vfstrace config console\";");echo XXRETCODE:$?' May 17 14:54:01 client-5 xinetd[3391]: EXIT: shell status=0 pid=26345 duration=0(sec) May 17 14:54:02 client-5 xinetd[3391]: START: shell pid=26438 from=192.168.4.5 May 17 14:54:02 client-5 rshd[26439]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:54:02 client-5 xinetd[3391]: EXIT: shell status=0 pid=26438 duration=0(sec) May 17 14:54:02 client-5 xinetd[3391]: START: shell pid=26610 from=192.168.4.5 May 17 14:54:02 client-5 rshd[26611]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 43: mds osc import failure during recovery\; don\'t LBUG ========================= 14:54:02 \(1305669242\)");echo XXRETCODE:$?' May 17 14:54:02 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 43: mds osc import failure during recovery; don't LBUG ========================= 14:54:02 (1305669242) May 17 14:54:02 client-5 xinetd[3391]: EXIT: shell status=0 pid=26610 duration=0(sec) May 17 14:54:02 client-5 xinetd[3391]: START: shell pid=26686 from=192.168.4.5 May 17 14:54:02 client-5 rshd[26687]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 14:54:03 client-5 xinetd[3391]: EXIT: shell status=0 pid=26686 duration=1(sec) May 17 14:54:03 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 14:54:22 client-5 xinetd[3391]: START: shell pid=26876 from=192.168.4.5 May 17 14:54:22 client-5 rshd[26877]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 14:54:22 client-5 xinetd[3391]: EXIT: shell status=0 pid=26876 duration=0(sec) May 17 14:54:24 client-5 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail. May 17 14:54:24 client-5 kernel: LustreError: Skipped 6 previous similar messages May 17 14:54:24 client-5 kernel: Lustre: 20198:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74cc73f2 to 0x62e5a8d74ccc28f May 17 14:54:32 client-5 xinetd[3391]: START: shell pid=26979 from=192.168.4.5 May 17 14:54:32 client-5 rshd[26980]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 14:54:32 client-5 xinetd[3391]: EXIT: shell status=0 pid=26979 duration=0(sec) May 17 14:54:33 client-5 xinetd[3391]: START: shell pid=27151 from=192.168.4.5 May 17 14:54:33 client-5 rshd[27152]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 44a: race in target handle connect ============================================= 14:54:32 \(1305669272\)");echo XXRETCODE:$?' May 17 14:54:33 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 44a: race in target handle connect ============================================= 14:54:32 (1305669272) May 17 14:54:33 client-5 xinetd[3391]: EXIT: shell status=0 pid=27151 duration=0(sec) May 17 14:54:54 client-5 kernel: Lustre: 20199:0:(import.c:529:import_select_connection()) lustre-MDT0000-mdc-ffff8102c72f6400: tried all connections, increasing latency to 16s May 17 14:54:54 client-5 kernel: Lustre: 20199:0:(import.c:529:import_select_connection()) Skipped 9 previous similar messages May 17 14:54:54 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -16 May 17 14:54:54 client-5 kernel: LustreError: Skipped 36 previous similar messages May 17 14:56:00 client-5 kernel: Lustre: 20198:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369045627436840 sent from lustre-MDT0000-mdc-ffff8102c72f6400 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305669334] [real_sent 1305669334] [current 1305669360] [deadline 26s] [delay 0s] req@ffff8102c32f0800 x1369045627436840/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 368/392 e 0 to 1 dl 1305669360 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 14:56:00 client-5 kernel: Lustre: 20198:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 56 previous similar messages May 17 14:56:44 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff8102c72f6400: Connection restored to service lustre-MDT0000 using nid 192.168.4.128@o2ib. May 17 14:56:44 client-5 kernel: Lustre: Skipped 27 previous similar messages May 17 14:57:19 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff8102c72f6400: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 14:57:19 client-5 kernel: Lustre: Skipped 21 previous similar messages May 17 15:00:14 client-5 xinetd[3391]: START: shell pid=27503 from=192.168.4.5 May 17 15:00:14 client-5 rshd[27504]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 15:00:14 client-5 xinetd[3391]: EXIT: shell status=0 pid=27503 duration=0(sec) May 17 15:00:14 client-5 xinetd[3391]: START: shell pid=27674 from=192.168.4.5 May 17 15:00:14 client-5 rshd[27676]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 44b: race in target handle connect ============================================= 15:00:14 \(1305669614\)");echo XXRETCODE:$?' May 17 15:00:14 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 44b: race in target handle connect ============================================= 15:00:14 (1305669614) May 17 15:00:14 client-5 xinetd[3391]: EXIT: shell status=0 pid=27674 duration=0(sec) May 17 15:03:44 client-5 kernel: Lustre: 20199:0:(import.c:529:import_select_connection()) lustre-MDT0000-mdc-ffff8102c72f6400: tried all connections, increasing latency to 21s May 17 15:03:44 client-5 kernel: Lustre: 20199:0:(import.c:529:import_select_connection()) Skipped 25 previous similar messages May 17 15:05:14 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -114 May 17 15:05:14 client-5 kernel: LustreError: Skipped 30 previous similar messages May 17 15:06:40 client-5 kernel: Lustre: 20198:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369045627437891 sent from lustre-MDT0000-mdc-ffff8102c72f6400 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305669974] [real_sent 1305669974] [current 1305670000] [deadline 26s] [delay 0s] req@ffff81030ac9b800 x1369045627437891/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 368/392 e 0 to 1 dl 1305670000 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 15:06:40 client-5 kernel: Lustre: 20198:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 15 previous similar messages May 17 15:06:59 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff8102c72f6400: Connection restored to service lustre-MDT0000 using nid 192.168.4.128@o2ib. May 17 15:06:59 client-5 kernel: Lustre: Skipped 14 previous similar messages May 17 15:07:44 client-5 xinetd[3391]: START: shell pid=28001 from=192.168.4.5 May 17 15:07:44 client-5 rshd[28002]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 15:07:44 client-5 xinetd[3391]: EXIT: shell status=0 pid=28001 duration=0(sec) May 17 15:07:44 client-5 xinetd[3391]: START: shell pid=28173 from=192.168.4.5 May 17 15:07:44 client-5 rshd[28174]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 44c: race in target handle connect ============================================= 15:07:44 \(1305670064\)");echo XXRETCODE:$?' May 17 15:07:44 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 44c: race in target handle connect ============================================= 15:07:44 (1305670064) May 17 15:07:44 client-5 xinetd[3391]: EXIT: shell status=0 pid=28173 duration=0(sec) May 17 15:07:44 client-5 xinetd[3391]: START: shell pid=28223 from=192.168.4.5 May 17 15:07:44 client-5 rshd[28225]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 15:07:44 client-5 xinetd[3391]: EXIT: shell status=0 pid=28223 duration=0(sec) May 17 15:07:45 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 15:07:54 client-5 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail. May 17 15:07:54 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff8102c72f6400: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 15:07:54 client-5 kernel: Lustre: Skipped 14 previous similar messages May 17 15:07:54 client-5 kernel: Lustre: 20198:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74ccc28f to 0x62e5a8d74cccb09 May 17 15:07:54 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 15:07:54 client-5 kernel: Lustre: Skipped 1 previous similar message May 17 15:07:55 client-5 xinetd[3391]: START: shell pid=28377 from=192.168.4.5 May 17 15:07:55 client-5 rshd[28378]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 15:08:04 client-5 kernel: LustreError: 167-0: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. May 17 15:08:04 client-5 kernel: LustreError: 28410:0:(lmv_obd.c:1198:lmv_statfs()) can't stat MDS #0 (lustre-MDT0000-mdc-ffff8102c72f6400), error -4 May 17 15:08:04 client-5 kernel: LustreError: 28410:0:(llite_lib.c:1370:ll_statfs_internal()) md_statfs fails: rc = -4 May 17 15:08:04 client-5 xinetd[3391]: EXIT: shell status=0 pid=28377 duration=9(sec) May 17 15:08:05 client-5 xinetd[3391]: START: shell pid=28419 from=192.168.4.5 May 17 15:08:05 client-5 rshd[28421]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 15:08:05 client-5 xinetd[3391]: EXIT: shell status=0 pid=28419 duration=0(sec) May 17 15:08:24 client-5 xinetd[3391]: START: shell pid=28573 from=192.168.4.5 May 17 15:08:24 client-5 rshd[28574]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 15:08:26 client-5 kernel: Lustre: 20198:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74cccb09 to 0x62e5a8d74ccccd0 May 17 15:08:31 client-5 xinetd[3391]: EXIT: shell status=0 pid=28573 duration=7(sec) May 17 15:08:31 client-5 xinetd[3391]: START: shell pid=28667 from=192.168.4.5 May 17 15:08:31 client-5 rshd[28668]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 15:08:31 client-5 xinetd[3391]: EXIT: shell status=0 pid=28667 duration=0(sec) May 17 15:08:32 client-5 xinetd[3391]: START: shell pid=28838 from=192.168.4.5 May 17 15:08:32 client-5 rshd[28840]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 45: Handle failed close ======================================================== 15:08:32 \(1305670112\)");echo XXRETCODE:$?' May 17 15:08:32 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 45: Handle failed close ======================================================== 15:08:32 (1305670112) May 17 15:08:32 client-5 xinetd[3391]: EXIT: shell status=0 pid=28838 duration=0(sec) May 17 15:08:32 client-5 kernel: Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request May 17 15:08:32 client-5 kernel: LustreError: 28882:0:(file.c:155:ll_close_inode_openhandle()) inode 144115540800045057 mdc close failed: rc = -108 May 17 15:08:32 client-5 kernel: LustreError: 167-0: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. May 17 15:08:33 client-5 xinetd[3391]: START: shell pid=28945 from=192.168.4.5 May 17 15:08:33 client-5 rshd[28946]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 15:08:33 client-5 xinetd[3391]: EXIT: shell status=0 pid=28945 duration=0(sec) May 17 15:08:33 client-5 xinetd[3391]: START: shell pid=29117 from=192.168.4.5 May 17 15:08:33 client-5 rshd[29118]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 46: Don\'t leak file handle after open resend \(3325\) ============================ 15:08:33 \(1305670113\)");echo XXRETCODE:$?' May 17 15:08:33 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 46: Don't leak file handle after open resend (3325) ============================ 15:08:33 (1305670113) May 17 15:08:33 client-5 xinetd[3391]: EXIT: shell status=0 pid=29117 duration=0(sec) May 17 15:09:20 client-5 xinetd[3391]: START: shell pid=29296 from=192.168.4.5 May 17 15:09:20 client-5 rshd[29298]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 15:09:28 client-5 kernel: LustreError: 20198:0:(client.c:2570:ptlrpc_replay_interpret()) @@@ status -116, old was 0 req@ffff8102d671dc00 x1369045627438280/t227633266693(227633266693) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:23/10 lens 360/424 e 1 to 0 dl 1305670213 ref 2 fl Interpret:R/ffffffff/ffffffff rc -116/-1 May 17 15:09:28 client-5 xinetd[3391]: EXIT: shell status=0 pid=29296 duration=8(sec) May 17 15:09:28 client-5 xinetd[3391]: START: shell pid=29389 from=192.168.4.5 May 17 15:09:28 client-5 rshd[29390]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 15:09:28 client-5 xinetd[3391]: EXIT: shell status=0 pid=29389 duration=0(sec) May 17 15:09:29 client-5 xinetd[3391]: START: shell pid=29561 from=192.168.4.5 May 17 15:09:29 client-5 rshd[29562]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 47: MDS-\>OSC failure during precreate cleanup \(2824\) =========================== 15:09:29 \(1305670169\)");echo XXRETCODE:$?' May 17 15:09:29 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 47: MDS->OSC failure during precreate cleanup (2824) =========================== 15:09:29 (1305670169) May 17 15:09:29 client-5 xinetd[3391]: EXIT: shell status=0 pid=29561 duration=0(sec) May 17 15:09:33 client-5 kernel: Lustre: 20198:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74ccccd0 to 0x62e5a8d74ccd161 May 17 15:09:43 client-5 xinetd[3391]: START: shell pid=29743 from=192.168.4.5 May 17 15:09:43 client-5 rshd[29745]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 15:09:48 client-5 xinetd[3391]: EXIT: shell status=0 pid=29743 duration=5(sec) May 17 15:10:50 client-5 xinetd[3391]: START: shell pid=29859 from=192.168.4.5 May 17 15:10:50 client-5 rshd[29860]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 15:10:50 client-5 xinetd[3391]: EXIT: shell status=0 pid=29859 duration=0(sec) May 17 15:10:50 client-5 xinetd[3391]: START: shell pid=30031 from=192.168.4.5 May 17 15:10:50 client-5 rshd[30032]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 48: MDS-\>OSC failure during precreate cleanup \(2824\) =========================== 15:10:50 \(1305670250\)");echo XXRETCODE:$?' May 17 15:10:50 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 48: MDS->OSC failure during precreate cleanup (2824) =========================== 15:10:50 (1305670250) May 17 15:10:50 client-5 xinetd[3391]: EXIT: shell status=0 pid=30031 duration=0(sec) May 17 15:10:50 client-5 xinetd[3391]: START: shell pid=30107 from=192.168.4.5 May 17 15:10:50 client-5 rshd[30108]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 15:10:50 client-5 xinetd[3391]: EXIT: shell status=0 pid=30107 duration=0(sec) May 17 15:10:51 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 15:11:10 client-5 kernel: Lustre: 20198:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74ccd161 to 0x62e5a8d74ccdb78 May 17 15:11:13 client-5 xinetd[3391]: START: shell pid=30357 from=192.168.4.5 May 17 15:11:13 client-5 rshd[30359]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 15:11:14 client-5 xinetd[3391]: EXIT: shell status=0 pid=30357 duration=1(sec) May 17 15:11:14 client-5 xinetd[3391]: START: shell pid=30530 from=192.168.4.5 May 17 15:11:14 client-5 rshd[30531]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 50: Double OSC recovery, don\'t LASSERT \(3812\) ================================== 15:11:14 \(1305670274\)");echo XXRETCODE:$?' May 17 15:11:14 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 50: Double OSC recovery, don't LASSERT (3812) ================================== 15:11:14 (1305670274) May 17 15:11:14 client-5 xinetd[3391]: EXIT: shell status=0 pid=30530 duration=0(sec) May 17 15:11:20 client-5 xinetd[3391]: START: shell pid=30720 from=192.168.4.5 May 17 15:11:20 client-5 rshd[30722]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 15:11:20 client-5 xinetd[3391]: EXIT: shell status=0 pid=30720 duration=0(sec) May 17 15:11:20 client-5 xinetd[3391]: START: shell pid=30893 from=192.168.4.5 May 17 15:11:20 client-5 rshd[30894]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 52: time out lock replay \(3764\) ================================================ 15:11:20 \(1305670280\)");echo XXRETCODE:$?' May 17 15:11:20 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 52: time out lock replay (3764) ================================================ 15:11:20 (1305670280) May 17 15:11:20 client-5 xinetd[3391]: EXIT: shell status=0 pid=30893 duration=0(sec) May 17 15:11:20 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc start May 17 15:11:20 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc stop May 17 15:11:20 client-5 xinetd[3391]: START: shell pid=30955 from=192.168.4.5 May 17 15:11:20 client-5 rshd[30956]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 15:11:20 client-5 xinetd[3391]: EXIT: shell status=0 pid=30955 duration=0(sec) May 17 15:11:21 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 15:11:40 client-5 kernel: Lustre: 20198:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74ccdb78 to 0x62e5a8d74ccea3c May 17 15:11:40 client-5 xinetd[3391]: START: shell pid=31149 from=192.168.4.5 May 17 15:11:40 client-5 rshd[31150]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 15:11:40 client-5 kernel: LustreError: 20198:0:(client.c:2570:ptlrpc_replay_interpret()) @@@ status -116, old was 0 req@ffff810313dda800 x1369045627438860/t236223201363(236223201363) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:23/10 lens 360/424 e 0 to 0 dl 1305670327 ref 2 fl Interpret:R/ffffffff/ffffffff rc -116/-1 May 17 15:11:40 client-5 xinetd[3391]: EXIT: shell status=0 pid=31149 duration=0(sec) May 17 15:11:40 client-5 xinetd[3391]: START: shell pid=31248 from=192.168.4.5 May 17 15:11:40 client-5 rshd[31249]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 15:11:41 client-5 xinetd[3391]: EXIT: shell status=0 pid=31248 duration=1(sec) May 17 15:11:41 client-5 xinetd[3391]: START: shell pid=31420 from=192.168.4.5 May 17 15:11:41 client-5 rshd[31421]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 53a: \|X\| close request while two MDC requests in flight ======================== 15:11:41 \(1305670301\)");echo XXRETCODE:$?' May 17 15:11:41 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 53a: |X| close request while two MDC requests in flight ======================== 15:11:41 (1305670301) May 17 15:11:41 client-5 xinetd[3391]: EXIT: shell status=0 pid=31420 duration=0(sec) May 17 15:11:41 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc start May 17 15:11:41 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc stop May 17 15:11:42 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc start May 17 15:11:42 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc stop May 17 15:11:43 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 15:12:02 client-5 xinetd[3391]: START: shell pid=31646 from=192.168.4.5 May 17 15:12:02 client-5 rshd[31648]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 15:12:02 client-5 xinetd[3391]: EXIT: shell status=0 pid=31646 duration=0(sec) May 17 15:12:02 client-5 xinetd[3391]: START: shell pid=31739 from=192.168.4.5 May 17 15:12:02 client-5 rshd[31741]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 15:12:02 client-5 xinetd[3391]: EXIT: shell status=0 pid=31739 duration=0(sec) May 17 15:12:02 client-5 xinetd[3391]: START: shell pid=31912 from=192.168.4.5 May 17 15:12:02 client-5 rshd[31913]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 53b: \|X\| open request while two MDC requests in flight ========================= 15:12:02 \(1305670322\)");echo XXRETCODE:$?' May 17 15:12:02 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 53b: |X| open request while two MDC requests in flight ========================= 15:12:02 (1305670322) May 17 15:12:02 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc start May 17 15:12:02 client-5 xinetd[3391]: EXIT: shell status=0 pid=31912 duration=0(sec) May 17 15:12:02 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc stop May 17 15:12:04 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc start May 17 15:12:04 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc stop May 17 15:12:04 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 15:12:05 client-5 kernel: Lustre: 20198:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74ccea3c to 0x62e5a8d74ccefbb May 17 15:12:23 client-5 xinetd[3391]: START: shell pid=32146 from=192.168.4.5 May 17 15:12:23 client-5 rshd[32147]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 15:12:23 client-5 kernel: LustreError: 20198:0:(client.c:2570:ptlrpc_replay_interpret()) @@@ status -116, old was 0 req@ffff8102f47e4000 x1369045627439010/t244813135886(244813135886) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:23/10 lens 360/424 e 0 to 0 dl 1305670370 ref 2 fl Interpret:R/ffffffff/ffffffff rc -116/-1 May 17 15:12:23 client-5 xinetd[3391]: EXIT: shell status=0 pid=32146 duration=0(sec) May 17 15:12:24 client-5 xinetd[3391]: START: shell pid=32237 from=192.168.4.5 May 17 15:12:24 client-5 rshd[32240]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 15:12:24 client-5 xinetd[3391]: EXIT: shell status=0 pid=32237 duration=0(sec) May 17 15:12:24 client-5 xinetd[3391]: START: shell pid=32411 from=192.168.4.5 May 17 15:12:24 client-5 rshd[32412]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 53c: \|X\| open request and close request while two MDC requests in flight ======= 15:12:24 \(1305670344\)");echo XXRETCODE:$?' May 17 15:12:24 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 53c: |X| open request and close request while two MDC requests in flight ======= 15:12:24 (1305670344) May 17 15:12:24 client-5 xinetd[3391]: EXIT: shell status=0 pid=32411 duration=0(sec) May 17 15:12:24 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc start May 17 15:12:24 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc stop May 17 15:12:25 client-5 kernel: LustreError: 30298:0:(client.c:1057:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@ffff81031ecba800 x1369045627439088/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:/ffffffff/ffffffff rc 0/-1 May 17 15:12:25 client-5 kernel: LustreError: 30298:0:(client.c:1057:ptlrpc_import_delay_req()) Skipped 185 previous similar messages May 17 15:12:25 client-5 kernel: Lustre: 20198:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74ccefbb to 0x62e5a8d74ccf381 May 17 15:12:26 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc start May 17 15:12:26 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc stop May 17 15:12:26 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 15:12:45 client-5 kernel: Lustre: 20198:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74ccf381 to 0x62e5a8d74ccf499 May 17 15:12:51 client-5 xinetd[3391]: START: shell pid=32703 from=192.168.4.5 May 17 15:12:51 client-5 rshd[32704]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 15:12:51 client-5 xinetd[3391]: EXIT: shell status=0 pid=32703 duration=0(sec) May 17 15:12:52 client-5 xinetd[3391]: START: shell pid=417 from=192.168.4.5 May 17 15:12:52 client-5 rshd[418]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 53d: \|X\| close reply while two MDC requests in flight ========================== 15:12:51 \(1305670371\)");echo XXRETCODE:$?' May 17 15:12:52 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 53d: |X| close reply while two MDC requests in flight ========================== 15:12:51 (1305670371) May 17 15:12:52 client-5 xinetd[3391]: EXIT: shell status=0 pid=417 duration=0(sec) May 17 15:12:52 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc start May 17 15:12:52 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc stop May 17 15:12:53 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc start May 17 15:12:53 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc stop May 17 15:13:12 client-5 kernel: Lustre: 20198:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74ccf499 to 0x62e5a8d74ccf8e4 May 17 15:13:12 client-5 xinetd[3391]: START: shell pid=624 from=192.168.4.5 May 17 15:13:12 client-5 rshd[625]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 15:13:12 client-5 xinetd[3391]: EXIT: shell status=0 pid=624 duration=0(sec) May 17 15:13:12 client-5 xinetd[3391]: START: shell pid=720 from=192.168.4.5 May 17 15:13:12 client-5 rshd[721]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 15:13:12 client-5 xinetd[3391]: EXIT: shell status=0 pid=720 duration=0(sec) May 17 15:13:12 client-5 xinetd[3391]: START: shell pid=891 from=192.168.4.5 May 17 15:13:13 client-5 rshd[893]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 53e: \|X\| open reply while two MDC requests in flight =========================== 15:13:12 \(1305670392\)");echo XXRETCODE:$?' May 17 15:13:13 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 53e: |X| open reply while two MDC requests in flight =========================== 15:13:12 (1305670392) May 17 15:13:13 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc start May 17 15:13:13 client-5 xinetd[3391]: EXIT: shell status=0 pid=891 duration=1(sec) May 17 15:13:13 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc stop May 17 15:13:14 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc start May 17 15:13:14 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc stop May 17 15:13:14 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 15:13:32 client-5 kernel: LustreError: 30298:0:(client.c:1057:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@ffff810319cc4400 x1369045627439290/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:/ffffffff/ffffffff rc 0/-1 May 17 15:13:32 client-5 kernel: LustreError: 30298:0:(client.c:1057:ptlrpc_import_delay_req()) Skipped 3 previous similar messages May 17 15:13:32 client-5 kernel: Lustre: 20198:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74ccf8e4 to 0x62e5a8d74ccfc4f May 17 15:13:33 client-5 xinetd[3391]: START: shell pid=1120 from=192.168.4.5 May 17 15:13:33 client-5 rshd[1121]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 15:13:34 client-5 kernel: LustreError: 20198:0:(client.c:2570:ptlrpc_replay_interpret()) @@@ status -116, old was 0 req@ffff81031c46fc00 x1369045627439262/t257698037775(257698037775) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:23/10 lens 360/424 e 0 to 0 dl 1305670440 ref 2 fl Interpret:R/ffffffff/ffffffff rc -116/-1 May 17 15:13:34 client-5 xinetd[3391]: EXIT: shell status=0 pid=1120 duration=1(sec) May 17 15:13:34 client-5 xinetd[3391]: START: shell pid=1212 from=192.168.4.5 May 17 15:13:34 client-5 rshd[1214]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 15:13:34 client-5 xinetd[3391]: EXIT: shell status=0 pid=1212 duration=0(sec) May 17 15:13:35 client-5 xinetd[3391]: START: shell pid=1385 from=192.168.4.5 May 17 15:13:35 client-5 rshd[1386]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 53f: \|X\| open reply and close reply while two MDC requests in flight =========== 15:13:35 \(1305670415\)");echo XXRETCODE:$?' May 17 15:13:35 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 53f: |X| open reply and close reply while two MDC requests in flight =========== 15:13:35 (1305670415) May 17 15:13:35 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc start May 17 15:13:35 client-5 xinetd[3391]: EXIT: shell status=0 pid=1385 duration=0(sec) May 17 15:13:35 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc stop May 17 15:13:36 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc start May 17 15:13:36 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc stop May 17 15:13:36 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 15:13:57 client-5 kernel: Lustre: 20198:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74ccfc4f to 0x62e5a8d74cd00c4 May 17 15:13:59 client-5 xinetd[3391]: START: shell pid=1682 from=192.168.4.5 May 17 15:13:59 client-5 rshd[1683]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 15:13:59 client-5 xinetd[3391]: EXIT: shell status=0 pid=1682 duration=0(sec) May 17 15:13:59 client-5 xinetd[3391]: START: shell pid=1856 from=192.168.4.5 May 17 15:13:59 client-5 rshd[1857]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 53g: \|X\| drop open reply and close request while close and open are both in flight ====================================================================================================== 15:13:59 \(1305670439\)");echo XXRETCODE:$?' May 17 15:13:59 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 53g: |X| drop open reply and close request while close and open are both in flight ====================================================================================================== 15:13:59 (1305670439) May 17 15:13:59 client-5 xinetd[3391]: EXIT: shell status=0 pid=1856 duration=0(sec) May 17 15:13:59 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc start May 17 15:13:59 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc stop May 17 15:14:01 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc start May 17 15:14:01 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc stop May 17 15:14:01 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 15:14:18 client-5 kernel: Lustre: 20199:0:(import.c:529:import_select_connection()) lustre-MDT0000-mdc-ffff8102c72f6400: tried all connections, increasing latency to 6s May 17 15:14:18 client-5 kernel: Lustre: 20199:0:(import.c:529:import_select_connection()) Skipped 29 previous similar messages May 17 15:14:24 client-5 kernel: Lustre: 20198:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74cd00c4 to 0x62e5a8d74cd04d7 May 17 15:14:36 client-5 xinetd[3391]: START: shell pid=2164 from=192.168.4.5 May 17 15:14:36 client-5 rshd[2167]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 15:14:36 client-5 xinetd[3391]: EXIT: shell status=0 pid=2164 duration=0(sec) May 17 15:14:36 client-5 xinetd[3391]: START: shell pid=2350 from=192.168.4.5 May 17 15:14:36 client-5 rshd[2351]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 53h: \|X\| open request and close reply while two MDC requests in flight ========= 15:14:36 \(1305670476\)");echo XXRETCODE:$?' May 17 15:14:36 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 53h: |X| open request and close reply while two MDC requests in flight ========= 15:14:36 (1305670476) May 17 15:14:36 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc start May 17 15:14:36 client-5 xinetd[3391]: EXIT: shell status=0 pid=2350 duration=0(sec) May 17 15:14:36 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc stop May 17 15:14:37 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc start May 17 15:14:37 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc stop May 17 15:14:39 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 15:14:59 client-5 kernel: Lustre: 20198:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74cd04d7 to 0x62e5a8d74cd08d5 May 17 15:15:17 client-5 xinetd[3391]: START: shell pid=2663 from=192.168.4.5 May 17 15:15:17 client-5 rshd[2664]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 15:15:17 client-5 xinetd[3391]: EXIT: shell status=0 pid=2663 duration=0(sec) May 17 15:15:17 client-5 xinetd[3391]: START: shell pid=2835 from=192.168.4.5 May 17 15:15:17 client-5 rshd[2836]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 55: let MDS_CHECK_RESENT return the original return code instead of 0 ========== 15:15:17 \(1305670517\)");echo XXRETCODE:$?' May 17 15:15:17 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 55: let MDS_CHECK_RESENT return the original return code instead of 0 ========== 15:15:17 (1305670517) May 17 15:15:17 client-5 xinetd[3391]: EXIT: shell status=0 pid=2835 duration=0(sec) May 17 15:15:58 client-5 xinetd[3391]: START: shell pid=2945 from=192.168.4.5 May 17 15:15:58 client-5 rshd[2946]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 15:15:58 client-5 xinetd[3391]: EXIT: shell status=0 pid=2945 duration=0(sec) May 17 15:15:59 client-5 xinetd[3391]: START: shell pid=3133 from=192.168.4.5 May 17 15:15:59 client-5 rshd[3134]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 56: don\'t replay a symlink open request \(3440\) ================================= 15:15:59 \(1305670559\)");echo XXRETCODE:$?' May 17 15:15:59 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 56: don't replay a symlink open request (3440) ================================= 15:15:59 (1305670559) May 17 15:15:59 client-5 xinetd[3391]: EXIT: shell status=0 pid=3133 duration=0(sec) May 17 15:15:59 client-5 xinetd[3391]: START: shell pid=3189 from=192.168.4.5 May 17 15:15:59 client-5 rshd[3191]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 15:15:59 client-5 xinetd[3391]: EXIT: shell status=0 pid=3189 duration=0(sec) May 17 15:15:59 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 15:16:18 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The obd_ping operation failed with -107 May 17 15:16:18 client-5 kernel: LustreError: Skipped 33 previous similar messages May 17 15:16:18 client-5 kernel: Lustre: 20198:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74cd08d5 to 0x62e5a8d74cd0d04 May 17 15:16:19 client-5 xinetd[3391]: START: shell pid=3400 from=192.168.4.5 May 17 15:16:19 client-5 rshd[3401]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 15:16:19 client-5 xinetd[3391]: EXIT: shell status=0 pid=3400 duration=0(sec) May 17 15:16:29 client-5 xinetd[3391]: START: shell pid=3500 from=192.168.4.5 May 17 15:16:29 client-5 rshd[3501]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 15:16:29 client-5 xinetd[3391]: EXIT: shell status=0 pid=3500 duration=0(sec) May 17 15:16:29 client-5 xinetd[3391]: START: shell pid=3686 from=192.168.4.5 May 17 15:16:29 client-5 rshd[3687]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 57: test recovery from llog for setattr op ===================================== 15:16:29 \(1305670589\)");echo XXRETCODE:$?' May 17 15:16:29 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 57: test recovery from llog for setattr op ===================================== 15:16:29 (1305670589) May 17 15:16:29 client-5 xinetd[3391]: EXIT: shell status=0 pid=3686 duration=0(sec) May 17 15:16:29 client-5 xinetd[3391]: START: shell pid=3746 from=192.168.4.5 May 17 15:16:30 client-5 rshd[3747]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 15:16:30 client-5 xinetd[3391]: EXIT: shell status=0 pid=3746 duration=1(sec) May 17 15:16:30 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 15:16:48 client-5 kernel: Lustre: 20198:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74cd0d04 to 0x62e5a8d74cd0f96 May 17 15:16:48 client-5 kernel: LustreError: 20198:0:(client.c:2570:ptlrpc_replay_interpret()) @@@ status -116, old was 0 req@ffff810309cc0400 x1369045627439757/t279172874243(279172874243) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:23/10 lens 360/424 e 0 to 0 dl 1305670630 ref 2 fl Interpret:R/ffffffff/ffffffff rc -116/-1 May 17 15:16:49 client-5 xinetd[3391]: START: shell pid=3929 from=192.168.4.5 May 17 15:16:49 client-5 rshd[3930]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 15:16:49 client-5 xinetd[3391]: EXIT: shell status=0 pid=3929 duration=0(sec) May 17 15:16:49 client-5 kernel: Lustre: 20197:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369045627439705 sent from lustre-MDT0000-mdc-ffff8102c72f6400 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305670568] [real_sent 1305670568] [current 1305670609] [deadline 41s] [delay 0s] req@ffff810306d87400 x1369045627439705/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 192/192 e 0 to 1 dl 1305670609 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 15:16:49 client-5 kernel: Lustre: 20197:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 45 previous similar messages May 17 15:16:50 client-5 xinetd[3391]: START: shell pid=4028 from=192.168.4.5 May 17 15:16:50 client-5 rshd[4031]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 15:16:50 client-5 xinetd[3391]: EXIT: shell status=0 pid=4028 duration=0(sec) May 17 15:16:51 client-5 xinetd[3391]: START: shell pid=4201 from=192.168.4.5 May 17 15:16:51 client-5 rshd[4203]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 58a: test recovery from llog for setattr op \(test llog_gen_rec\) ================ 15:16:51 \(1305670611\)");echo XXRETCODE:$?' May 17 15:16:51 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 58a: test recovery from llog for setattr op (test llog_gen_rec) ================ 15:16:51 (1305670611) May 17 15:16:51 client-5 xinetd[3391]: EXIT: shell status=0 pid=4201 duration=0(sec) May 17 15:16:53 client-5 xinetd[3391]: START: shell pid=4265 from=192.168.4.5 May 17 15:16:53 client-5 rshd[4266]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 15:16:53 client-5 xinetd[3391]: EXIT: shell status=0 pid=4265 duration=0(sec) May 17 15:16:54 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 15:17:13 client-5 xinetd[3391]: START: shell pid=4447 from=192.168.4.5 May 17 15:17:13 client-5 rshd[4448]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 15:17:13 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff8102c72f6400: Connection restored to service lustre-MDT0000 using nid 192.168.4.128@o2ib. May 17 15:17:13 client-5 kernel: Lustre: Skipped 36 previous similar messages May 17 15:17:13 client-5 xinetd[3391]: EXIT: shell status=0 pid=4447 duration=0(sec) May 17 15:17:13 client-5 kernel: Lustre: 20198:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74cd0f96 to 0x62e5a8d74ce246a May 17 15:17:20 client-5 xinetd[3391]: START: shell pid=4552 from=192.168.4.5 May 17 15:17:20 client-5 rshd[4553]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 15:17:20 client-5 xinetd[3391]: EXIT: shell status=0 pid=4552 duration=0(sec) May 17 15:17:20 client-5 xinetd[3391]: START: shell pid=4724 from=192.168.4.5 May 17 15:17:20 client-5 rshd[4725]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 58b: test replay of setxattr op ================================================ 15:17:20 \(1305670640\)");echo XXRETCODE:$?' May 17 15:17:20 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 58b: test replay of setxattr op ================================================ 15:17:20 (1305670640) May 17 15:17:20 client-5 xinetd[3391]: EXIT: shell status=0 pid=4724 duration=0(sec) May 17 15:17:20 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 15:17:20 client-5 kernel: Lustre: Increasing default stripe size to min 1048576 May 17 15:17:20 client-5 kernel: Lustre: 4768:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff81030f0a3800->192.168.4.128@o2ib netid 50000: select flavor null May 17 15:17:20 client-5 kernel: Lustre: 4768:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 5 previous similar messages May 17 15:17:21 client-5 kernel: Lustre: Client lustre-client has started May 17 15:17:21 client-5 kernel: Lustre: 4775:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 15:17:21 client-5 xinetd[3391]: START: shell pid=4796 from=192.168.4.5 May 17 15:17:21 client-5 rshd[4797]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 15:17:21 client-5 xinetd[3391]: EXIT: shell status=0 pid=4796 duration=0(sec) May 17 15:17:21 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 15:17:40 client-5 xinetd[3391]: START: shell pid=4983 from=192.168.4.5 May 17 15:17:40 client-5 rshd[4984]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 15:17:40 client-5 kernel: Lustre: 20198:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74ce246a to 0x62e5a8d74d0066e May 17 15:17:41 client-5 xinetd[3391]: EXIT: shell status=0 pid=4983 duration=1(sec) May 17 15:17:41 client-5 kernel: LustreError: 5039:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 15:17:41 client-5 kernel: LustreError: 5039:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 15:17:41 client-5 kernel: Lustre: client ffff81030f0a3800 umount complete May 17 15:17:41 client-5 xinetd[3391]: START: shell pid=5096 from=192.168.4.5 May 17 15:17:41 client-5 rshd[5097]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 15:17:41 client-5 xinetd[3391]: EXIT: shell status=0 pid=5096 duration=0(sec) May 17 15:17:41 client-5 xinetd[3391]: START: shell pid=5268 from=192.168.4.5 May 17 15:17:41 client-5 rshd[5269]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 58c: resend\/reconstruct setxattr op ============================================ 15:17:41 \(1305670661\)");echo XXRETCODE:$?' May 17 15:17:41 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 58c: resend/reconstruct setxattr op ============================================ 15:17:41 (1305670661) May 17 15:17:41 client-5 xinetd[3391]: EXIT: shell status=0 pid=5268 duration=0(sec) May 17 15:17:41 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 15:17:41 client-5 kernel: Lustre: Increasing default stripe size to min 1048576 May 17 15:17:41 client-5 kernel: Lustre: 5312:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff8102fde10c00->192.168.4.128@o2ib netid 50000: select flavor null May 17 15:17:41 client-5 kernel: Lustre: 5312:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 6 previous similar messages May 17 15:17:42 client-5 kernel: Lustre: Client lustre-client has started May 17 15:17:42 client-5 kernel: Lustre: 5319:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 15:17:42 client-5 kernel: Lustre: 5319:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message May 17 15:18:23 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff8102c72f6400: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 15:18:23 client-5 kernel: Lustre: Skipped 22 previous similar messages May 17 15:19:04 client-5 kernel: LustreError: 5397:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 15:19:04 client-5 kernel: LustreError: 5397:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 15:19:04 client-5 kernel: Lustre: client ffff8102fde10c00 umount complete May 17 15:19:04 client-5 xinetd[3391]: START: shell pid=5452 from=192.168.4.5 May 17 15:19:04 client-5 rshd[5455]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 15:19:05 client-5 xinetd[3391]: EXIT: shell status=0 pid=5452 duration=1(sec) May 17 15:19:05 client-5 xinetd[3391]: START: shell pid=5626 from=192.168.4.5 May 17 15:19:05 client-5 rshd[5627]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 59: test log_commit_thread vs filter_destroy race ============================== 15:19:05 \(1305670745\)");echo XXRETCODE:$?' May 17 15:19:05 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 59: test log_commit_thread vs filter_destroy race ============================== 15:19:05 (1305670745) May 17 15:19:05 client-5 xinetd[3391]: EXIT: shell status=0 pid=5626 duration=0(sec) May 17 15:19:19 client-5 xinetd[3391]: START: shell pid=5816 from=192.168.4.5 May 17 15:19:19 client-5 rshd[5817]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 15:19:24 client-5 xinetd[3391]: EXIT: shell status=0 pid=5816 duration=5(sec) May 17 15:19:36 client-5 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail. May 17 15:19:36 client-5 kernel: LustreError: Skipped 16 previous similar messages May 17 15:19:43 client-5 xinetd[3391]: START: shell pid=5968 from=192.168.4.5 May 17 15:19:43 client-5 rshd[5969]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 15:19:43 client-5 xinetd[3391]: EXIT: shell status=0 pid=5968 duration=0(sec) May 17 15:19:46 client-5 kernel: Lustre: 20198:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74d0066e to 0x62e5a8d74d0479e May 17 15:19:46 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 15:19:46 client-5 kernel: Lustre: Skipped 16 previous similar messages May 17 15:20:03 client-5 xinetd[3391]: START: shell pid=6075 from=192.168.4.5 May 17 15:20:04 client-5 rshd[6077]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 15:20:04 client-5 xinetd[3391]: EXIT: shell status=0 pid=6075 duration=1(sec) May 17 15:20:04 client-5 xinetd[3391]: START: shell pid=6248 from=192.168.4.5 May 17 15:20:04 client-5 rshd[6249]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 60: test llog post recovery init vs llog unlink ================================ 15:20:04 \(1305670804\)");echo XXRETCODE:$?' May 17 15:20:04 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 60: test llog post recovery init vs llog unlink ================================ 15:20:04 (1305670804) May 17 15:20:04 client-5 xinetd[3391]: EXIT: shell status=0 pid=6248 duration=0(sec) May 17 15:20:04 client-5 xinetd[3391]: START: shell pid=6300 from=192.168.4.5 May 17 15:20:04 client-5 rshd[6302]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 15:20:04 client-5 xinetd[3391]: EXIT: shell status=0 pid=6300 duration=0(sec) May 17 15:20:05 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 15:20:24 client-5 xinetd[3391]: START: shell pid=6484 from=192.168.4.5 May 17 15:20:24 client-5 rshd[6485]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 15:20:24 client-5 xinetd[3391]: EXIT: shell status=0 pid=6484 duration=0(sec) May 17 15:20:24 client-5 xinetd[3391]: START: shell pid=6577 from=192.168.4.5 May 17 15:20:24 client-5 rshd[6579]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 15:20:24 client-5 xinetd[3391]: EXIT: shell status=0 pid=6577 duration=0(sec) May 17 15:20:25 client-5 xinetd[3391]: START: shell pid=6750 from=192.168.4.5 May 17 15:20:25 client-5 rshd[6751]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 61a: test race llog recovery vs llog cleanup =================================== 15:20:25 \(1305670825\)");echo XXRETCODE:$?' May 17 15:20:25 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 61a: test race llog recovery vs llog cleanup =================================== 15:20:25 (1305670825) May 17 15:20:25 client-5 xinetd[3391]: EXIT: shell status=0 pid=6750 duration=0(sec) May 17 15:20:26 client-5 xinetd[3391]: START: shell pid=6828 from=192.168.4.5 May 17 15:20:26 client-5 rshd[6829]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 15:20:26 client-5 xinetd[3391]: EXIT: shell status=0 pid=6828 duration=0(sec) May 17 15:20:26 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-OST0000 May 17 15:20:33 client-5 kernel: Lustre: 20198:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74d0479e to 0x62e5a8d74d18116 May 17 15:21:26 client-5 xinetd[3391]: START: shell pid=7167 from=192.168.4.5 May 17 15:21:26 client-5 rshd[7168]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 15:21:33 client-5 xinetd[3391]: EXIT: shell status=0 pid=7167 duration=7(sec) May 17 15:22:03 client-5 xinetd[3391]: START: shell pid=7294 from=192.168.4.5 May 17 15:22:03 client-5 rshd[7297]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 15:22:03 client-5 xinetd[3391]: EXIT: shell status=0 pid=7294 duration=0(sec) May 17 15:22:03 client-5 xinetd[3391]: START: shell pid=7468 from=192.168.4.5 May 17 15:22:03 client-5 rshd[7469]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 61b: test race mds llog sync vs llog cleanup =================================== 15:22:03 \(1305670923\)");echo XXRETCODE:$?' May 17 15:22:03 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 61b: test race mds llog sync vs llog cleanup =================================== 15:22:03 (1305670923) May 17 15:22:03 client-5 xinetd[3391]: EXIT: shell status=0 pid=7468 duration=0(sec) May 17 15:22:23 client-5 kernel: Lustre: 20198:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74d18116 to 0x62e5a8d74d182dd May 17 15:23:43 client-5 xinetd[3391]: START: shell pid=7754 from=192.168.4.5 May 17 15:23:43 client-5 rshd[7755]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 15:23:44 client-5 kernel: Lustre: 20198:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74d182dd to 0x62e5a8d74d18465 May 17 15:24:09 client-5 xinetd[3391]: EXIT: shell status=0 pid=7754 duration=26(sec) May 17 15:24:09 client-5 xinetd[3391]: START: shell pid=7854 from=192.168.4.5 May 17 15:24:09 client-5 rshd[7855]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 15:24:09 client-5 xinetd[3391]: EXIT: shell status=0 pid=7854 duration=0(sec) May 17 15:24:09 client-5 xinetd[3391]: START: shell pid=8025 from=192.168.4.5 May 17 15:24:09 client-5 rshd[8027]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 61c: test race mds llog sync vs llog cleanup =================================== 15:24:09 \(1305671049\)");echo XXRETCODE:$?' May 17 15:24:10 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 61c: test race mds llog sync vs llog cleanup =================================== 15:24:09 (1305671049) May 17 15:24:10 client-5 xinetd[3391]: EXIT: shell status=0 pid=8025 duration=1(sec) May 17 15:24:29 client-5 kernel: Lustre: 20199:0:(import.c:529:import_select_connection()) lustre-OST0000-osc-ffff8102c72f6400: tried all connections, increasing latency to 21s May 17 15:24:29 client-5 kernel: Lustre: 20199:0:(import.c:529:import_select_connection()) Skipped 15 previous similar messages May 17 15:24:45 client-5 xinetd[3391]: START: shell pid=8243 from=192.168.4.5 May 17 15:24:45 client-5 rshd[8244]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 15:24:45 client-5 xinetd[3391]: EXIT: shell status=0 pid=8243 duration=0(sec) May 17 15:24:45 client-5 xinetd[3391]: START: shell pid=8368 from=192.168.4.5 May 17 15:24:45 client-5 rshd[8370]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 15:24:46 client-5 xinetd[3391]: EXIT: shell status=0 pid=8368 duration=1(sec) May 17 15:24:46 client-5 xinetd[3391]: START: shell pid=8532 from=192.168.4.5 May 17 15:24:46 client-5 rshd[8533]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark SKIP: replay-single test_61d skipping ALWAYS excluded test 61d");echo XXRETCODE:$?' May 17 15:24:46 client-5 kernel: Lustre: DEBUG MARKER: SKIP: replay-single test_61d skipping ALWAYS excluded test 61d May 17 15:24:46 client-5 xinetd[3391]: EXIT: shell status=0 pid=8532 duration=0(sec) May 17 15:24:46 client-5 xinetd[3391]: START: shell pid=8635 from=192.168.4.5 May 17 15:24:46 client-5 rshd[8638]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 62: don\'t mis-drop resent replay =============================================== 15:24:46 \(1305671086\)");echo XXRETCODE:$?' May 17 15:24:46 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 62: don't mis-drop resent replay =============================================== 15:24:46 (1305671086) May 17 15:24:46 client-5 xinetd[3391]: EXIT: shell status=0 pid=8635 duration=0(sec) May 17 15:24:46 client-5 xinetd[3391]: START: shell pid=8689 from=192.168.4.5 May 17 15:24:46 client-5 rshd[8690]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 15:24:46 client-5 xinetd[3391]: EXIT: shell status=0 pid=8689 duration=0(sec) May 17 15:24:47 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 15:25:04 client-5 kernel: Lustre: 20198:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74d18465 to 0x62e5a8d74d1895f May 17 15:25:06 client-5 xinetd[3391]: START: shell pid=8886 from=192.168.4.5 May 17 15:25:06 client-5 rshd[8887]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 15:25:57 client-5 kernel: LustreError: 20198:0:(client.c:2515:ptlrpc_replay_interpret()) request replay timed out, restarting recovery May 17 15:25:57 client-5 xinetd[3391]: EXIT: shell status=0 pid=8886 duration=51(sec) May 17 15:25:58 client-5 xinetd[3391]: START: shell pid=8986 from=192.168.4.5 May 17 15:25:58 client-5 rshd[8988]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 15:25:58 client-5 xinetd[3391]: EXIT: shell status=0 pid=8986 duration=0(sec) May 17 15:25:58 client-5 xinetd[3391]: START: shell pid=9159 from=192.168.4.5 May 17 15:25:58 client-5 rshd[9160]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 65a: AT: verify early replies ================================================== 15:25:58 \(1305671158\)");echo XXRETCODE:$?' May 17 15:25:58 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 65a: AT: verify early replies ================================================== 15:25:58 (1305671158) May 17 15:25:58 client-5 xinetd[3391]: EXIT: shell status=0 pid=9159 duration=0(sec) May 17 15:26:25 client-5 xinetd[3391]: START: shell pid=9381 from=192.168.4.5 May 17 15:26:25 client-5 rshd[9382]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl set_param debug=\"inode super warning dlmtrace error emerg ha rpctrace vfstrace config console\";");echo XXRETCODE:$?' May 17 15:26:25 client-5 xinetd[3391]: EXIT: shell status=0 pid=9381 duration=0(sec) May 17 15:26:35 client-5 xinetd[3391]: START: shell pid=9477 from=192.168.4.5 May 17 15:26:35 client-5 rshd[9478]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 15:26:35 client-5 xinetd[3391]: EXIT: shell status=0 pid=9477 duration=0(sec) May 17 15:26:35 client-5 xinetd[3391]: START: shell pid=9649 from=192.168.4.5 May 17 15:26:35 client-5 rshd[9650]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 65b: AT: verify early replies on packed reply \/ bulk =========================== 15:26:35 \(1305671195\)");echo XXRETCODE:$?' May 17 15:26:35 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 65b: AT: verify early replies on packed reply / bulk =========================== 15:26:35 (1305671195) May 17 15:26:35 client-5 xinetd[3391]: EXIT: shell status=0 pid=9649 duration=0(sec) May 17 15:27:02 client-5 xinetd[3391]: START: shell pid=9878 from=192.168.4.5 May 17 15:27:02 client-5 rshd[9879]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl set_param debug=\"inode super warning dlmtrace error emerg ha rpctrace vfstrace config console\";");echo XXRETCODE:$?' May 17 15:27:02 client-5 xinetd[3391]: EXIT: shell status=0 pid=9878 duration=0(sec) May 17 15:27:02 client-5 xinetd[3391]: START: shell pid=9971 from=192.168.4.5 May 17 15:27:02 client-5 rshd[9972]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 15:27:03 client-5 xinetd[3391]: EXIT: shell status=0 pid=9971 duration=1(sec) May 17 15:27:03 client-5 xinetd[3391]: START: shell pid=10141 from=192.168.4.5 May 17 15:27:03 client-5 rshd[10144]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 66a: AT: verify MDT service time adjusts with no early replies ================= 15:27:03 \(1305671223\)");echo XXRETCODE:$?' May 17 15:27:03 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 66a: AT: verify MDT service time adjusts with no early replies ================= 15:27:03 (1305671223) May 17 15:27:03 client-5 xinetd[3391]: EXIT: shell status=0 pid=10141 duration=0(sec) May 17 15:27:48 client-5 xinetd[3391]: START: shell pid=10398 from=192.168.4.5 May 17 15:27:48 client-5 rshd[10401]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 15:27:48 client-5 xinetd[3391]: EXIT: shell status=0 pid=10398 duration=0(sec) May 17 15:27:49 client-5 xinetd[3391]: START: shell pid=10572 from=192.168.4.5 May 17 15:27:49 client-5 rshd[10573]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 66b: AT: verify net latency adjusts ============================================ 15:27:49 \(1305671269\)");echo XXRETCODE:$?' May 17 15:27:49 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 66b: AT: verify net latency adjusts ============================================ 15:27:49 (1305671269) May 17 15:27:49 client-5 xinetd[3391]: EXIT: shell status=0 pid=10572 duration=0(sec) May 17 15:28:10 client-5 kernel: LustreError: 10713:0:(lvfs_lib.c:150:__obd_fail_timeout_set()) obd_fail_timeout id 50c sleeping for 26000ms May 17 15:28:36 client-5 kernel: LustreError: 10713:0:(lvfs_lib.c:154:__obd_fail_timeout_set()) obd_fail_timeout id 50c awake May 17 15:28:36 client-5 kernel: LustreError: 10713:0:(lvfs_lib.c:150:__obd_fail_timeout_set()) obd_fail_timeout id 50c sleeping for 26000ms May 17 15:29:02 client-5 kernel: LustreError: 10713:0:(lvfs_lib.c:154:__obd_fail_timeout_set()) obd_fail_timeout id 50c awake May 17 15:29:02 client-5 xinetd[3391]: START: shell pid=10780 from=192.168.4.5 May 17 15:29:02 client-5 rshd[10782]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 15:29:02 client-5 xinetd[3391]: EXIT: shell status=0 pid=10780 duration=0(sec) May 17 15:29:02 client-5 xinetd[3391]: START: shell pid=10951 from=192.168.4.5 May 17 15:29:02 client-5 rshd[10954]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 67a: AT: verify slow request processing doesn\'t induce reconnects ============== 15:29:02 \(1305671342\)");echo XXRETCODE:$?' May 17 15:29:02 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 67a: AT: verify slow request processing doesn't induce reconnects ============== 15:29:02 (1305671342) May 17 15:29:02 client-5 xinetd[3391]: EXIT: shell status=0 pid=10951 duration=0(sec) May 17 15:29:24 client-5 xinetd[3391]: START: shell pid=11177 from=192.168.4.5 May 17 15:29:24 client-5 rshd[11178]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 15:29:24 client-5 xinetd[3391]: EXIT: shell status=0 pid=11177 duration=0(sec) May 17 15:29:24 client-5 xinetd[3391]: START: shell pid=11349 from=192.168.4.5 May 17 15:29:24 client-5 rshd[11350]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 67b: AT: verify instant slowdown doesn\'t induce reconnects ===================== 15:29:24 \(1305671364\)");echo XXRETCODE:$?' May 17 15:29:24 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 67b: AT: verify instant slowdown doesn't induce reconnects ===================== 15:29:24 (1305671364) May 17 15:29:24 client-5 xinetd[3391]: EXIT: shell status=0 pid=11349 duration=0(sec) May 17 15:29:46 client-5 xinetd[3391]: START: shell pid=11607 from=192.168.4.5 May 17 15:29:46 client-5 rshd[11608]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "df /mnt/lustre; uname -n >> /mnt/lustre/recon");echo XXRETCODE:$?' May 17 15:29:46 client-5 xinetd[3391]: EXIT: shell status=0 pid=11607 duration=0(sec) May 17 15:29:46 client-5 xinetd[3391]: START: shell pid=11713 from=192.168.4.5 May 17 15:29:46 client-5 rshd[11714]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark phase 2");echo XXRETCODE:$?' May 17 15:29:46 client-5 kernel: Lustre: DEBUG MARKER: phase 2 May 17 15:29:46 client-5 xinetd[3391]: EXIT: shell status=0 pid=11713 duration=0(sec) May 17 15:29:47 client-5 xinetd[3391]: START: shell pid=11776 from=192.168.4.5 May 17 15:29:47 client-5 rshd[11777]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "df /mnt/lustre; uname -n >> /mnt/lustre/recon");echo XXRETCODE:$?' May 17 15:29:47 client-5 xinetd[3391]: EXIT: shell status=0 pid=11776 duration=0(sec) May 17 15:29:47 client-5 xinetd[3391]: START: shell pid=11882 from=192.168.4.5 May 17 15:29:47 client-5 rshd[11884]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 15:29:47 client-5 xinetd[3391]: EXIT: shell status=0 pid=11882 duration=0(sec) May 17 15:29:47 client-5 xinetd[3391]: START: shell pid=12055 from=192.168.4.5 May 17 15:29:47 client-5 rshd[12056]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 68: AT: verify slowing locks =================================================== 15:29:47 \(1305671387\)");echo XXRETCODE:$?' May 17 15:29:47 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 68: AT: verify slowing locks =================================================== 15:29:47 (1305671387) May 17 15:29:47 client-5 xinetd[3391]: EXIT: shell status=0 pid=12055 duration=0(sec) May 17 15:30:08 client-5 kernel: LustreError: 12228:0:(lvfs_lib.c:150:__obd_fail_timeout_set()) obd_fail_timeout id 312 sleeping for 19000ms May 17 15:30:27 client-5 kernel: LustreError: 12228:0:(lvfs_lib.c:154:__obd_fail_timeout_set()) obd_fail_timeout id 312 awake May 17 15:30:27 client-5 kernel: LustreError: 12231:0:(lvfs_lib.c:150:__obd_fail_timeout_set()) obd_fail_timeout id 312 sleeping for 25000ms May 17 15:30:53 client-5 kernel: LustreError: 12231:0:(lvfs_lib.c:154:__obd_fail_timeout_set()) obd_fail_timeout id 312 awake May 17 15:30:53 client-5 xinetd[3391]: START: shell pid=12297 from=192.168.4.5 May 17 15:30:53 client-5 rshd[12299]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 15:30:53 client-5 xinetd[3391]: EXIT: shell status=0 pid=12297 duration=0(sec) May 17 15:30:53 client-5 xinetd[3391]: START: shell pid=12469 from=192.168.4.5 May 17 15:30:53 client-5 rshd[12471]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 70a: check multi client t-f ==================================================== 15:30:53 \(1305671453\)");echo XXRETCODE:$?' May 17 15:30:53 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 70a: check multi client t-f ==================================================== 15:30:53 (1305671453) May 17 15:30:53 client-5 xinetd[3391]: EXIT: shell status=0 pid=12469 duration=0(sec) May 17 15:30:53 client-5 xinetd[3391]: START: shell pid=12512 from=192.168.4.5 May 17 15:30:53 client-5 rshd[12513]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c " running=\$(mount | grep -c /mnt/lustre' '); rc=0; if [ \$running -eq 0 ] ; then mkdir -p /mnt/lustre; mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre; rc=\$?; fi; exit \$rc");echo XXRETCODE:$?' May 17 15:30:53 client-5 xinetd[3391]: EXIT: shell status=0 pid=12512 duration=0(sec) May 17 15:30:53 client-5 xinetd[3391]: START: shell pid=12557 from=192.168.4.5 May 17 15:30:53 client-5 rshd[12558]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mount | grep -w /mnt/lustre");echo XXRETCODE:$?' May 17 15:30:53 client-5 xinetd[3391]: EXIT: shell status=0 pid=12557 duration=0(sec) May 17 15:30:53 client-5 xinetd[3391]: START: shell pid=12601 from=192.168.4.5 May 17 15:30:53 client-5 rshd[12602]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param debug=\"0x33f0406\"; lctl set_param subsystem_debug=\"0xffb7e3ff\"; lctl set_param debug_mb=32;");echo XXRETCODE:$?' May 17 15:30:53 client-5 kernel: Lustre: 12636:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 15:30:53 client-5 kernel: Lustre: 12636:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message May 17 15:30:53 client-5 xinetd[3391]: EXIT: shell status=0 pid=12601 duration=0(sec) May 17 15:30:58 client-5 xinetd[3391]: START: shell pid=12716 from=192.168.4.5 May 17 15:30:58 client-5 rshd[12717]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 15:30:58 client-5 xinetd[3391]: EXIT: shell status=0 pid=12716 duration=0(sec) May 17 15:30:58 client-5 xinetd[3391]: START: shell pid=12887 from=192.168.4.5 May 17 15:30:58 client-5 rshd[12889]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 70b: mds recovery\; 2 clients =================================================== 15:30:58 \(1305671458\)");echo XXRETCODE:$?' May 17 15:30:58 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 70b: mds recovery; 2 clients =================================================== 15:30:58 (1305671458) May 17 15:30:58 client-5 xinetd[3391]: EXIT: shell status=0 pid=12887 duration=0(sec) May 17 15:30:58 client-5 xinetd[3391]: START: shell pid=12930 from=192.168.4.5 May 17 15:30:58 client-5 rshd[12931]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c " running=\$(mount | grep -c /mnt/lustre' '); rc=0; if [ \$running -eq 0 ] ; then mkdir -p /mnt/lustre; mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre; rc=\$?; fi; exit \$rc");echo XXRETCODE:$?' May 17 15:30:58 client-5 xinetd[3391]: EXIT: shell status=0 pid=12930 duration=0(sec) May 17 15:30:58 client-5 xinetd[3391]: START: shell pid=12975 from=192.168.4.5 May 17 15:30:58 client-5 rshd[12976]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mount | grep -w /mnt/lustre");echo XXRETCODE:$?' May 17 15:30:58 client-5 xinetd[3391]: EXIT: shell status=0 pid=12975 duration=0(sec) May 17 15:30:58 client-5 xinetd[3391]: START: shell pid=13018 from=192.168.4.5 May 17 15:30:59 client-5 rshd[13020]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param debug=\"0x33f0406\"; lctl set_param subsystem_debug=\"0xffb7e3ff\"; lctl set_param debug_mb=32;");echo XXRETCODE:$?' May 17 15:30:59 client-5 kernel: Lustre: 13054:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 15:30:59 client-5 kernel: Lustre: 13054:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message May 17 15:30:59 client-5 xinetd[3391]: EXIT: shell status=0 pid=13018 duration=1(sec) May 17 15:30:59 client-5 xinetd[3391]: START: shell pid=13073 from=192.168.4.5 May 17 15:30:59 client-5 rshd[13104]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "set -x; MISSING_DBENCH_OK= PATH=:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/tests:/usr/lib64/openmpi/1.4-gcc/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/usr/lib64/lustre/utils:/usr/lib64/lustre/utils/gss:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey::/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests/:/usr/share/doc/dbench/loadfiles DBENCH_LIB=/usr/share/doc/dbench/loadfiles TESTSUITE=replay-single TESTNAME=test_70b LCTL=/usr/sbin/lctl rundbench 1 -t 300");echo XXRETCODE:$?' May 17 15:30:59 client-5 xinetd[3391]: START: shell pid=13146 from=192.168.4.5 May 17 15:30:59 client-5 rshd[13148]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark Started rundbench load pid=13056 ...");echo XXRETCODE:$?' May 17 15:30:59 client-5 kernel: Lustre: DEBUG MARKER: Started rundbench load pid=13056 ... May 17 15:30:59 client-5 xinetd[3391]: EXIT: shell status=0 pid=13146 duration=0(sec) May 17 15:31:01 client-5 xinetd[3391]: START: shell pid=13211 from=192.168.4.5 May 17 15:31:01 client-5 rshd[13212]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "killall -0 rundbench");echo XXRETCODE:$?' May 17 15:31:01 client-5 xinetd[3391]: EXIT: shell status=0 pid=13211 duration=0(sec) May 17 15:31:02 client-5 xinetd[3391]: START: shell pid=13263 from=192.168.4.5 May 17 15:31:02 client-5 rshd[13264]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 15:31:02 client-5 xinetd[3391]: EXIT: shell status=0 pid=13263 duration=0(sec) May 17 15:31:02 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 15:31:04 client-5 xinetd[3391]: START: shell pid=13385 from=192.168.4.5 May 17 15:31:04 client-5 rshd[13387]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark test_70b fail mds1 1 times");echo XXRETCODE:$?' May 17 15:31:04 client-5 kernel: Lustre: DEBUG MARKER: test_70b fail mds1 1 times May 17 15:31:04 client-5 xinetd[3391]: EXIT: shell status=0 pid=13385 duration=0(sec) May 17 15:31:04 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_close operation failed with -107 May 17 15:31:04 client-5 kernel: LustreError: Skipped 30 previous similar messages May 17 15:31:04 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff8102c72f6400: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 15:31:04 client-5 kernel: Lustre: Skipped 10 previous similar messages May 17 15:31:16 client-5 kernel: Lustre: 20197:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369045627471329 sent from MGC192.168.4.128@o2ib to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305671469] [real_sent 1305671469] [current 1305671476] [deadline 7s] [delay 0s] req@ffff810315181800 x1369045627471329/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 192/192 e 0 to 1 dl 1305671476 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 15:31:16 client-5 kernel: Lustre: 20197:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 35 previous similar messages May 17 15:31:16 client-5 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail. May 17 15:31:16 client-5 kernel: LustreError: Skipped 4 previous similar messages May 17 15:31:23 client-5 xinetd[3391]: START: shell pid=13543 from=192.168.4.5 May 17 15:31:23 client-5 rshd[13544]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 15:31:26 client-5 kernel: Lustre: 20198:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74d1895f to 0x62e5a8d74d4c637 May 17 15:31:26 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 15:31:26 client-5 kernel: Lustre: Skipped 4 previous similar messages May 17 15:31:26 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Connection restored to service MGS using nid 192.168.4.128@o2ib. May 17 15:31:26 client-5 kernel: Lustre: Skipped 20 previous similar messages May 17 15:31:39 client-5 kernel: LustreError: 20198:0:(client.c:2570:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff81031f648800 x1369045627463380/t313532613098(313532613098) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 552/544 e 0 to 0 dl 1305671550 ref 2 fl Interpret:RP/ffffffff/ffffffff rc 301/-1 May 17 15:31:39 client-5 kernel: LustreError: 20198:0:(client.c:2570:ptlrpc_replay_interpret()) Skipped 3 previous similar messages May 17 15:31:40 client-5 xinetd[3391]: EXIT: shell status=0 pid=13543 duration=17(sec) May 17 15:31:40 client-5 xinetd[3391]: START: shell pid=13589 from=192.168.4.5 May 17 15:31:40 client-5 rshd[13590]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "killall -0 rundbench");echo XXRETCODE:$?' May 17 15:31:40 client-5 xinetd[3391]: EXIT: shell status=0 pid=13589 duration=0(sec) May 17 15:31:41 client-5 xinetd[3391]: START: shell pid=13641 from=192.168.4.5 May 17 15:31:41 client-5 rshd[13642]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 15:31:41 client-5 xinetd[3391]: EXIT: shell status=0 pid=13641 duration=0(sec) May 17 15:31:42 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 15:31:43 client-5 xinetd[3391]: START: shell pid=13763 from=192.168.4.5 May 17 15:31:43 client-5 rshd[13765]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark test_70b fail mds1 2 times");echo XXRETCODE:$?' May 17 15:31:43 client-5 kernel: Lustre: DEBUG MARKER: test_70b fail mds1 2 times May 17 15:31:43 client-5 xinetd[3391]: EXIT: shell status=0 pid=13763 duration=0(sec) May 17 15:32:02 client-5 xinetd[3391]: START: shell pid=13917 from=192.168.4.5 May 17 15:32:02 client-5 rshd[13918]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 15:32:13 client-5 kernel: Lustre: 20198:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74d4c637 to 0x62e5a8d74d72ae1 May 17 15:32:23 client-5 kernel: LustreError: 20198:0:(client.c:2570:ptlrpc_replay_interpret()) @@@ status -116, old was 0 req@ffff810319dc0800 x1369045627473201/t317827581890(317827581890) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:23/10 lens 360/424 e 0 to 0 dl 1305671575 ref 2 fl Interpret:R/ffffffff/ffffffff rc -116/-1 May 17 15:32:24 client-5 xinetd[3391]: EXIT: shell status=0 pid=13917 duration=22(sec) May 17 15:32:24 client-5 xinetd[3391]: START: shell pid=13966 from=192.168.4.5 May 17 15:32:24 client-5 rshd[13967]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "killall -0 rundbench");echo XXRETCODE:$?' May 17 15:32:24 client-5 xinetd[3391]: EXIT: shell status=0 pid=13966 duration=0(sec) May 17 15:32:25 client-5 xinetd[3391]: START: shell pid=14018 from=192.168.4.5 May 17 15:32:25 client-5 rshd[14019]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 15:32:25 client-5 xinetd[3391]: EXIT: shell status=0 pid=14018 duration=0(sec) May 17 15:32:26 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 15:32:27 client-5 xinetd[3391]: START: shell pid=14141 from=192.168.4.5 May 17 15:32:27 client-5 rshd[14142]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark test_70b fail mds1 3 times");echo XXRETCODE:$?' May 17 15:32:27 client-5 kernel: Lustre: DEBUG MARKER: test_70b fail mds1 3 times May 17 15:32:27 client-5 xinetd[3391]: EXIT: shell status=0 pid=14141 duration=0(sec) May 17 15:32:45 client-5 kernel: Lustre: 20198:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74d72ae1 to 0x62e5a8d74d99010 May 17 15:32:46 client-5 xinetd[3391]: START: shell pid=14295 from=192.168.4.5 May 17 15:32:46 client-5 rshd[14296]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 15:33:07 client-5 xinetd[3391]: EXIT: shell status=0 pid=14295 duration=21(sec) May 17 15:33:07 client-5 xinetd[3391]: START: shell pid=14339 from=192.168.4.5 May 17 15:33:07 client-5 rshd[14340]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "killall -0 rundbench");echo XXRETCODE:$?' May 17 15:33:07 client-5 xinetd[3391]: EXIT: shell status=0 pid=14339 duration=0(sec) May 17 15:33:08 client-5 xinetd[3391]: START: shell pid=14391 from=192.168.4.5 May 17 15:33:08 client-5 rshd[14392]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 15:33:08 client-5 xinetd[3391]: EXIT: shell status=0 pid=14391 duration=0(sec) May 17 15:33:08 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 15:33:09 client-5 xinetd[3391]: START: shell pid=14514 from=192.168.4.5 May 17 15:33:10 client-5 rshd[14515]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark test_70b fail mds1 4 times");echo XXRETCODE:$?' May 17 15:33:10 client-5 kernel: Lustre: DEBUG MARKER: test_70b fail mds1 4 times May 17 15:33:10 client-5 xinetd[3391]: EXIT: shell status=0 pid=14514 duration=1(sec) May 17 15:33:29 client-5 xinetd[3391]: START: shell pid=14667 from=192.168.4.5 May 17 15:33:29 client-5 rshd[14668]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 15:33:30 client-5 kernel: Lustre: 20198:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74d99010 to 0x62e5a8d74dbe9df May 17 15:33:51 client-5 xinetd[3391]: EXIT: shell status=0 pid=14667 duration=22(sec) May 17 15:33:51 client-5 xinetd[3391]: START: shell pid=14716 from=192.168.4.5 May 17 15:33:51 client-5 rshd[14717]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "killall -0 rundbench");echo XXRETCODE:$?' May 17 15:33:51 client-5 xinetd[3391]: EXIT: shell status=0 pid=14716 duration=0(sec) May 17 15:33:52 client-5 xinetd[3391]: START: shell pid=14768 from=192.168.4.5 May 17 15:33:52 client-5 rshd[14769]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 15:33:52 client-5 xinetd[3391]: EXIT: shell status=0 pid=14768 duration=0(sec) May 17 15:33:53 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 15:33:54 client-5 xinetd[3391]: START: shell pid=14891 from=192.168.4.5 May 17 15:33:54 client-5 rshd[14892]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark test_70b fail mds1 5 times");echo XXRETCODE:$?' May 17 15:33:54 client-5 kernel: Lustre: DEBUG MARKER: test_70b fail mds1 5 times May 17 15:33:54 client-5 xinetd[3391]: EXIT: shell status=0 pid=14891 duration=0(sec) May 17 15:34:12 client-5 kernel: Lustre: 20198:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74dbe9df to 0x62e5a8d74de5806 May 17 15:34:13 client-5 xinetd[3391]: START: shell pid=15045 from=192.168.4.5 May 17 15:34:13 client-5 rshd[15046]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 15:34:32 client-5 kernel: Lustre: 20199:0:(import.c:529:import_select_connection()) lustre-MDT0000-mdc-ffff8102c72f6400: tried all connections, increasing latency to 26s May 17 15:34:32 client-5 kernel: Lustre: 20199:0:(import.c:529:import_select_connection()) Skipped 11 previous similar messages May 17 15:34:33 client-5 kernel: LustreError: 20198:0:(client.c:2570:ptlrpc_replay_interpret()) @@@ status -116, old was 0 req@ffff81030caa0400 x1369045627487334/t330712483757(330712483757) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:23/10 lens 360/424 e 0 to 0 dl 1305671705 ref 2 fl Interpret:R/ffffffff/ffffffff rc -116/-1 May 17 15:34:33 client-5 kernel: LustreError: 20198:0:(client.c:2570:ptlrpc_replay_interpret()) Skipped 1 previous similar message May 17 15:34:34 client-5 xinetd[3391]: EXIT: shell status=0 pid=15045 duration=21(sec) May 17 15:34:34 client-5 xinetd[3391]: START: shell pid=15088 from=192.168.4.5 May 17 15:34:34 client-5 rshd[15090]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "killall -0 rundbench");echo XXRETCODE:$?' May 17 15:34:34 client-5 xinetd[3391]: EXIT: shell status=0 pid=15088 duration=0(sec) May 17 15:34:35 client-5 xinetd[3391]: START: shell pid=15141 from=192.168.4.5 May 17 15:34:35 client-5 rshd[15142]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 15:34:35 client-5 xinetd[3391]: EXIT: shell status=0 pid=15141 duration=0(sec) May 17 15:34:36 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 15:34:37 client-5 xinetd[3391]: START: shell pid=15264 from=192.168.4.5 May 17 15:34:37 client-5 rshd[15265]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark test_70b fail mds1 6 times");echo XXRETCODE:$?' May 17 15:34:37 client-5 kernel: Lustre: DEBUG MARKER: test_70b fail mds1 6 times May 17 15:34:37 client-5 xinetd[3391]: EXIT: shell status=0 pid=15264 duration=0(sec) May 17 15:34:56 client-5 xinetd[3391]: START: shell pid=15421 from=192.168.4.5 May 17 15:34:56 client-5 rshd[15422]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 15:34:57 client-5 kernel: Lustre: 20198:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74de5806 to 0x62e5a8d74e0be5b May 17 15:35:18 client-5 xinetd[3391]: EXIT: shell status=0 pid=15421 duration=22(sec) May 17 15:35:18 client-5 xinetd[3391]: START: shell pid=15466 from=192.168.4.5 May 17 15:35:18 client-5 rshd[15467]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "killall -0 rundbench");echo XXRETCODE:$?' May 17 15:35:18 client-5 xinetd[3391]: EXIT: shell status=0 pid=15466 duration=0(sec) May 17 15:35:19 client-5 xinetd[3391]: START: shell pid=15518 from=192.168.4.5 May 17 15:35:19 client-5 rshd[15519]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 15:35:19 client-5 xinetd[3391]: EXIT: shell status=0 pid=15518 duration=0(sec) May 17 15:35:20 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 15:35:21 client-5 xinetd[3391]: START: shell pid=15641 from=192.168.4.5 May 17 15:35:21 client-5 rshd[15642]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark test_70b fail mds1 7 times");echo XXRETCODE:$?' May 17 15:35:21 client-5 kernel: Lustre: DEBUG MARKER: test_70b fail mds1 7 times May 17 15:35:21 client-5 xinetd[3391]: EXIT: shell status=0 pid=15641 duration=0(sec) May 17 15:35:39 client-5 kernel: Lustre: 20198:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74e0be5b to 0x62e5a8d74e331bb May 17 15:35:40 client-5 xinetd[3391]: START: shell pid=15795 from=192.168.4.5 May 17 15:35:40 client-5 rshd[15796]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 15:36:00 client-5 xinetd[3391]: EXIT: shell status=0 pid=15795 duration=20(sec) May 17 15:36:00 client-5 xinetd[3391]: START: shell pid=15839 from=192.168.4.5 May 17 15:36:00 client-5 rshd[15840]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "killall -0 rundbench");echo XXRETCODE:$?' May 17 15:36:00 client-5 xinetd[3391]: EXIT: shell status=0 pid=15839 duration=0(sec) May 17 15:36:01 client-5 xinetd[3391]: START: shell pid=15891 from=192.168.4.5 May 17 15:36:01 client-5 rshd[15892]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 15:36:01 client-5 xinetd[3391]: EXIT: shell status=0 pid=15891 duration=0(sec) May 17 15:36:02 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 15:36:03 client-5 xinetd[3391]: START: shell pid=16014 from=192.168.4.5 May 17 15:36:03 client-5 rshd[16015]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark test_70b fail mds1 8 times");echo XXRETCODE:$?' May 17 15:36:03 client-5 kernel: Lustre: DEBUG MARKER: test_70b fail mds1 8 times May 17 15:36:03 client-5 xinetd[3391]: EXIT: shell status=0 pid=16014 duration=0(sec) May 17 15:36:21 client-5 kernel: Lustre: 20198:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74e331bb to 0x62e5a8d74e58e7e May 17 15:36:22 client-5 xinetd[3391]: START: shell pid=16172 from=192.168.4.5 May 17 15:36:22 client-5 rshd[16173]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 15:36:41 client-5 kernel: LustreError: 20198:0:(client.c:2570:ptlrpc_replay_interpret()) @@@ status -116, old was 0 req@ffff81031464f800 x1369045627501837/t343597385474(343597385474) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:23/10 lens 360/424 e 0 to 0 dl 1305671833 ref 2 fl Interpret:R/ffffffff/ffffffff rc -116/-1 May 17 15:36:41 client-5 kernel: LustreError: 20198:0:(client.c:2570:ptlrpc_replay_interpret()) Skipped 4 previous similar messages May 17 15:36:42 client-5 xinetd[3391]: EXIT: shell status=0 pid=16172 duration=20(sec) May 17 15:37:00 client-5 xinetd[3391]: EXIT: shell status=0 pid=13073 duration=361(sec) May 17 15:37:00 client-5 xinetd[3391]: START: shell pid=16277 from=192.168.4.5 May 17 15:37:00 client-5 rshd[16278]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 15:37:00 client-5 xinetd[3391]: EXIT: shell status=0 pid=16277 duration=0(sec) May 17 15:37:00 client-5 xinetd[3391]: START: shell pid=16449 from=192.168.4.5 May 17 15:37:00 client-5 rshd[16450]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 73a: open\(O_CREAT\), unlink, replay, reconnect before open replay , close ======= 15:37:00 \(1305671820\)");echo XXRETCODE:$?' May 17 15:37:00 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 73a: open(O_CREAT), unlink, replay, reconnect before open replay , close ======= 15:37:00 (1305671820) May 17 15:37:00 client-5 xinetd[3391]: EXIT: shell status=0 pid=16449 duration=0(sec) May 17 15:37:00 client-5 xinetd[3391]: START: shell pid=16504 from=192.168.4.5 May 17 15:37:00 client-5 rshd[16505]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 15:37:01 client-5 xinetd[3391]: EXIT: shell status=0 pid=16504 duration=1(sec) May 17 15:37:01 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 15:37:21 client-5 xinetd[3391]: START: shell pid=16694 from=192.168.4.5 May 17 15:37:21 client-5 rshd[16695]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 15:37:21 client-5 kernel: Lustre: 20198:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74e58e7e to 0x62e5a8d74f21dd1 May 17 15:37:47 client-5 xinetd[3391]: EXIT: shell status=0 pid=16694 duration=26(sec) May 17 15:37:47 client-5 xinetd[3391]: START: shell pid=16791 from=192.168.4.5 May 17 15:37:47 client-5 rshd[16792]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 15:37:47 client-5 xinetd[3391]: EXIT: shell status=0 pid=16791 duration=0(sec) May 17 15:37:47 client-5 xinetd[3391]: START: shell pid=16962 from=192.168.4.5 May 17 15:37:48 client-5 rshd[16964]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 73b: open\(O_CREAT\), unlink, replay, reconnect at open_replay reply, close ====== 15:37:47 \(1305671867\)");echo XXRETCODE:$?' May 17 15:37:48 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 73b: open(O_CREAT), unlink, replay, reconnect at open_replay reply, close ====== 15:37:47 (1305671867) May 17 15:37:48 client-5 xinetd[3391]: EXIT: shell status=0 pid=16962 duration=1(sec) May 17 15:37:48 client-5 xinetd[3391]: START: shell pid=17018 from=192.168.4.5 May 17 15:37:48 client-5 rshd[17019]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 15:37:48 client-5 xinetd[3391]: EXIT: shell status=0 pid=17018 duration=0(sec) May 17 15:37:48 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 15:38:06 client-5 kernel: Lustre: 20198:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74f21dd1 to 0x62e5a8d74f22086 May 17 15:38:08 client-5 xinetd[3391]: START: shell pid=17209 from=192.168.4.5 May 17 15:38:08 client-5 rshd[17210]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 15:38:54 client-5 xinetd[3391]: EXIT: shell status=0 pid=17209 duration=46(sec) May 17 15:38:54 client-5 xinetd[3391]: START: shell pid=17305 from=192.168.4.5 May 17 15:38:54 client-5 rshd[17306]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 15:38:54 client-5 xinetd[3391]: EXIT: shell status=0 pid=17305 duration=0(sec) May 17 15:38:55 client-5 xinetd[3391]: START: shell pid=17475 from=192.168.4.5 May 17 15:38:55 client-5 rshd[17478]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 73c: open\(O_CREAT\), unlink, replay, reconnect at last_replay, close ============ 15:38:54 \(1305671934\)");echo XXRETCODE:$?' May 17 15:38:55 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 73c: open(O_CREAT), unlink, replay, reconnect at last_replay, close ============ 15:38:54 (1305671934) May 17 15:38:55 client-5 xinetd[3391]: EXIT: shell status=0 pid=17475 duration=0(sec) May 17 15:38:55 client-5 xinetd[3391]: START: shell pid=17532 from=192.168.4.5 May 17 15:38:55 client-5 rshd[17533]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 15:38:55 client-5 xinetd[3391]: EXIT: shell status=0 pid=17532 duration=0(sec) May 17 15:38:55 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 15:39:13 client-5 kernel: Lustre: 20198:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74f22086 to 0x62e5a8d74f223a4 May 17 15:39:15 client-5 xinetd[3391]: START: shell pid=17723 from=192.168.4.5 May 17 15:39:15 client-5 rshd[17725]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 15:39:15 client-5 xinetd[3391]: EXIT: shell status=0 pid=17723 duration=0(sec) May 17 15:39:15 client-5 xinetd[3391]: START: shell pid=17814 from=192.168.4.5 May 17 15:39:16 client-5 rshd[17815]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 15:39:16 client-5 xinetd[3391]: EXIT: shell status=0 pid=17814 duration=1(sec) May 17 15:39:16 client-5 xinetd[3391]: START: shell pid=17986 from=192.168.4.5 May 17 15:39:16 client-5 rshd[17987]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 74: Ensure applications don\'t fail waiting for OST recovery ==================== 15:39:16 \(1305671956\)");echo XXRETCODE:$?' May 17 15:39:16 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 74: Ensure applications don't fail waiting for OST recovery ==================== 15:39:16 (1305671956) May 17 15:39:16 client-5 xinetd[3391]: EXIT: shell status=0 pid=17986 duration=0(sec) May 17 15:39:17 client-5 xinetd[3391]: START: shell pid=18060 from=192.168.4.5 May 17 15:39:17 client-5 rshd[18061]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:; lsof -t /mnt/lustre || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre is still busy, wait one second && sleep 1; umount /mnt/lustre; fi fi");echo XXRETCODE:$?' May 17 15:39:18 client-5 kernel: LustreError: 18102:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -107 from cancel RPC: canceling anyway May 17 15:39:18 client-5 kernel: LustreError: 18102:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -107 May 17 15:39:18 client-5 kernel: LustreError: 18102:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 15:39:18 client-5 kernel: LustreError: 18102:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Skipped 1 previous similar message May 17 15:39:18 client-5 kernel: LustreError: 18102:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 15:39:18 client-5 kernel: LustreError: 18102:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) Skipped 1 previous similar message May 17 15:39:18 client-5 kernel: Lustre: client ffff8102c72f6400 umount complete May 17 15:39:18 client-5 xinetd[3391]: EXIT: shell status=0 pid=18060 duration=1(sec) May 17 15:39:38 client-5 xinetd[3391]: START: shell pid=18222 from=192.168.4.5 May 17 15:39:38 client-5 rshd[18224]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c " running=\$(mount | grep -c /mnt/lustre' '); rc=0; if [ \$running -eq 0 ] ; then mkdir -p /mnt/lustre; mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre; rc=\$?; fi; exit \$rc");echo XXRETCODE:$?' May 17 15:39:38 client-5 kernel: Lustre: 18263:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 15:39:38 client-5 kernel: Lustre: 18263:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 6 previous similar messages May 17 15:39:38 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 15:39:38 client-5 kernel: Lustre: Increasing default stripe size to min 1048576 May 17 15:39:38 client-5 kernel: Lustre: Client lustre-client has started May 17 15:39:38 client-5 xinetd[3391]: EXIT: shell status=0 pid=18222 duration=0(sec) May 17 15:39:38 client-5 xinetd[3391]: START: shell pid=18283 from=192.168.4.5 May 17 15:39:38 client-5 rshd[18284]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mount | grep -w /mnt/lustre");echo XXRETCODE:$?' May 17 15:39:38 client-5 xinetd[3391]: EXIT: shell status=0 pid=18283 duration=0(sec) May 17 15:39:38 client-5 xinetd[3391]: START: shell pid=18327 from=192.168.4.5 May 17 15:39:38 client-5 rshd[18328]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param debug=\"0x33f0406\"; lctl set_param subsystem_debug=\"0xffb7e3ff\"; lctl set_param debug_mb=32;");echo XXRETCODE:$?' May 17 15:39:38 client-5 kernel: Lustre: 18362:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 15:39:38 client-5 kernel: Lustre: 18362:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message May 17 15:39:38 client-5 xinetd[3391]: EXIT: shell status=0 pid=18327 duration=0(sec) May 17 15:39:44 client-5 xinetd[3391]: START: shell pid=18399 from=192.168.4.5 May 17 15:39:44 client-5 rshd[18400]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 15:40:53 client-5 xinetd[3391]: EXIT: shell status=0 pid=18399 duration=69(sec) May 17 15:40:53 client-5 xinetd[3391]: START: shell pid=18491 from=192.168.4.5 May 17 15:40:53 client-5 rshd[18494]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 15:40:53 client-5 xinetd[3391]: EXIT: shell status=0 pid=18491 duration=0(sec) May 17 15:40:53 client-5 xinetd[3391]: START: shell pid=18665 from=192.168.4.5 May 17 15:40:53 client-5 rshd[18666]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 80a: CMD: unlink cross-node dir \(fail mds with inode\) ========================== 15:40:53 \(1305672053\)");echo XXRETCODE:$?' May 17 15:40:53 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 80a: CMD: unlink cross-node dir (fail mds with inode) ========================== 15:40:53 (1305672053) May 17 15:40:53 client-5 xinetd[3391]: EXIT: shell status=0 pid=18665 duration=0(sec) May 17 15:40:54 client-5 xinetd[3391]: START: shell pid=18758 from=192.168.4.5 May 17 15:40:54 client-5 rshd[18759]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark SKIP: replay-single test_80a needs \>= 2 MDTs");echo XXRETCODE:$?' May 17 15:40:54 client-5 kernel: Lustre: DEBUG MARKER: SKIP: replay-single test_80a needs >= 2 MDTs May 17 15:40:54 client-5 xinetd[3391]: EXIT: shell status=0 pid=18758 duration=0(sec) May 17 15:40:54 client-5 xinetd[3391]: START: shell pid=18848 from=192.168.4.5 May 17 15:40:54 client-5 rshd[18850]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 15:40:54 client-5 xinetd[3391]: EXIT: shell status=0 pid=18848 duration=0(sec) May 17 15:40:54 client-5 xinetd[3391]: START: shell pid=19021 from=192.168.4.5 May 17 15:40:54 client-5 rshd[19022]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 80b: CMD: unlink cross-node dir \(fail mds with name\) =========================== 15:40:54 \(1305672054\)");echo XXRETCODE:$?' May 17 15:40:54 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 80b: CMD: unlink cross-node dir (fail mds with name) =========================== 15:40:54 (1305672054) May 17 15:40:54 client-5 xinetd[3391]: EXIT: shell status=0 pid=19021 duration=0(sec) May 17 15:40:54 client-5 xinetd[3391]: START: shell pid=19114 from=192.168.4.5 May 17 15:40:54 client-5 rshd[19115]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark SKIP: replay-single test_80b needs \>= 2 MDTs");echo XXRETCODE:$?' May 17 15:40:54 client-5 kernel: Lustre: DEBUG MARKER: SKIP: replay-single test_80b needs >= 2 MDTs May 17 15:40:54 client-5 xinetd[3391]: EXIT: shell status=0 pid=19114 duration=0(sec) May 17 15:40:54 client-5 xinetd[3391]: START: shell pid=19205 from=192.168.4.5 May 17 15:40:54 client-5 rshd[19206]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 15:40:54 client-5 xinetd[3391]: EXIT: shell status=0 pid=19205 duration=0(sec) May 17 15:40:55 client-5 xinetd[3391]: START: shell pid=19377 from=192.168.4.5 May 17 15:40:55 client-5 rshd[19378]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 81a: CMD: unlink cross-node file \(fail mds with name\) ========================== 15:40:55 \(1305672055\)");echo XXRETCODE:$?' May 17 15:40:55 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 81a: CMD: unlink cross-node file (fail mds with name) ========================== 15:40:55 (1305672055) May 17 15:40:55 client-5 xinetd[3391]: EXIT: shell status=0 pid=19377 duration=0(sec) May 17 15:40:55 client-5 xinetd[3391]: START: shell pid=19470 from=192.168.4.5 May 17 15:40:55 client-5 rshd[19471]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark SKIP: replay-single test_81a needs \>= 2 MDTs");echo XXRETCODE:$?' May 17 15:40:55 client-5 kernel: Lustre: DEBUG MARKER: SKIP: replay-single test_81a needs >= 2 MDTs May 17 15:40:55 client-5 xinetd[3391]: EXIT: shell status=0 pid=19470 duration=0(sec) May 17 15:40:55 client-5 xinetd[3391]: START: shell pid=19559 from=192.168.4.5 May 17 15:40:55 client-5 rshd[19562]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 15:40:55 client-5 xinetd[3391]: EXIT: shell status=0 pid=19559 duration=0(sec) May 17 15:40:55 client-5 xinetd[3391]: START: shell pid=19733 from=192.168.4.5 May 17 15:40:55 client-5 rshd[19734]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 82a: CMD: mkdir cross-node dir \(fail mds with inode\) =========================== 15:40:55 \(1305672055\)");echo XXRETCODE:$?' May 17 15:40:55 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 82a: CMD: mkdir cross-node dir (fail mds with inode) =========================== 15:40:55 (1305672055) May 17 15:40:55 client-5 xinetd[3391]: EXIT: shell status=0 pid=19733 duration=0(sec) May 17 15:40:55 client-5 xinetd[3391]: START: shell pid=19826 from=192.168.4.5 May 17 15:40:55 client-5 rshd[19827]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark SKIP: replay-single test_82a needs \>= 2 MDTs");echo XXRETCODE:$?' May 17 15:40:56 client-5 kernel: Lustre: DEBUG MARKER: SKIP: replay-single test_82a needs >= 2 MDTs May 17 15:40:56 client-5 xinetd[3391]: EXIT: shell status=0 pid=19826 duration=1(sec) May 17 15:40:56 client-5 xinetd[3391]: START: shell pid=19917 from=192.168.4.5 May 17 15:40:56 client-5 rshd[19918]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 15:40:56 client-5 xinetd[3391]: EXIT: shell status=0 pid=19917 duration=0(sec) May 17 15:40:56 client-5 xinetd[3391]: START: shell pid=20094 from=192.168.4.5 May 17 15:40:56 client-5 rshd[20095]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 82b: CMD: mkdir cross-node dir \(fail mds with name\) ============================ 15:40:56 \(1305672056\)");echo XXRETCODE:$?' May 17 15:40:56 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 82b: CMD: mkdir cross-node dir (fail mds with name) ============================ 15:40:56 (1305672056) May 17 15:40:56 client-5 xinetd[3391]: EXIT: shell status=0 pid=20094 duration=0(sec) May 17 15:40:56 client-5 xinetd[3391]: START: shell pid=20211 from=192.168.4.5 May 17 15:40:56 client-5 rshd[20214]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark SKIP: replay-single test_82b needs \>= 2 MDTs");echo XXRETCODE:$?' May 17 15:40:56 client-5 kernel: Lustre: DEBUG MARKER: SKIP: replay-single test_82b needs >= 2 MDTs May 17 15:40:56 client-5 xinetd[3391]: EXIT: shell status=0 pid=20211 duration=0(sec) May 17 15:40:56 client-5 xinetd[3391]: START: shell pid=20304 from=192.168.4.5 May 17 15:40:56 client-5 rshd[20305]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 15:40:56 client-5 xinetd[3391]: EXIT: shell status=0 pid=20304 duration=0(sec) May 17 15:40:57 client-5 xinetd[3391]: START: shell pid=20477 from=192.168.4.5 May 17 15:40:57 client-5 rshd[20478]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 83a: fail log_add during unlink recovery ======================================= 15:40:57 \(1305672057\)");echo XXRETCODE:$?' May 17 15:40:57 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 83a: fail log_add during unlink recovery ======================================= 15:40:57 (1305672057) May 17 15:40:57 client-5 xinetd[3391]: EXIT: shell status=0 pid=20477 duration=0(sec) May 17 15:40:57 client-5 xinetd[3391]: START: shell pid=20579 from=192.168.4.5 May 17 15:40:57 client-5 rshd[20580]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 15:40:57 client-5 xinetd[3391]: EXIT: shell status=0 pid=20579 duration=0(sec) May 17 15:40:57 client-5 xinetd[3391]: START: shell pid=20751 from=192.168.4.5 May 17 15:40:57 client-5 rshd[20752]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 83b: fail log_add during unlink recovery ======================================= 15:40:57 \(1305672057\)");echo XXRETCODE:$?' May 17 15:40:57 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 83b: fail log_add during unlink recovery ======================================= 15:40:57 (1305672057) May 17 15:40:57 client-5 xinetd[3391]: EXIT: shell status=0 pid=20751 duration=0(sec) May 17 15:40:57 client-5 xinetd[3391]: START: shell pid=20804 from=192.168.4.5 May 17 15:40:58 client-5 rshd[20805]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 15:40:58 client-5 xinetd[3391]: EXIT: shell status=0 pid=20804 duration=1(sec) May 17 15:40:58 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 15:41:10 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff810333eb4400: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 15:41:10 client-5 kernel: Lustre: Skipped 11 previous similar messages May 17 15:41:16 client-5 kernel: Lustre: 20198:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369045627537041 sent from MGC192.168.4.128@o2ib to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305672070] [real_sent 1305672070] [current 1305672076] [deadline 6s] [delay 0s] req@ffff8102fd196000 x1369045627537041/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 368/392 e 0 to 1 dl 1305672076 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 15:41:16 client-5 kernel: Lustre: 20198:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 24 previous similar messages May 17 15:41:18 client-5 xinetd[3391]: START: shell pid=20999 from=192.168.4.5 May 17 15:41:18 client-5 rshd[21000]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 15:41:20 client-5 kernel: Lustre: 20198:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74f22605 to 0x62e5a8d74f22f2e May 17 15:41:20 client-5 kernel: LustreError: 20198:0:(client.c:2570:ptlrpc_replay_interpret()) @@@ status -116, old was 0 req@ffff81030d051c00 x1369045627536981/t365072220216(365072220216) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:23/10 lens 360/424 e 0 to 0 dl 1305672092 ref 2 fl Interpret:R/ffffffff/ffffffff rc -116/-1 May 17 15:41:20 client-5 xinetd[3391]: EXIT: shell status=0 pid=20999 duration=2(sec) May 17 15:41:20 client-5 xinetd[3391]: START: shell pid=21091 from=192.168.4.5 May 17 15:41:20 client-5 rshd[21092]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 15:41:20 client-5 xinetd[3391]: EXIT: shell status=0 pid=21091 duration=0(sec) May 17 15:41:21 client-5 xinetd[3391]: START: shell pid=21263 from=192.168.4.5 May 17 15:41:21 client-5 rshd[21264]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 84a: stale open during export disconnect ======================================= 15:41:20 \(1305672080\)");echo XXRETCODE:$?' May 17 15:41:21 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 84a: stale open during export disconnect ======================================= 15:41:20 (1305672080) May 17 15:41:21 client-5 xinetd[3391]: EXIT: shell status=0 pid=21263 duration=0(sec) May 17 15:41:30 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The obd_ping operation failed with -107 May 17 15:41:30 client-5 kernel: LustreError: Skipped 38 previous similar messages May 17 15:41:30 client-5 kernel: LustreError: 167-0: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. May 17 15:41:30 client-5 kernel: LustreError: 21306:0:(mdc_locks.c:722:mdc_enqueue()) ldlm_cli_enqueue: -4 May 17 15:41:30 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff810333eb4400: Connection restored to service lustre-MDT0000 using nid 192.168.4.128@o2ib. May 17 15:41:30 client-5 kernel: Lustre: Skipped 23 previous similar messages May 17 15:41:31 client-5 xinetd[3391]: START: shell pid=21376 from=192.168.4.5 May 17 15:41:31 client-5 rshd[21378]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 15:41:31 client-5 xinetd[3391]: EXIT: shell status=0 pid=21376 duration=0(sec) May 17 15:41:32 client-5 xinetd[3391]: START: shell pid=21549 from=192.168.4.5 May 17 15:41:32 client-5 rshd[21550]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 85a: check the cancellation of unused locks during recovery\(IBITS\) ============= 15:41:31 \(1305672091\)");echo XXRETCODE:$?' May 17 15:41:32 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 85a: check the cancellation of unused locks during recovery(IBITS) ============= 15:41:31 (1305672091) May 17 15:41:32 client-5 xinetd[3391]: EXIT: shell status=0 pid=21549 duration=0(sec) May 17 15:41:47 client-5 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail. May 17 15:41:47 client-5 kernel: LustreError: Skipped 11 previous similar messages May 17 15:41:52 client-5 xinetd[3391]: START: shell pid=21812 from=192.168.4.5 May 17 15:41:52 client-5 rshd[21813]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 15:42:02 client-5 kernel: Lustre: 20198:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74f22f2e to 0x62e5a8d74f2532a May 17 15:42:02 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 15:42:02 client-5 kernel: Lustre: Skipped 12 previous similar messages May 17 15:42:02 client-5 xinetd[3391]: EXIT: shell status=0 pid=21812 duration=10(sec) May 17 15:42:02 client-5 xinetd[3391]: START: shell pid=21904 from=192.168.4.5 May 17 15:42:02 client-5 rshd[21905]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 15:42:02 client-5 xinetd[3391]: EXIT: shell status=0 pid=21904 duration=0(sec) May 17 15:42:02 client-5 xinetd[3391]: START: shell pid=22075 from=192.168.4.5 May 17 15:42:03 client-5 rshd[22077]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 85b: check the cancellation of unused locks during recovery\(EXTENT\) ============ 15:42:02 \(1305672122\)");echo XXRETCODE:$?' May 17 15:42:03 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 85b: check the cancellation of unused locks during recovery(EXTENT) ============ 15:42:02 (1305672122) May 17 15:42:03 client-5 xinetd[3391]: EXIT: shell status=0 pid=22075 duration=1(sec) May 17 15:42:06 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 15:42:07 client-5 kernel: Lustre: Increasing default stripe size to min 1048576 May 17 15:42:08 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 15:42:22 client-5 xinetd[3391]: START: shell pid=22461 from=192.168.4.5 May 17 15:42:22 client-5 rshd[22462]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 15:42:31 client-5 xinetd[3391]: EXIT: shell status=0 pid=22461 duration=9(sec) May 17 15:42:31 client-5 xinetd[3391]: START: shell pid=22551 from=192.168.4.5 May 17 15:42:31 client-5 rshd[22553]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 15:42:31 client-5 xinetd[3391]: EXIT: shell status=0 pid=22551 duration=0(sec) May 17 15:42:31 client-5 xinetd[3391]: START: shell pid=22723 from=192.168.4.5 May 17 15:42:31 client-5 rshd[22725]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 86: umount server after clear nid_stats should not hit LBUG ==================== 15:42:31 \(1305672151\)");echo XXRETCODE:$?' May 17 15:42:31 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 86: umount server after clear nid_stats should not hit LBUG ==================== 15:42:31 (1305672151) May 17 15:42:31 client-5 xinetd[3391]: EXIT: shell status=0 pid=22723 duration=0(sec) May 17 15:42:31 client-5 xinetd[3391]: START: shell pid=22766 from=192.168.4.5 May 17 15:42:31 client-5 rshd[22767]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:; lsof -t /mnt/lustre || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre is still busy, wait one second && sleep 1; umount /mnt/lustre; fi fi");echo XXRETCODE:$?' May 17 15:42:32 client-5 kernel: LustreError: 22808:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 15:42:32 client-5 kernel: LustreError: 22808:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Skipped 3 previous similar messages May 17 15:42:32 client-5 kernel: LustreError: 22808:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 15:42:32 client-5 kernel: LustreError: 22808:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) Skipped 3 previous similar messages May 17 15:42:32 client-5 kernel: Lustre: client ffff810333eb4400 umount complete May 17 15:42:32 client-5 xinetd[3391]: EXIT: shell status=0 pid=22766 duration=1(sec) May 17 15:42:42 client-5 xinetd[3391]: START: shell pid=22882 from=192.168.4.5 May 17 15:42:42 client-5 rshd[22883]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c " running=\$(mount | grep -c /mnt/lustre' '); rc=0; if [ \$running -eq 0 ] ; then mkdir -p /mnt/lustre; mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre; rc=\$?; fi; exit \$rc");echo XXRETCODE:$?' May 17 15:42:42 client-5 kernel: Lustre: 22922:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 15:42:42 client-5 kernel: Lustre: 22922:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 7 previous similar messages May 17 15:42:42 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 15:42:42 client-5 kernel: Lustre: Increasing default stripe size to min 1048576 May 17 15:42:42 client-5 kernel: Lustre: Client lustre-client has started May 17 15:42:42 client-5 xinetd[3391]: EXIT: shell status=0 pid=22882 duration=0(sec) May 17 15:42:42 client-5 xinetd[3391]: START: shell pid=22942 from=192.168.4.5 May 17 15:42:42 client-5 rshd[22943]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mount | grep -w /mnt/lustre");echo XXRETCODE:$?' May 17 15:42:42 client-5 xinetd[3391]: EXIT: shell status=0 pid=22942 duration=0(sec) May 17 15:42:42 client-5 xinetd[3391]: START: shell pid=22986 from=192.168.4.5 May 17 15:42:42 client-5 rshd[22987]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param debug=\"0x33f0406\"; lctl set_param subsystem_debug=\"0xffb7e3ff\"; lctl set_param debug_mb=32;");echo XXRETCODE:$?' May 17 15:42:42 client-5 kernel: Lustre: 23021:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 15:42:42 client-5 kernel: Lustre: 23021:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message May 17 15:42:42 client-5 xinetd[3391]: EXIT: shell status=0 pid=22986 duration=0(sec) May 17 15:42:42 client-5 xinetd[3391]: START: shell pid=23078 from=192.168.4.5 May 17 15:42:42 client-5 rshd[23080]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 15:42:42 client-5 xinetd[3391]: EXIT: shell status=0 pid=23078 duration=0(sec) May 17 15:42:43 client-5 xinetd[3391]: START: shell pid=23251 from=192.168.4.5 May 17 15:42:43 client-5 rshd[23252]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 87: write replay =============================================================== 15:42:43 \(1305672163\)");echo XXRETCODE:$?' May 17 15:42:43 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 87: write replay =============================================================== 15:42:43 (1305672163) May 17 15:42:43 client-5 xinetd[3391]: EXIT: shell status=0 pid=23251 duration=0(sec) May 17 15:42:43 client-5 xinetd[3391]: START: shell pid=23310 from=192.168.4.5 May 17 15:42:43 client-5 rshd[23311]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 15:42:43 client-5 xinetd[3391]: EXIT: shell status=0 pid=23310 duration=0(sec) May 17 15:42:44 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-OST0000 May 17 15:42:45 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 15:42:45 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 15:42:59 client-5 xinetd[3391]: START: shell pid=23511 from=192.168.4.5 May 17 15:42:59 client-5 rshd[23512]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 15:43:02 client-5 xinetd[3391]: EXIT: shell status=0 pid=23511 duration=3(sec) May 17 15:43:02 client-5 xinetd[3391]: START: shell pid=23605 from=192.168.4.5 May 17 15:43:02 client-5 rshd[23606]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 15:43:02 client-5 xinetd[3391]: EXIT: shell status=0 pid=23605 duration=0(sec) May 17 15:43:03 client-5 xinetd[3391]: START: shell pid=23777 from=192.168.4.5 May 17 15:43:03 client-5 rshd[23778]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 87b: write replay with changed data \(checksum resend\) ========================== 15:43:03 \(1305672183\)");echo XXRETCODE:$?' May 17 15:43:03 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 87b: write replay with changed data (checksum resend) ========================== 15:43:03 (1305672183) May 17 15:43:03 client-5 xinetd[3391]: EXIT: shell status=0 pid=23777 duration=0(sec) May 17 15:43:03 client-5 xinetd[3391]: START: shell pid=23836 from=192.168.4.5 May 17 15:43:03 client-5 rshd[23837]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 15:43:03 client-5 xinetd[3391]: EXIT: shell status=0 pid=23836 duration=0(sec) May 17 15:43:03 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-OST0000 May 17 15:43:06 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 15:43:06 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 15:43:19 client-5 xinetd[3391]: START: shell pid=24040 from=192.168.4.5 May 17 15:43:19 client-5 rshd[24041]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 15:43:22 client-5 xinetd[3391]: EXIT: shell status=0 pid=24040 duration=3(sec) May 17 15:43:22 client-5 xinetd[3391]: START: shell pid=24134 from=192.168.4.5 May 17 15:43:22 client-5 rshd[24135]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 15:43:22 client-5 xinetd[3391]: EXIT: shell status=0 pid=24134 duration=0(sec) May 17 15:43:23 client-5 xinetd[3391]: START: shell pid=24306 from=192.168.4.5 May 17 15:43:23 client-5 rshd[24307]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 88: MDS should not assign same objid to different files ======================== 15:43:23 \(1305672203\)");echo XXRETCODE:$?' May 17 15:43:23 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 88: MDS should not assign same objid to different files ======================== 15:43:23 (1305672203) May 17 15:43:23 client-5 xinetd[3391]: EXIT: shell status=0 pid=24306 duration=0(sec) May 17 15:43:23 client-5 xinetd[3391]: START: shell pid=24360 from=192.168.4.5 May 17 15:43:23 client-5 rshd[24361]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 15:43:23 client-5 xinetd[3391]: EXIT: shell status=0 pid=24360 duration=0(sec) May 17 15:43:23 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-OST0000 May 17 15:43:23 client-5 xinetd[3391]: START: shell pid=24439 from=192.168.4.5 May 17 15:43:24 client-5 rshd[24441]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 15:43:24 client-5 xinetd[3391]: EXIT: shell status=0 pid=24439 duration=1(sec) May 17 15:43:24 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 15:43:44 client-5 kernel: Lustre: 20198:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74f26ae6 to 0x62e5a8d74f27504 May 17 15:43:57 client-5 xinetd[3391]: START: shell pid=24828 from=192.168.4.5 May 17 15:43:57 client-5 rshd[24829]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 15:44:00 client-5 xinetd[3391]: EXIT: shell status=0 pid=24828 duration=3(sec) May 17 15:44:02 client-5 xinetd[3391]: START: shell pid=24973 from=192.168.4.5 May 17 15:44:03 client-5 rshd[24974]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 15:44:03 client-5 xinetd[3391]: EXIT: shell status=0 pid=24973 duration=1(sec) May 17 15:44:03 client-5 xinetd[3391]: START: shell pid=25145 from=192.168.4.5 May 17 15:44:03 client-5 rshd[25146]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single test 89: no disk space leak on late ost connection ================================== 15:44:03 \(1305672243\)");echo XXRETCODE:$?' May 17 15:44:03 client-5 kernel: Lustre: DEBUG MARKER: == replay-single test 89: no disk space leak on late ost connection ================================== 15:44:03 (1305672243) May 17 15:44:03 client-5 xinetd[3391]: EXIT: shell status=0 pid=25145 duration=0(sec) May 17 15:44:03 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 15:44:03 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 15:44:36 client-5 kernel: Lustre: 20199:0:(import.c:529:import_select_connection()) MGC192.168.4.128@o2ib: tried all connections, increasing latency to 11s May 17 15:44:36 client-5 kernel: Lustre: 20199:0:(import.c:529:import_select_connection()) Skipped 25 previous similar messages May 17 15:44:36 client-5 kernel: Lustre: 20198:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74f27504 to 0x62e5a8d74f2871e May 17 15:44:37 client-5 kernel: LustreError: 25387:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 15:44:37 client-5 kernel: LustreError: 25387:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Skipped 1 previous similar message May 17 15:44:37 client-5 kernel: LustreError: 25387:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 15:44:37 client-5 kernel: LustreError: 25387:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) Skipped 1 previous similar message May 17 15:44:37 client-5 kernel: Lustre: client ffff810305214c00 umount complete May 17 15:44:37 client-5 kernel: Lustre: 25424:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 15:44:37 client-5 kernel: Lustre: 25424:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 7 previous similar messages May 17 15:44:37 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 15:44:37 client-5 kernel: Lustre: Increasing default stripe size to min 1048576 May 17 15:44:37 client-5 kernel: Lustre: Client lustre-client has started May 17 15:44:37 client-5 kernel: Lustre: 25441:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 15:44:37 client-5 kernel: Lustre: 25441:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message May 17 15:46:07 client-5 xinetd[3391]: START: shell pid=25590 from=192.168.4.5 May 17 15:46:07 client-5 rshd[25591]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 15:46:07 client-5 xinetd[3391]: EXIT: shell status=0 pid=25590 duration=0(sec) May 17 15:46:07 client-5 xinetd[3391]: START: shell pid=25755 from=192.168.4.5 May 17 15:46:07 client-5 rshd[25756]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-single replay-single.sh test complete, duration 4260 sec =================================== 15:46:07 \(1305672367\)");echo XXRETCODE:$?' May 17 15:46:07 client-5 kernel: Lustre: DEBUG MARKER: == replay-single replay-single.sh test complete, duration 4260 sec =================================== 15:46:07 (1305672367) May 17 15:46:07 client-5 xinetd[3391]: EXIT: shell status=0 pid=25755 duration=0(sec) May 17 15:46:08 client-5 xinetd[3391]: START: shell pid=25873 from=192.168.4.5 May 17 15:46:08 client-5 rshd[25874]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark -----============= acceptance-small: conf-sanity ============----- Tue May 17 15:46:07 PDT 2011");echo XXRETCODE:$?' May 17 15:46:08 client-5 kernel: Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue May 17 15:46:07 PDT 2011 May 17 15:46:08 client-5 xinetd[3391]: EXIT: shell status=0 pid=25873 duration=0(sec) May 17 15:46:08 client-5 xinetd[3391]: START: shell pid=26041 from=192.168.4.5 May 17 15:46:08 client-5 rshd[26043]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark excepting tests: ");echo XXRETCODE:$?' May 17 15:46:08 client-5 kernel: Lustre: DEBUG MARKER: excepting tests: May 17 15:46:08 client-5 xinetd[3391]: EXIT: shell status=0 pid=26041 duration=0(sec) May 17 15:46:08 client-5 xinetd[3391]: START: shell pid=26085 from=192.168.4.5 May 17 15:46:08 client-5 rshd[26086]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:; lsof -t /mnt/lustre || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre is still busy, wait one second && sleep 1; umount /mnt/lustre; fi fi");echo XXRETCODE:$?' May 17 15:46:08 client-5 kernel: LustreError: 26127:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 15:46:08 client-5 kernel: LustreError: 26127:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Skipped 1 previous similar message May 17 15:46:08 client-5 kernel: LustreError: 26127:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 15:46:08 client-5 kernel: LustreError: 26127:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) Skipped 1 previous similar message May 17 15:46:08 client-5 kernel: Lustre: client ffff810319676000 umount complete May 17 15:46:08 client-5 xinetd[3391]: EXIT: shell status=0 pid=26085 duration=0(sec) May 17 15:46:08 client-5 xinetd[3391]: START: shell pid=26136 from=192.168.4.5 May 17 15:46:08 client-5 rshd[26137]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:; lsof -t /mnt/lustre2 || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre2 | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre2 is still busy, wait one second && sleep 1; umount /mnt/lustre2; fi fi");echo XXRETCODE:$?' May 17 15:46:08 client-5 xinetd[3391]: EXIT: shell status=0 pid=26136 duration=0(sec) May 17 15:46:33 client-5 kernel: Lustre: 26395:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 15:46:33 client-5 kernel: Lustre: 26395:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message May 17 15:46:45 client-5 xinetd[3391]: START: shell pid=26991 from=192.168.4.5 May 17 15:46:45 client-5 rshd[26992]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark ost now in FULL state");echo XXRETCODE:$?' May 17 15:46:45 client-5 kernel: Lustre: DEBUG MARKER: ost now in FULL state May 17 15:46:45 client-5 xinetd[3391]: EXIT: shell status=0 pid=26991 duration=0(sec) May 17 15:46:57 client-5 xinetd[3391]: START: shell pid=27159 from=192.168.4.5 May 17 15:46:57 client-5 rshd[27161]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == conf-sanity test 0: single mount setup ============================================================ 15:46:57 \(1305672417\)");echo XXRETCODE:$?' May 17 15:46:57 client-5 kernel: Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ============================================================ 15:46:57 (1305672417) May 17 15:46:57 client-5 xinetd[3391]: EXIT: shell status=0 pid=27159 duration=0(sec) May 17 15:46:58 client-5 kernel: Lustre: 27287:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 15:46:58 client-5 kernel: Lustre: 27287:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 7 previous similar messages May 17 15:46:58 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 15:47:08 client-5 kernel: Lustre: 27287:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-OST0000-osc-ffff81031812e400->192.168.4.129@o2ib netid 50000: select flavor null May 17 15:47:08 client-5 kernel: Lustre: 27287:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 1 previous similar message May 17 15:47:08 client-5 kernel: Lustre: Client lustre-client has started May 17 15:47:08 client-5 kernel: Lustre: 27304:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 15:47:08 client-5 kernel: Lustre: 27304:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message May 17 15:47:08 client-5 kernel: LustreError: 27344:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 15:47:08 client-5 kernel: LustreError: 27344:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Skipped 1 previous similar message May 17 15:47:08 client-5 kernel: LustreError: 27344:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 15:47:08 client-5 kernel: LustreError: 27344:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) Skipped 1 previous similar message May 17 15:47:08 client-5 kernel: Lustre: client ffff81031812e400 umount complete May 17 15:47:29 client-5 kernel: Lustre: Acceptor stopping May 17 15:47:30 client-5 kernel: Lustre: Removed LNI 10.10.4.5@tcp May 17 15:47:32 client-5 kernel: Lustre: Removed LNI 192.168.4.5@o2ib May 17 15:47:33 client-5 xinetd[3391]: START: shell pid=27800 from=192.168.4.5 May 17 15:47:33 client-5 rshd[27801]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 15:47:33 client-5 xinetd[3391]: EXIT: shell status=0 pid=27800 duration=0(sec) May 17 15:47:34 client-5 kernel: Lustre: 27936:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 15:47:34 client-5 kernel: Lustre: 27939:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 15:47:34 client-5 kernel: Lustre: OBD class driver, http://www.lustre.org/ May 17 15:47:34 client-5 kernel: Lustre: Lustre Version: 2.0.60 May 17 15:47:34 client-5 kernel: Lustre: Build Version: jenkins-g985ac5b-PRISTINE-2.6.18-194.17.1.el5 May 17 15:47:34 client-5 kernel: Lustre: Lustre LU module (ffffffff888eb1c0). May 17 15:47:34 client-5 kernel: Lustre: Register global MR array, MR size: 0xffffffffffffffff, array size: 1 May 17 15:47:34 client-5 kernel: Lustre: Added LNI 192.168.4.5@o2ib [8/64/0/180] May 17 15:47:34 client-5 kernel: Lustre: Added LNI 10.10.4.5@tcp [8/256/0/180] May 17 15:47:34 client-5 kernel: Lustre: Accept all, port 7988 May 17 15:47:35 client-5 kernel: Lustre: Lustre OSC module (ffffffff88b3c6c0). May 17 15:47:35 client-5 kernel: Lustre: Lustre LOV module (ffffffff88bab540). May 17 15:47:35 client-5 kernel: Lustre: Lustre client module (ffffffff88c643a0). May 17 15:47:35 client-5 xinetd[3391]: START: shell pid=28338 from=192.168.4.5 May 17 15:47:35 client-5 rshd[28339]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == conf-sanity test 1: start up ost twice \(should return errors\) ===================================== 15:47:34 \(1305672454\)");echo XXRETCODE:$?' May 17 15:47:35 client-5 kernel: Lustre: DEBUG MARKER: == conf-sanity test 1: start up ost twice (should return errors) ===================================== 15:47:34 (1305672454) May 17 15:47:35 client-5 xinetd[3391]: EXIT: shell status=0 pid=28338 duration=0(sec) May 17 15:47:37 client-5 kernel: Lustre: 28493:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 15:47:37 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 15:47:37 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 15:47:37 client-5 kernel: Lustre: 28493:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff810313de1000->192.168.4.128@o2ib netid 50000: select flavor null May 17 15:47:37 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 15:47:42 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 15:47:47 client-5 kernel: Lustre: 28493:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-OST0000-osc-ffff810313de1000->192.168.4.129@o2ib netid 50000: select flavor null May 17 15:47:47 client-5 kernel: Lustre: Client lustre-client has started May 17 15:47:47 client-5 kernel: Lustre: 28510:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 15:47:47 client-5 kernel: LustreError: 28550:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 15:47:47 client-5 kernel: LustreError: 28550:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 15:47:47 client-5 kernel: LustreError: 28550:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 15:47:47 client-5 kernel: LustreError: 28550:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 15:47:47 client-5 kernel: Lustre: client ffff810313de1000 umount complete May 17 15:48:09 client-5 kernel: Lustre: Acceptor stopping May 17 15:48:10 client-5 kernel: Lustre: Removed LNI 10.10.4.5@tcp May 17 15:48:12 client-5 kernel: Lustre: Removed LNI 192.168.4.5@o2ib May 17 15:48:13 client-5 xinetd[3391]: START: shell pid=29003 from=192.168.4.5 May 17 15:48:13 client-5 rshd[29004]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 15:48:13 client-5 xinetd[3391]: EXIT: shell status=0 pid=29003 duration=0(sec) May 17 15:48:13 client-5 kernel: Lustre: 29139:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 15:48:13 client-5 kernel: Lustre: 29142:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 15:48:14 client-5 kernel: Lustre: OBD class driver, http://www.lustre.org/ May 17 15:48:14 client-5 kernel: Lustre: Lustre Version: 2.0.60 May 17 15:48:14 client-5 kernel: Lustre: Build Version: jenkins-g985ac5b-PRISTINE-2.6.18-194.17.1.el5 May 17 15:48:14 client-5 kernel: Lustre: Lustre LU module (ffffffff888eb1c0). May 17 15:48:14 client-5 kernel: Lustre: Register global MR array, MR size: 0xffffffffffffffff, array size: 1 May 17 15:48:14 client-5 kernel: Lustre: Added LNI 192.168.4.5@o2ib [8/64/0/180] May 17 15:48:14 client-5 kernel: Lustre: Added LNI 10.10.4.5@tcp [8/256/0/180] May 17 15:48:14 client-5 kernel: Lustre: Accept all, port 7988 May 17 15:48:14 client-5 kernel: Lustre: Lustre OSC module (ffffffff88b3c6c0). May 17 15:48:14 client-5 kernel: Lustre: Lustre LOV module (ffffffff88bab540). May 17 15:48:14 client-5 kernel: Lustre: Lustre client module (ffffffff88c643a0). May 17 15:48:14 client-5 xinetd[3391]: START: shell pid=29541 from=192.168.4.5 May 17 15:48:14 client-5 rshd[29542]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == conf-sanity test 2: start up mds twice \(should return err\) ======================================== 15:48:13 \(1305672493\)");echo XXRETCODE:$?' May 17 15:48:14 client-5 kernel: Lustre: DEBUG MARKER: == conf-sanity test 2: start up mds twice (should return err) ======================================== 15:48:13 (1305672493) May 17 15:48:14 client-5 xinetd[3391]: EXIT: shell status=0 pid=29541 duration=0(sec) May 17 15:48:16 client-5 kernel: Lustre: 29699:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 15:48:16 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 15:48:16 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 15:48:16 client-5 kernel: Lustre: 29699:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff8102f0a88000->192.168.4.128@o2ib netid 50000: select flavor null May 17 15:48:16 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 15:48:21 client-5 kernel: Lustre: 29699:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-OST0000-osc-ffff8102f0a88000->192.168.4.129@o2ib netid 50000: select flavor null May 17 15:48:21 client-5 kernel: Lustre: Client lustre-client has started May 17 15:48:21 client-5 kernel: Lustre: 29716:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 15:48:21 client-5 kernel: LustreError: 29756:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 15:48:21 client-5 kernel: LustreError: 29756:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 15:48:21 client-5 kernel: LustreError: 29756:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 15:48:21 client-5 kernel: LustreError: 29756:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 15:48:21 client-5 kernel: Lustre: client ffff8102f0a88000 umount complete May 17 15:48:43 client-5 kernel: Lustre: Acceptor stopping May 17 15:48:44 client-5 kernel: Lustre: Removed LNI 10.10.4.5@tcp May 17 15:48:46 client-5 kernel: Lustre: Removed LNI 192.168.4.5@o2ib May 17 15:48:47 client-5 xinetd[3391]: START: shell pid=30211 from=192.168.4.5 May 17 15:48:47 client-5 rshd[30212]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 15:48:47 client-5 xinetd[3391]: EXIT: shell status=0 pid=30211 duration=0(sec) May 17 15:48:47 client-5 kernel: Lustre: 30347:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 15:48:47 client-5 kernel: Lustre: 30349:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 15:48:47 client-5 kernel: Lustre: OBD class driver, http://www.lustre.org/ May 17 15:48:47 client-5 kernel: Lustre: Lustre Version: 2.0.60 May 17 15:48:47 client-5 kernel: Lustre: Build Version: jenkins-g985ac5b-PRISTINE-2.6.18-194.17.1.el5 May 17 15:48:47 client-5 kernel: Lustre: Lustre LU module (ffffffff888eb1c0). May 17 15:48:48 client-5 kernel: Lustre: Register global MR array, MR size: 0xffffffffffffffff, array size: 1 May 17 15:48:48 client-5 kernel: Lustre: Added LNI 192.168.4.5@o2ib [8/64/0/180] May 17 15:48:48 client-5 kernel: Lustre: Added LNI 10.10.4.5@tcp [8/256/0/180] May 17 15:48:48 client-5 kernel: Lustre: Accept all, port 7988 May 17 15:48:48 client-5 kernel: Lustre: Lustre OSC module (ffffffff88b3c6c0). May 17 15:48:48 client-5 kernel: Lustre: Lustre LOV module (ffffffff88bab540). May 17 15:48:48 client-5 kernel: Lustre: Lustre client module (ffffffff88c643a0). May 17 15:48:48 client-5 xinetd[3391]: START: shell pid=30752 from=192.168.4.5 May 17 15:48:48 client-5 rshd[30753]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == conf-sanity test 3: mount client twice \(should return err\) ======================================== 15:48:47 \(1305672527\)");echo XXRETCODE:$?' May 17 15:48:48 client-5 kernel: Lustre: DEBUG MARKER: == conf-sanity test 3: mount client twice (should return err) ======================================== 15:48:47 (1305672527) May 17 15:48:48 client-5 xinetd[3391]: EXIT: shell status=0 pid=30752 duration=0(sec) May 17 15:48:49 client-5 kernel: Lustre: 30882:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 15:48:49 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 15:48:49 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 15:48:49 client-5 kernel: Lustre: 30882:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff81030f648c00->192.168.4.128@o2ib netid 50000: select flavor null May 17 15:48:49 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 15:48:54 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 15:48:59 client-5 kernel: Lustre: 30882:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-OST0000-osc-ffff81030f648c00->192.168.4.129@o2ib netid 50000: select flavor null May 17 15:48:59 client-5 kernel: Lustre: Client lustre-client has started May 17 15:48:59 client-5 kernel: Lustre: 30899:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 15:49:00 client-5 kernel: LustreError: 30949:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 15:49:00 client-5 kernel: LustreError: 30949:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 15:49:00 client-5 kernel: LustreError: 30949:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 15:49:00 client-5 kernel: LustreError: 30949:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 15:49:00 client-5 kernel: Lustre: client ffff81030f648c00 umount complete May 17 15:49:21 client-5 kernel: Lustre: Acceptor stopping May 17 15:49:22 client-5 kernel: Lustre: Removed LNI 10.10.4.5@tcp May 17 15:49:24 client-5 kernel: Lustre: Removed LNI 192.168.4.5@o2ib May 17 15:49:26 client-5 xinetd[3391]: START: shell pid=31401 from=192.168.4.5 May 17 15:49:26 client-5 rshd[31402]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 15:49:26 client-5 xinetd[3391]: EXIT: shell status=0 pid=31401 duration=0(sec) May 17 15:49:26 client-5 kernel: Lustre: 31537:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 15:49:26 client-5 kernel: Lustre: 31539:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 15:49:26 client-5 kernel: Lustre: OBD class driver, http://www.lustre.org/ May 17 15:49:26 client-5 kernel: Lustre: Lustre Version: 2.0.60 May 17 15:49:26 client-5 kernel: Lustre: Build Version: jenkins-g985ac5b-PRISTINE-2.6.18-194.17.1.el5 May 17 15:49:26 client-5 kernel: Lustre: Lustre LU module (ffffffff888eb1c0). May 17 15:49:26 client-5 kernel: Lustre: Register global MR array, MR size: 0xffffffffffffffff, array size: 1 May 17 15:49:26 client-5 kernel: Lustre: Added LNI 192.168.4.5@o2ib [8/64/0/180] May 17 15:49:26 client-5 kernel: Lustre: Added LNI 10.10.4.5@tcp [8/256/0/180] May 17 15:49:26 client-5 kernel: Lustre: Accept all, port 7988 May 17 15:49:26 client-5 kernel: Lustre: Lustre OSC module (ffffffff88b3c6c0). May 17 15:49:26 client-5 kernel: Lustre: Lustre LOV module (ffffffff88bab540). May 17 15:49:26 client-5 kernel: Lustre: Lustre client module (ffffffff88c643a0). May 17 15:49:26 client-5 xinetd[3391]: START: shell pid=31939 from=192.168.4.5 May 17 15:49:27 client-5 rshd[31940]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == conf-sanity test 4: force cleanup ost, then cleanup =============================================== 15:49:26 \(1305672566\)");echo XXRETCODE:$?' May 17 15:49:27 client-5 kernel: Lustre: DEBUG MARKER: == conf-sanity test 4: force cleanup ost, then cleanup =============================================== 15:49:26 (1305672566) May 17 15:49:27 client-5 xinetd[3391]: EXIT: shell status=0 pid=31939 duration=1(sec) May 17 15:49:28 client-5 kernel: Lustre: 32072:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 15:49:28 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 15:49:28 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 15:49:28 client-5 kernel: Lustre: 32072:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff81030805d000->192.168.4.128@o2ib netid 50000: select flavor null May 17 15:49:28 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 15:49:33 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 15:49:38 client-5 kernel: Lustre: 32072:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-OST0000-osc-ffff81030805d000->192.168.4.129@o2ib netid 50000: select flavor null May 17 15:49:38 client-5 kernel: Lustre: Client lustre-client has started May 17 15:49:38 client-5 kernel: Lustre: 32089:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 15:49:47 client-5 kernel: Lustre: 32144:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369096916566044 sent from lustre-OST0000-osc-ffff81030805d000 to NID 192.168.4.129@o2ib has timed out for slow reply: [sent 1305672581] [real_sent 1305672581] [current 1305672587] [deadline 6s] [delay 0s] req@ffff810319d5c000 x1369096916566044/t0(0) o-1->lustre-OST0000_UUID@192.168.4.129@o2ib:28/4 lens 192/192 e 0 to 1 dl 1305672587 ref 2 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 15:49:47 client-5 kernel: LustreError: 32144:0:(lov_obd.c:334:lov_disconnect_obd()) Target lustre-OST0000_UUID disconnect error -110 May 17 15:49:47 client-5 kernel: LustreError: 32144:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 15:49:47 client-5 kernel: LustreError: 32144:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 15:49:47 client-5 kernel: LustreError: 32144:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 15:49:47 client-5 kernel: LustreError: 32144:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 15:49:47 client-5 kernel: Lustre: client ffff81030805d000 umount complete May 17 15:50:00 client-5 kernel: Lustre: Acceptor stopping May 17 15:50:01 client-5 kernel: Lustre: Removed LNI 10.10.4.5@tcp May 17 15:50:03 client-5 kernel: Lustre: Removed LNI 192.168.4.5@o2ib May 17 15:50:04 client-5 xinetd[3391]: START: shell pid=32592 from=192.168.4.5 May 17 15:50:04 client-5 rshd[32593]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 15:50:04 client-5 xinetd[3391]: EXIT: shell status=0 pid=32592 duration=0(sec) May 17 15:50:05 client-5 kernel: Lustre: 32729:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 15:50:05 client-5 kernel: Lustre: 32730:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 15:50:05 client-5 kernel: Lustre: OBD class driver, http://www.lustre.org/ May 17 15:50:05 client-5 kernel: Lustre: Lustre Version: 2.0.60 May 17 15:50:05 client-5 kernel: Lustre: Build Version: jenkins-g985ac5b-PRISTINE-2.6.18-194.17.1.el5 May 17 15:50:05 client-5 kernel: Lustre: Lustre LU module (ffffffff888eb1c0). May 17 15:50:05 client-5 kernel: Lustre: Register global MR array, MR size: 0xffffffffffffffff, array size: 1 May 17 15:50:05 client-5 kernel: Lustre: Added LNI 192.168.4.5@o2ib [8/64/0/180] May 17 15:50:05 client-5 kernel: Lustre: Added LNI 10.10.4.5@tcp [8/256/0/180] May 17 15:50:05 client-5 kernel: Lustre: Accept all, port 7988 May 17 15:50:05 client-5 kernel: Lustre: Lustre OSC module (ffffffff88b3c6c0). May 17 15:50:05 client-5 kernel: Lustre: Lustre LOV module (ffffffff88bab540). May 17 15:50:05 client-5 kernel: Lustre: Lustre client module (ffffffff88c643a0). May 17 15:50:05 client-5 xinetd[3391]: START: shell pid=688 from=192.168.4.5 May 17 15:50:05 client-5 rshd[689]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == conf-sanity test 5a: force cleanup mds, then cleanup ============================================== 15:50:04 \(1305672604\)");echo XXRETCODE:$?' May 17 15:50:05 client-5 kernel: Lustre: DEBUG MARKER: == conf-sanity test 5a: force cleanup mds, then cleanup ============================================== 15:50:04 (1305672604) May 17 15:50:05 client-5 xinetd[3391]: EXIT: shell status=0 pid=688 duration=0(sec) May 17 15:50:07 client-5 kernel: Lustre: 820:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 15:50:07 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 15:50:07 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 15:50:07 client-5 kernel: Lustre: 820:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff8102eed4c800->192.168.4.128@o2ib netid 50000: select flavor null May 17 15:50:07 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 15:50:12 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 15:50:17 client-5 kernel: Lustre: 820:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-OST0000-osc-ffff8102eed4c800->192.168.4.129@o2ib netid 50000: select flavor null May 17 15:50:17 client-5 kernel: Lustre: Client lustre-client has started May 17 15:50:17 client-5 kernel: Lustre: 837:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 15:50:30 client-5 kernel: LustreError: 875:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 15:50:30 client-5 kernel: LustreError: 875:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 15:50:34 client-5 kernel: Lustre: 367:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369096957460509 sent from MGC192.168.4.128@o2ib to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305672627] [real_sent 1305672627] [current 1305672634] [deadline 7s] [delay 0s] req@ffff8103171fbc00 x1369096957460509/t0(0) o-1->MGS@MGC192.168.4.128@o2ib_0:26/25 lens 192/192 e 0 to 1 dl 1305672634 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 15:50:34 client-5 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail. May 17 15:50:34 client-5 kernel: Lustre: 367:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369096957460510 sent from lustre-MDT0000-mdc-ffff8102eed4c800 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305672627] [real_sent 1305672627] [current 1305672634] [deadline 7s] [delay 0s] req@ffff81032b300000 x1369096957460510/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 192/192 e 0 to 1 dl 1305672634 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 15:50:36 client-5 kernel: Lustre: 875:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369096957460514 sent from lustre-MDT0000-mdc-ffff8102eed4c800 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305672630] [real_sent 1305672630] [current 1305672636] [deadline 6s] [delay 0s] req@ffff8102f989c800 x1369096957460514/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 192/192 e 0 to 1 dl 1305672636 ref 2 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 15:50:36 client-5 kernel: LustreError: 875:0:(lmv_obd.c:662:lmv_disconnect_mdc()) Target lustre-MDT0000_UUID disconnect error -110 May 17 15:50:37 client-5 kernel: Lustre: client ffff8102eed4c800 umount complete May 17 15:50:48 client-5 kernel: Lustre: Acceptor stopping May 17 15:50:49 client-5 kernel: Lustre: Removed LNI 10.10.4.5@tcp May 17 15:50:51 client-5 kernel: Lustre: Removed LNI 192.168.4.5@o2ib May 17 15:50:53 client-5 xinetd[3391]: START: shell pid=1335 from=192.168.4.5 May 17 15:50:53 client-5 rshd[1337]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 15:50:53 client-5 xinetd[3391]: EXIT: shell status=0 pid=1335 duration=0(sec) May 17 15:50:53 client-5 kernel: Lustre: 1472:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 15:50:53 client-5 kernel: Lustre: 1473:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 15:50:53 client-5 kernel: Lustre: OBD class driver, http://www.lustre.org/ May 17 15:50:53 client-5 kernel: Lustre: Lustre Version: 2.0.60 May 17 15:50:53 client-5 kernel: Lustre: Build Version: jenkins-g985ac5b-PRISTINE-2.6.18-194.17.1.el5 May 17 15:50:53 client-5 kernel: Lustre: Lustre LU module (ffffffff888eb1c0). May 17 15:50:53 client-5 kernel: Lustre: Register global MR array, MR size: 0xffffffffffffffff, array size: 1 May 17 15:50:53 client-5 kernel: Lustre: Added LNI 192.168.4.5@o2ib [8/64/0/180] May 17 15:50:53 client-5 kernel: Lustre: Added LNI 10.10.4.5@tcp [8/256/0/180] May 17 15:50:53 client-5 kernel: Lustre: Accept all, port 7988 May 17 15:50:54 client-5 kernel: Lustre: Lustre OSC module (ffffffff88b3c6c0). May 17 15:50:54 client-5 kernel: Lustre: Lustre LOV module (ffffffff88bab540). May 17 15:50:54 client-5 kernel: Lustre: Lustre client module (ffffffff88c643a0). May 17 15:50:54 client-5 xinetd[3391]: START: shell pid=1879 from=192.168.4.5 May 17 15:50:54 client-5 rshd[1880]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == conf-sanity test 5b: Try to start a client with no MGS \(should return errs\) ======================= 15:50:53 \(1305672653\)");echo XXRETCODE:$?' May 17 15:50:54 client-5 kernel: Lustre: DEBUG MARKER: == conf-sanity test 5b: Try to start a client with no MGS (should return errs) ======================= 15:50:53 (1305672653) May 17 15:50:54 client-5 xinetd[3391]: EXIT: shell status=0 pid=1879 duration=0(sec) May 17 15:51:17 client-5 kernel: Lustre: 1977:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 15:51:17 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 15:51:22 client-5 kernel: Lustre: 1579:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369097007792129 sent from MGC192.168.4.128@o2ib to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305672677] [real_sent 1305672677] [current 1305672682] [deadline 5s] [delay 0s] req@ffff8102c72f6400 x1369097007792129/t0(0) o-1->MGS@MGC192.168.4.128@o2ib_0:26/25 lens 368/392 e 0 to 1 dl 1305672682 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 15:51:23 client-5 kernel: LustreError: 1977:0:(client.c:1046:ptlrpc_import_delay_req()) @@@ send limit expired req@ffff81032a0f8800 x1369097007792130/t0(0) o-1->MGS@MGC192.168.4.128@o2ib_0:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:W/ffffffff/ffffffff rc 0/-1 May 17 15:51:34 client-5 kernel: LustreError: 1987:0:(client.c:1046:ptlrpc_import_delay_req()) @@@ send limit expired req@ffff81033a620c00 x1369097007792132/t0(0) o-1->MGS@MGC192.168.4.128@o2ib_0:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:W/ffffffff/ffffffff rc 0/-1 May 17 15:51:42 client-5 kernel: Lustre: 1580:0:(import.c:529:import_select_connection()) MGC192.168.4.128@o2ib: tried all connections, increasing latency to 5s May 17 15:51:46 client-5 kernel: LustreError: 1987:0:(client.c:1046:ptlrpc_import_delay_req()) @@@ send limit expired req@ffff81033a620c00 x1369097007792133/t0(0) o-1->MGS@MGC192.168.4.128@o2ib_0:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:W/ffffffff/ffffffff rc 0/-1 May 17 15:51:52 client-5 kernel: Lustre: 1579:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369097007792134 sent from MGC192.168.4.128@o2ib to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305672702] [real_sent 1305672702] [current 1305672712] [deadline 10s] [delay 0s] req@ffff810315a6bc00 x1369097007792134/t0(0) o-1->MGS@MGC192.168.4.128@o2ib_0:26/25 lens 368/392 e 0 to 1 dl 1305672712 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 15:51:56 client-5 kernel: LustreError: 1987:0:(client.c:1046:ptlrpc_import_delay_req()) @@@ send limit expired req@ffff810318ad4000 x1369097007792135/t0(0) o-1->MGS@MGC192.168.4.128@o2ib_0:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:W/ffffffff/ffffffff rc 0/-1 May 17 15:52:07 client-5 kernel: LustreError: 1987:0:(client.c:1046:ptlrpc_import_delay_req()) @@@ send limit expired req@ffff8102f0a88c00 x1369097007792136/t0(0) o-1->MGS@MGC192.168.4.128@o2ib_0:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:W/ffffffff/ffffffff rc 0/-1 May 17 15:52:14 client-5 kernel: LustreError: 1977:0:(client.c:1046:ptlrpc_import_delay_req()) @@@ send limit expired req@ffff81032a0f8800 x1369097007792131/t0(0) o-1->MGS@MGC192.168.4.128@o2ib_0:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:W/ffffffff/ffffffff rc 0/-1 May 17 15:52:14 client-5 kernel: LustreError: 156-2: The client profile 'lustre-client' could not be read from the MGS. Does that filesystem exist? May 17 15:52:14 client-5 kernel: LustreError: 1987:0:(client.c:1057:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@ffff810326276c00 x1369097007792137/t0(0) o-1->MGS@MGC192.168.4.128@o2ib_0:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:W/ffffffff/ffffffff rc 0/-1 May 17 15:52:14 client-5 kernel: Lustre: client ffff8102d671d000 umount complete May 17 15:52:14 client-5 kernel: LustreError: 1977:0:(obd_mount.c:2149:lustre_fill_super()) Unable to mount (-22) May 17 15:52:37 client-5 kernel: Lustre: Acceptor stopping May 17 15:52:38 client-5 kernel: Lustre: Removed LNI 10.10.4.5@tcp May 17 15:52:40 client-5 kernel: Lustre: Removed LNI 192.168.4.5@o2ib May 17 15:52:41 client-5 xinetd[3391]: START: shell pid=2468 from=192.168.4.5 May 17 15:52:41 client-5 rshd[2469]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 15:52:41 client-5 xinetd[3391]: EXIT: shell status=0 pid=2468 duration=0(sec) May 17 15:52:41 client-5 kernel: Lustre: 2609:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 15:52:41 client-5 kernel: Lustre: 2613:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 15:52:41 client-5 kernel: Lustre: OBD class driver, http://www.lustre.org/ May 17 15:52:41 client-5 kernel: Lustre: Lustre Version: 2.0.60 May 17 15:52:41 client-5 kernel: Lustre: Build Version: jenkins-g985ac5b-PRISTINE-2.6.18-194.17.1.el5 May 17 15:52:41 client-5 kernel: Lustre: Lustre LU module (ffffffff888eb1c0). May 17 15:52:41 client-5 kernel: Lustre: Register global MR array, MR size: 0xffffffffffffffff, array size: 1 May 17 15:52:41 client-5 kernel: Lustre: Added LNI 192.168.4.5@o2ib [8/64/0/180] May 17 15:52:41 client-5 kernel: Lustre: Added LNI 10.10.4.5@tcp [8/256/0/180] May 17 15:52:41 client-5 kernel: Lustre: Accept all, port 7988 May 17 15:52:42 client-5 kernel: Lustre: Lustre OSC module (ffffffff88b3c6c0). May 17 15:52:42 client-5 kernel: Lustre: Lustre LOV module (ffffffff88bab540). May 17 15:52:42 client-5 kernel: Lustre: Lustre client module (ffffffff88c643a0). May 17 15:52:42 client-5 xinetd[3391]: START: shell pid=3009 from=192.168.4.5 May 17 15:52:42 client-5 rshd[3012]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == conf-sanity test 5c: cleanup after failed mount \(bug 2712\) \(should return errs\) =================== 15:52:41 \(1305672761\)");echo XXRETCODE:$?' May 17 15:52:42 client-5 kernel: Lustre: DEBUG MARKER: == conf-sanity test 5c: cleanup after failed mount (bug 2712) (should return errs) =================== 15:52:41 (1305672761) May 17 15:52:42 client-5 xinetd[3391]: EXIT: shell status=0 pid=3009 duration=0(sec) May 17 15:52:43 client-5 kernel: Lustre: 3162:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 15:52:43 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 15:52:43 client-5 kernel: LustreError: 3162:0:(mgc_request.c:287:config_log_add()) logname wrong.lustre-client is too long May 17 15:52:43 client-5 kernel: LustreError: 15b-f: MGC192.168.4.128@o2ib: The configuration from log 'wrong.lustre-client'failed from the MGS (-22). Make sure this client and the MGS are running compatible versions of Lustre. May 17 15:52:43 client-5 kernel: LustreError: 15c-8: MGC192.168.4.128@o2ib: The configuration from log 'wrong.lustre-client' failed (-22). This may be the result of communication errors between this node and the MGS, a bad configuration, or other errors. See the syslog for more information. May 17 15:52:43 client-5 kernel: LustreError: 3162:0:(llite_lib.c:884:ll_fill_super()) Unable to process log: -22 May 17 15:52:43 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 15:52:43 client-5 kernel: Lustre: client ffff810310926000 umount complete May 17 15:52:43 client-5 kernel: LustreError: 3162:0:(obd_mount.c:2149:lustre_fill_super()) Unable to mount (-22) May 17 15:52:59 client-5 kernel: Lustre: Acceptor stopping May 17 15:53:00 client-5 kernel: Lustre: Removed LNI 10.10.4.5@tcp May 17 15:53:02 client-5 kernel: Lustre: Removed LNI 192.168.4.5@o2ib May 17 15:53:03 client-5 xinetd[3391]: START: shell pid=3678 from=192.168.4.5 May 17 15:53:03 client-5 rshd[3679]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 15:53:04 client-5 xinetd[3391]: EXIT: shell status=0 pid=3678 duration=1(sec) May 17 15:53:04 client-5 kernel: Lustre: 3814:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 15:53:04 client-5 kernel: Lustre: 3818:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 15:53:04 client-5 kernel: Lustre: OBD class driver, http://www.lustre.org/ May 17 15:53:04 client-5 kernel: Lustre: Lustre Version: 2.0.60 May 17 15:53:04 client-5 kernel: Lustre: Build Version: jenkins-g985ac5b-PRISTINE-2.6.18-194.17.1.el5 May 17 15:53:04 client-5 kernel: Lustre: Lustre LU module (ffffffff888eb1c0). May 17 15:53:04 client-5 kernel: Lustre: Register global MR array, MR size: 0xffffffffffffffff, array size: 1 May 17 15:53:04 client-5 kernel: Lustre: Added LNI 192.168.4.5@o2ib [8/64/0/180] May 17 15:53:04 client-5 kernel: Lustre: Added LNI 10.10.4.5@tcp [8/256/0/180] May 17 15:53:04 client-5 kernel: Lustre: Accept all, port 7988 May 17 15:53:04 client-5 kernel: Lustre: Lustre OSC module (ffffffff88b3c6c0). May 17 15:53:04 client-5 kernel: Lustre: Lustre LOV module (ffffffff88bab540). May 17 15:53:04 client-5 kernel: Lustre: Lustre client module (ffffffff88c643a0). May 17 15:53:04 client-5 xinetd[3391]: START: shell pid=4215 from=192.168.4.5 May 17 15:53:04 client-5 rshd[4216]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == conf-sanity test 5d: mount with ost down ========================================================== 15:53:04 \(1305672784\)");echo XXRETCODE:$?' May 17 15:53:04 client-5 kernel: Lustre: DEBUG MARKER: == conf-sanity test 5d: mount with ost down ========================================================== 15:53:04 (1305672784) May 17 15:53:04 client-5 xinetd[3391]: EXIT: shell status=0 pid=4215 duration=0(sec) May 17 15:53:36 client-5 kernel: Lustre: 4383:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 15:53:36 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 15:53:36 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 15:53:36 client-5 kernel: Lustre: 4383:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff8102e7caf400->192.168.4.128@o2ib netid 50000: select flavor null May 17 15:53:36 client-5 kernel: Lustre: Client lustre-client has started May 17 15:53:36 client-5 kernel: Lustre: 4400:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 15:53:41 client-5 kernel: Lustre: 3921:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369097145155595 sent from lustre-OST0000-osc-ffff8102e7caf400 to NID 192.168.4.129@o2ib has timed out for slow reply: [sent 1305672816] [real_sent 1305672816] [current 1305672821] [deadline 5s] [delay 0s] req@ffff8102f4582800 x1369097145155595/t0(0) o-1->lustre-OST0000_UUID@192.168.4.129@o2ib:28/4 lens 368/392 e 0 to 1 dl 1305672821 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 15:53:41 client-5 kernel: LustreError: 4425:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 15:53:41 client-5 kernel: LustreError: 4425:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 15:53:41 client-5 kernel: LustreError: 4425:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 15:53:41 client-5 kernel: LustreError: 4425:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 15:53:41 client-5 kernel: Lustre: client ffff8102e7caf400 umount complete May 17 15:53:56 client-5 kernel: Lustre: Acceptor stopping May 17 15:53:57 client-5 kernel: Lustre: Removed LNI 10.10.4.5@tcp May 17 15:53:59 client-5 kernel: Lustre: Removed LNI 192.168.4.5@o2ib May 17 15:54:00 client-5 xinetd[3391]: START: shell pid=4870 from=192.168.4.5 May 17 15:54:00 client-5 rshd[4871]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 15:54:00 client-5 xinetd[3391]: EXIT: shell status=0 pid=4870 duration=0(sec) May 17 15:54:00 client-5 kernel: Lustre: 5006:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 15:54:00 client-5 kernel: Lustre: 5009:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 15:54:01 client-5 kernel: Lustre: OBD class driver, http://www.lustre.org/ May 17 15:54:01 client-5 kernel: Lustre: Lustre Version: 2.0.60 May 17 15:54:01 client-5 kernel: Lustre: Build Version: jenkins-g985ac5b-PRISTINE-2.6.18-194.17.1.el5 May 17 15:54:01 client-5 kernel: Lustre: Lustre LU module (ffffffff888eb1c0). May 17 15:54:01 client-5 kernel: Lustre: Register global MR array, MR size: 0xffffffffffffffff, array size: 1 May 17 15:54:01 client-5 kernel: Lustre: Added LNI 192.168.4.5@o2ib [8/64/0/180] May 17 15:54:01 client-5 kernel: Lustre: Added LNI 10.10.4.5@tcp [8/256/0/180] May 17 15:54:01 client-5 kernel: Lustre: Accept all, port 7988 May 17 15:54:01 client-5 kernel: Lustre: Lustre OSC module (ffffffff88b3c6c0). May 17 15:54:01 client-5 kernel: Lustre: Lustre LOV module (ffffffff88bab540). May 17 15:54:01 client-5 kernel: Lustre: Lustre client module (ffffffff88c643a0). May 17 15:54:01 client-5 xinetd[3391]: START: shell pid=5410 from=192.168.4.5 May 17 15:54:01 client-5 rshd[5411]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == conf-sanity test 5e: delayed connect, don\'t crash \(bug 10268\) ===================================== 15:54:00 \(1305672840\)");echo XXRETCODE:$?' May 17 15:54:01 client-5 kernel: Lustre: DEBUG MARKER: == conf-sanity test 5e: delayed connect, don't crash (bug 10268) ===================================== 15:54:00 (1305672840) May 17 15:54:01 client-5 xinetd[3391]: EXIT: shell status=0 pid=5410 duration=0(sec) May 17 15:54:03 client-5 kernel: Lustre: 5548:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 15:54:03 client-5 kernel: LustreError: 5112:0:(lvfs_lib.c:150:__obd_fail_timeout_set()) obd_fail_timeout id 506 sleeping for 5005ms May 17 15:54:03 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 15:54:08 client-5 kernel: LustreError: 5112:0:(lvfs_lib.c:154:__obd_fail_timeout_set()) obd_fail_timeout id 506 awake May 17 15:54:08 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 15:54:08 client-5 kernel: LustreError: 5548:0:(client.c:1046:ptlrpc_import_delay_req()) @@@ send limit expired req@ffff810321d2ac00 x1369097204924418/t0(0) o-1->MGS@MGC192.168.4.128@o2ib_0:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:W/ffffffff/ffffffff rc 0/-1 May 17 15:54:08 client-5 kernel: Lustre: 5548:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff810327dda000->192.168.4.128@o2ib netid 50000: select flavor null May 17 15:54:08 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 15:54:13 client-5 kernel: Lustre: 5548:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-OST0000-osc-ffff810327dda000->192.168.4.129@o2ib netid 50000: select flavor null May 17 15:54:13 client-5 kernel: Lustre: Client lustre-client has started May 17 15:54:13 client-5 kernel: Lustre: 5566:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 15:54:13 client-5 kernel: LustreError: 5587:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 15:54:13 client-5 kernel: LustreError: 5587:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 15:54:13 client-5 kernel: LustreError: 5587:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 15:54:13 client-5 kernel: LustreError: 5587:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 15:54:13 client-5 kernel: Lustre: client ffff810327dda000 umount complete May 17 15:54:33 client-5 kernel: Lustre: Acceptor stopping May 17 15:54:34 client-5 kernel: Lustre: Removed LNI 10.10.4.5@tcp May 17 15:54:36 client-5 kernel: Lustre: Removed LNI 192.168.4.5@o2ib May 17 15:54:37 client-5 xinetd[3391]: START: shell pid=6039 from=192.168.4.5 May 17 15:54:37 client-5 rshd[6042]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 15:54:37 client-5 xinetd[3391]: EXIT: shell status=0 pid=6039 duration=0(sec) May 17 15:54:37 client-5 kernel: Lustre: 6178:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 15:54:37 client-5 kernel: Lustre: 6179:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 15:54:37 client-5 kernel: Lustre: OBD class driver, http://www.lustre.org/ May 17 15:54:37 client-5 kernel: Lustre: Lustre Version: 2.0.60 May 17 15:54:37 client-5 kernel: Lustre: Build Version: jenkins-g985ac5b-PRISTINE-2.6.18-194.17.1.el5 May 17 15:54:37 client-5 kernel: Lustre: Lustre LU module (ffffffff888eb1c0). May 17 15:54:37 client-5 kernel: Lustre: Register global MR array, MR size: 0xffffffffffffffff, array size: 1 May 17 15:54:37 client-5 kernel: Lustre: Added LNI 192.168.4.5@o2ib [8/64/0/180] May 17 15:54:37 client-5 kernel: Lustre: Added LNI 10.10.4.5@tcp [8/256/0/180] May 17 15:54:37 client-5 kernel: Lustre: Accept all, port 7988 May 17 15:54:38 client-5 kernel: Lustre: Lustre OSC module (ffffffff88b3c6c0). May 17 15:54:38 client-5 kernel: Lustre: Lustre LOV module (ffffffff88bab540). May 17 15:54:38 client-5 kernel: Lustre: Lustre client module (ffffffff88c643a0). May 17 15:54:38 client-5 xinetd[3391]: START: shell pid=6582 from=192.168.4.5 May 17 15:54:38 client-5 rshd[6583]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == conf-sanity test 5f: mds down, cleanup after failed mount \(bug 2712\) ============================== 15:54:37 \(1305672877\)");echo XXRETCODE:$?' May 17 15:54:38 client-5 kernel: Lustre: DEBUG MARKER: == conf-sanity test 5f: mds down, cleanup after failed mount (bug 2712) ============================== 15:54:37 (1305672877) May 17 15:54:38 client-5 xinetd[3391]: EXIT: shell status=0 pid=6582 duration=0(sec) May 17 15:54:38 client-5 xinetd[3391]: START: shell pid=6678 from=192.168.4.5 May 17 15:54:38 client-5 rshd[6679]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark SKIP: conf-sanity test_5f combined mgs and mds");echo XXRETCODE:$?' May 17 15:54:38 client-5 kernel: Lustre: DEBUG MARKER: SKIP: conf-sanity test_5f combined mgs and mds May 17 15:54:38 client-5 xinetd[3391]: EXIT: shell status=0 pid=6678 duration=0(sec) May 17 15:54:38 client-5 xinetd[3391]: START: shell pid=6769 from=192.168.4.5 May 17 15:54:38 client-5 rshd[6770]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 15:54:38 client-5 xinetd[3391]: EXIT: shell status=0 pid=6769 duration=0(sec) May 17 15:54:38 client-5 xinetd[3391]: START: shell pid=6940 from=192.168.4.5 May 17 15:54:38 client-5 rshd[6942]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == conf-sanity test 6: manual umount, then mount again =============================================== 15:54:38 \(1305672878\)");echo XXRETCODE:$?' May 17 15:54:38 client-5 kernel: Lustre: DEBUG MARKER: == conf-sanity test 6: manual umount, then mount again =============================================== 15:54:38 (1305672878) May 17 15:54:38 client-5 xinetd[3391]: EXIT: shell status=0 pid=6940 duration=0(sec) May 17 15:54:40 client-5 kernel: Lustre: 7068:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 15:54:40 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 15:54:40 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 15:54:40 client-5 kernel: Lustre: 7068:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff8102f38e0400->192.168.4.128@o2ib netid 50000: select flavor null May 17 15:54:40 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 15:54:45 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 15:54:50 client-5 kernel: Lustre: 7068:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-OST0000-osc-ffff8102f38e0400->192.168.4.129@o2ib netid 50000: select flavor null May 17 15:54:50 client-5 kernel: Lustre: Client lustre-client has started May 17 15:54:50 client-5 kernel: Lustre: 7085:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 15:54:50 client-5 kernel: LustreError: 7093:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 15:54:50 client-5 kernel: LustreError: 7093:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 15:54:50 client-5 kernel: Lustre: client ffff8102f38e0400 umount complete May 17 15:54:50 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 15:54:50 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 15:54:50 client-5 kernel: Lustre: Client lustre-client has started May 17 15:54:50 client-5 kernel: LustreError: 7142:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 15:54:50 client-5 kernel: LustreError: 7142:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 15:54:50 client-5 kernel: Lustre: client ffff8102cfb8ec00 umount complete May 17 15:55:12 client-5 kernel: Lustre: Acceptor stopping May 17 15:55:13 client-5 kernel: Lustre: Removed LNI 10.10.4.5@tcp May 17 15:55:15 client-5 kernel: Lustre: Removed LNI 192.168.4.5@o2ib May 17 15:55:16 client-5 xinetd[3391]: START: shell pid=7598 from=192.168.4.5 May 17 15:55:16 client-5 rshd[7599]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 15:55:16 client-5 xinetd[3391]: EXIT: shell status=0 pid=7598 duration=0(sec) May 17 15:55:17 client-5 kernel: Lustre: 7733:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 15:55:17 client-5 kernel: Lustre: 7735:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 15:55:17 client-5 kernel: Lustre: OBD class driver, http://www.lustre.org/ May 17 15:55:17 client-5 kernel: Lustre: Lustre Version: 2.0.60 May 17 15:55:17 client-5 kernel: Lustre: Build Version: jenkins-g985ac5b-PRISTINE-2.6.18-194.17.1.el5 May 17 15:55:17 client-5 kernel: Lustre: Lustre LU module (ffffffff888eb1c0). May 17 15:55:17 client-5 kernel: Lustre: Register global MR array, MR size: 0xffffffffffffffff, array size: 1 May 17 15:55:17 client-5 kernel: Lustre: Added LNI 192.168.4.5@o2ib [8/64/0/180] May 17 15:55:17 client-5 kernel: Lustre: Added LNI 10.10.4.5@tcp [8/256/0/180] May 17 15:55:17 client-5 kernel: Lustre: Accept all, port 7988 May 17 15:55:17 client-5 kernel: Lustre: Lustre OSC module (ffffffff88b3c6c0). May 17 15:55:17 client-5 kernel: Lustre: Lustre LOV module (ffffffff88bab540). May 17 15:55:17 client-5 kernel: Lustre: Lustre client module (ffffffff88c643a0). May 17 15:55:17 client-5 xinetd[3391]: START: shell pid=8135 from=192.168.4.5 May 17 15:55:17 client-5 rshd[8136]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == conf-sanity test 7: manual umount, then cleanup =================================================== 15:55:17 \(1305672917\)");echo XXRETCODE:$?' May 17 15:55:17 client-5 kernel: Lustre: DEBUG MARKER: == conf-sanity test 7: manual umount, then cleanup =================================================== 15:55:17 (1305672917) May 17 15:55:17 client-5 xinetd[3391]: EXIT: shell status=0 pid=8135 duration=0(sec) May 17 15:55:19 client-5 kernel: Lustre: 8268:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 15:55:19 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 15:55:19 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 15:55:19 client-5 kernel: Lustre: 8268:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff8102f640d000->192.168.4.128@o2ib netid 50000: select flavor null May 17 15:55:19 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 15:55:24 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 15:55:29 client-5 kernel: Lustre: 8268:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-OST0000-osc-ffff8102f640d000->192.168.4.129@o2ib netid 50000: select flavor null May 17 15:55:29 client-5 kernel: Lustre: Client lustre-client has started May 17 15:55:29 client-5 kernel: Lustre: 8285:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 15:55:29 client-5 kernel: LustreError: 8293:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 15:55:29 client-5 kernel: LustreError: 8293:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 15:55:29 client-5 kernel: Lustre: client ffff8102f640d000 umount complete May 17 15:55:48 client-5 kernel: Lustre: Acceptor stopping May 17 15:55:49 client-5 kernel: Lustre: Removed LNI 10.10.4.5@tcp May 17 15:55:51 client-5 kernel: Lustre: Removed LNI 192.168.4.5@o2ib May 17 15:55:52 client-5 xinetd[3391]: START: shell pid=8745 from=192.168.4.5 May 17 15:55:52 client-5 rshd[8746]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 15:55:52 client-5 xinetd[3391]: EXIT: shell status=0 pid=8745 duration=0(sec) May 17 15:55:52 client-5 kernel: Lustre: 8880:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 15:55:52 client-5 kernel: Lustre: 8882:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 15:55:52 client-5 kernel: Lustre: OBD class driver, http://www.lustre.org/ May 17 15:55:52 client-5 kernel: Lustre: Lustre Version: 2.0.60 May 17 15:55:52 client-5 kernel: Lustre: Build Version: jenkins-g985ac5b-PRISTINE-2.6.18-194.17.1.el5 May 17 15:55:52 client-5 kernel: Lustre: Lustre LU module (ffffffff888eb1c0). May 17 15:55:53 client-5 kernel: Lustre: Register global MR array, MR size: 0xffffffffffffffff, array size: 1 May 17 15:55:53 client-5 kernel: Lustre: Added LNI 192.168.4.5@o2ib [8/64/0/180] May 17 15:55:53 client-5 kernel: Lustre: Added LNI 10.10.4.5@tcp [8/256/0/180] May 17 15:55:53 client-5 kernel: Lustre: Accept all, port 7988 May 17 15:55:53 client-5 kernel: Lustre: Lustre OSC module (ffffffff88b3c6c0). May 17 15:55:53 client-5 kernel: Lustre: Lustre LOV module (ffffffff88bab540). May 17 15:55:53 client-5 kernel: Lustre: Lustre client module (ffffffff88c643a0). May 17 15:55:53 client-5 xinetd[3391]: START: shell pid=9283 from=192.168.4.5 May 17 15:55:53 client-5 rshd[9284]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == conf-sanity test 8: double mount setup ============================================================ 15:55:52 \(1305672952\)");echo XXRETCODE:$?' May 17 15:55:53 client-5 kernel: Lustre: DEBUG MARKER: == conf-sanity test 8: double mount setup ============================================================ 15:55:52 (1305672952) May 17 15:55:53 client-5 xinetd[3391]: EXIT: shell status=0 pid=9283 duration=0(sec) May 17 15:55:54 client-5 kernel: Lustre: 9415:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 15:55:55 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 15:55:55 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 15:55:55 client-5 kernel: Lustre: 9415:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff8103188d7400->192.168.4.128@o2ib netid 50000: select flavor null May 17 15:55:55 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 15:56:00 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 15:56:05 client-5 kernel: Lustre: 9415:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-OST0000-osc-ffff8103188d7400->192.168.4.129@o2ib netid 50000: select flavor null May 17 15:56:05 client-5 kernel: Lustre: Client lustre-client has started May 17 15:56:05 client-5 kernel: Lustre: 9432:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 15:56:05 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 15:56:05 client-5 kernel: Lustre: Client lustre-client has started May 17 15:56:05 client-5 kernel: LustreError: 9496:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 15:56:05 client-5 kernel: LustreError: 9496:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 15:56:05 client-5 kernel: Lustre: client ffff8102f9898c00 umount complete May 17 15:56:05 client-5 kernel: LustreError: 9515:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 15:56:05 client-5 kernel: LustreError: 9515:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 15:56:05 client-5 kernel: Lustre: client ffff8103188d7400 umount complete May 17 15:56:27 client-5 kernel: Lustre: Acceptor stopping May 17 15:56:28 client-5 kernel: Lustre: Removed LNI 10.10.4.5@tcp May 17 15:56:30 client-5 kernel: Lustre: Removed LNI 192.168.4.5@o2ib May 17 15:56:31 client-5 xinetd[3391]: START: shell pid=9972 from=192.168.4.5 May 17 15:56:31 client-5 rshd[9973]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 15:56:31 client-5 xinetd[3391]: EXIT: shell status=0 pid=9972 duration=0(sec) May 17 15:56:31 client-5 kernel: Lustre: 10108:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 15:56:31 client-5 kernel: Lustre: 10109:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 15:56:31 client-5 kernel: Lustre: OBD class driver, http://www.lustre.org/ May 17 15:56:31 client-5 kernel: Lustre: Lustre Version: 2.0.60 May 17 15:56:31 client-5 kernel: Lustre: Build Version: jenkins-g985ac5b-PRISTINE-2.6.18-194.17.1.el5 May 17 15:56:31 client-5 kernel: Lustre: Lustre LU module (ffffffff888eb1c0). May 17 15:56:31 client-5 kernel: Lustre: Register global MR array, MR size: 0xffffffffffffffff, array size: 1 May 17 15:56:31 client-5 kernel: Lustre: Added LNI 192.168.4.5@o2ib [8/64/0/180] May 17 15:56:31 client-5 kernel: Lustre: Added LNI 10.10.4.5@tcp [8/256/0/180] May 17 15:56:31 client-5 kernel: Lustre: Accept all, port 7988 May 17 15:56:32 client-5 kernel: Lustre: Lustre OSC module (ffffffff88b3c6c0). May 17 15:56:32 client-5 kernel: Lustre: Lustre LOV module (ffffffff88bab540). May 17 15:56:32 client-5 kernel: Lustre: Lustre client module (ffffffff88c643a0). May 17 15:56:32 client-5 xinetd[3391]: START: shell pid=10510 from=192.168.4.5 May 17 15:56:32 client-5 rshd[10511]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == conf-sanity test 9: test ptldebug and subsystem for mkfs ========================================== 15:56:31 \(1305672991\)");echo XXRETCODE:$?' May 17 15:56:32 client-5 kernel: Lustre: DEBUG MARKER: == conf-sanity test 9: test ptldebug and subsystem for mkfs ========================================== 15:56:31 (1305672991) May 17 15:56:32 client-5 xinetd[3391]: EXIT: shell status=0 pid=10510 duration=0(sec) May 17 15:56:57 client-5 xinetd[3391]: START: shell pid=10713 from=192.168.4.5 May 17 15:56:57 client-5 rshd[10714]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 15:56:57 client-5 xinetd[3391]: EXIT: shell status=0 pid=10713 duration=0(sec) May 17 15:56:57 client-5 xinetd[3391]: START: shell pid=10885 from=192.168.4.5 May 17 15:56:57 client-5 rshd[10886]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == conf-sanity test 17: Verify failed mds_postsetup won\'t fail assertion \(2936\) \(should return errs\) ====================================================================================================== 15:56:57 \(1305673017\)");echo XXRETCODE:$?' May 17 15:56:58 client-5 kernel: Lustre: DEBUG MARKER: == conf-sanity test 17: Verify failed mds_postsetup won't fail assertion (2936) (should return errs) ====================================================================================================== 15:56:57 (1305673017) May 17 15:56:58 client-5 xinetd[3391]: EXIT: shell status=0 pid=10885 duration=1(sec) May 17 15:56:59 client-5 kernel: Lustre: 11012:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 15:56:59 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 15:56:59 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 15:56:59 client-5 kernel: Lustre: 11012:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff810310104000->192.168.4.128@o2ib netid 50000: select flavor null May 17 15:56:59 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 15:57:04 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 15:57:09 client-5 kernel: Lustre: 11012:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-OST0000-osc-ffff810310104000->192.168.4.129@o2ib netid 50000: select flavor null May 17 15:57:09 client-5 kernel: Lustre: Client lustre-client has started May 17 15:57:09 client-5 kernel: Lustre: 11029:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 15:57:09 client-5 kernel: LustreError: 11069:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 15:57:09 client-5 kernel: LustreError: 11069:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 15:57:09 client-5 kernel: LustreError: 11069:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 15:57:09 client-5 kernel: LustreError: 11069:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 15:57:09 client-5 kernel: Lustre: client ffff810310104000 umount complete May 17 15:57:31 client-5 kernel: Lustre: Acceptor stopping May 17 15:57:32 client-5 kernel: Lustre: Removed LNI 10.10.4.5@tcp May 17 15:57:34 client-5 kernel: Lustre: Removed LNI 192.168.4.5@o2ib May 17 15:58:03 client-5 xinetd[3391]: START: shell pid=11551 from=192.168.4.5 May 17 15:58:03 client-5 rshd[11552]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:; lsof -t /mnt/lustre || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre is still busy, wait one second && sleep 1; umount /mnt/lustre; fi fi");echo XXRETCODE:$?' May 17 15:58:04 client-5 xinetd[3391]: EXIT: shell status=0 pid=11551 duration=1(sec) May 17 15:58:04 client-5 xinetd[3391]: START: shell pid=11595 from=192.168.4.5 May 17 15:58:04 client-5 rshd[11596]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:; lsof -t /mnt/lustre2 || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre2 | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre2 is still busy, wait one second && sleep 1; umount /mnt/lustre2; fi fi");echo XXRETCODE:$?' May 17 15:58:04 client-5 xinetd[3391]: EXIT: shell status=0 pid=11595 duration=0(sec) May 17 15:58:12 client-5 kernel: Lustre: 11824:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 15:58:12 client-5 kernel: Lustre: 11825:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 15:58:12 client-5 kernel: Lustre: OBD class driver, http://www.lustre.org/ May 17 15:58:12 client-5 kernel: Lustre: Lustre Version: 2.0.60 May 17 15:58:12 client-5 kernel: Lustre: Build Version: jenkins-g985ac5b-PRISTINE-2.6.18-194.17.1.el5 May 17 15:58:12 client-5 kernel: Lustre: Lustre LU module (ffffffff888eb1c0). May 17 15:58:12 client-5 kernel: Lustre: Register global MR array, MR size: 0xffffffffffffffff, array size: 1 May 17 15:58:12 client-5 kernel: Lustre: Added LNI 192.168.4.5@o2ib [8/64/0/180] May 17 15:58:12 client-5 kernel: Lustre: Added LNI 10.10.4.5@tcp [8/256/0/180] May 17 15:58:12 client-5 kernel: Lustre: Accept all, port 7988 May 17 15:58:12 client-5 kernel: Lustre: Lustre OSC module (ffffffff88b3c6c0). May 17 15:58:12 client-5 kernel: Lustre: Lustre LOV module (ffffffff88bab540). May 17 15:58:12 client-5 kernel: Lustre: Lustre client module (ffffffff88c643a0). May 17 15:58:23 client-5 xinetd[3391]: START: shell pid=12701 from=192.168.4.5 May 17 15:58:23 client-5 rshd[12702]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark ost now in FULL state");echo XXRETCODE:$?' May 17 15:58:23 client-5 kernel: Lustre: DEBUG MARKER: ost now in FULL state May 17 15:58:23 client-5 xinetd[3391]: EXIT: shell status=0 pid=12701 duration=0(sec) May 17 15:58:34 client-5 xinetd[3391]: START: shell pid=12858 from=192.168.4.5 May 17 15:58:34 client-5 rshd[12859]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 15:58:35 client-5 xinetd[3391]: EXIT: shell status=0 pid=12858 duration=1(sec) May 17 15:58:35 client-5 xinetd[3391]: START: shell pid=13030 from=192.168.4.5 May 17 15:58:35 client-5 rshd[13031]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == conf-sanity test 18: check mkfs creates large journals ============================================ 15:58:35 \(1305673115\)");echo XXRETCODE:$?' May 17 15:58:35 client-5 kernel: Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ============================================ 15:58:35 (1305673115) May 17 15:58:35 client-5 xinetd[3391]: EXIT: shell status=0 pid=13030 duration=0(sec) May 17 15:58:35 client-5 xinetd[3391]: START: shell pid=13144 from=192.168.4.5 May 17 15:58:35 client-5 rshd[13145]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark use device \/dev\/sdb with MIN=2000000");echo XXRETCODE:$?' May 17 15:58:35 client-5 kernel: Lustre: DEBUG MARKER: use device /dev/sdb with MIN=2000000 May 17 15:58:35 client-5 xinetd[3391]: EXIT: shell status=0 pid=13144 duration=0(sec) May 17 15:58:35 client-5 xinetd[3391]: START: shell pid=13187 from=192.168.4.5 May 17 15:58:35 client-5 rshd[13188]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:; lsof -t /mnt/lustre || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre is still busy, wait one second && sleep 1; umount /mnt/lustre; fi fi");echo XXRETCODE:$?' May 17 15:58:35 client-5 xinetd[3391]: EXIT: shell status=0 pid=13187 duration=0(sec) May 17 15:58:35 client-5 xinetd[3391]: START: shell pid=13231 from=192.168.4.5 May 17 15:58:35 client-5 rshd[13232]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:; lsof -t /mnt/lustre2 || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre2 | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre2 is still busy, wait one second && sleep 1; umount /mnt/lustre2; fi fi");echo XXRETCODE:$?' May 17 15:58:35 client-5 xinetd[3391]: EXIT: shell status=0 pid=13231 duration=0(sec) May 17 15:58:37 client-5 kernel: Lustre: 13438:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 15:58:50 client-5 xinetd[3391]: START: shell pid=14022 from=192.168.4.5 May 17 15:58:50 client-5 rshd[14023]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark ost now in FULL state");echo XXRETCODE:$?' May 17 15:58:50 client-5 kernel: Lustre: DEBUG MARKER: ost now in FULL state May 17 15:58:50 client-5 xinetd[3391]: EXIT: shell status=0 pid=14022 duration=0(sec) May 17 15:59:03 client-5 kernel: Lustre: 14215:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 15:59:03 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 15:59:03 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 15:59:03 client-5 kernel: Lustre: 14215:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff810312c5ac00->192.168.4.128@o2ib netid 50000: select flavor null May 17 15:59:03 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 15:59:08 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 15:59:13 client-5 kernel: Lustre: 14215:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-OST0000-osc-ffff810312c5ac00->192.168.4.129@o2ib netid 50000: select flavor null May 17 15:59:13 client-5 kernel: Lustre: Client lustre-client has started May 17 15:59:13 client-5 kernel: Lustre: 14232:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 15:59:13 client-5 kernel: Lustre: 14232:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message May 17 15:59:14 client-5 xinetd[3391]: START: shell pid=14323 from=192.168.4.5 May 17 15:59:14 client-5 rshd[14324]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark Success: mkfs creates large journals. Size: 76M");echo XXRETCODE:$?' May 17 15:59:14 client-5 kernel: Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 76M May 17 15:59:14 client-5 xinetd[3391]: EXIT: shell status=0 pid=14323 duration=0(sec) May 17 15:59:14 client-5 kernel: LustreError: 14376:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 15:59:14 client-5 kernel: LustreError: 14376:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 15:59:14 client-5 kernel: LustreError: 14376:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 15:59:14 client-5 kernel: LustreError: 14376:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 15:59:14 client-5 kernel: Lustre: client ffff810312c5ac00 umount complete May 17 15:59:31 client-5 kernel: Lustre: Acceptor stopping May 17 15:59:32 client-5 kernel: Lustre: Removed LNI 10.10.4.5@tcp May 17 15:59:34 client-5 kernel: Lustre: Removed LNI 192.168.4.5@o2ib May 17 15:59:35 client-5 xinetd[3391]: START: shell pid=14779 from=192.168.4.5 May 17 15:59:36 client-5 rshd[14780]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:; lsof -t /mnt/lustre || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre is still busy, wait one second && sleep 1; umount /mnt/lustre; fi fi");echo XXRETCODE:$?' May 17 15:59:36 client-5 xinetd[3391]: EXIT: shell status=0 pid=14779 duration=1(sec) May 17 15:59:36 client-5 xinetd[3391]: START: shell pid=14824 from=192.168.4.5 May 17 15:59:36 client-5 rshd[14825]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:; lsof -t /mnt/lustre2 || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre2 | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre2 is still busy, wait one second && sleep 1; umount /mnt/lustre2; fi fi");echo XXRETCODE:$?' May 17 15:59:36 client-5 xinetd[3391]: EXIT: shell status=0 pid=14824 duration=0(sec) May 17 15:59:37 client-5 kernel: Lustre: 15044:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 15:59:37 client-5 kernel: Lustre: 15046:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 15:59:37 client-5 kernel: Lustre: OBD class driver, http://www.lustre.org/ May 17 15:59:37 client-5 kernel: Lustre: Lustre Version: 2.0.60 May 17 15:59:37 client-5 kernel: Lustre: Build Version: jenkins-g985ac5b-PRISTINE-2.6.18-194.17.1.el5 May 17 15:59:37 client-5 kernel: Lustre: Lustre LU module (ffffffff888eb1c0). May 17 15:59:37 client-5 kernel: Lustre: Register global MR array, MR size: 0xffffffffffffffff, array size: 1 May 17 15:59:37 client-5 kernel: Lustre: Added LNI 192.168.4.5@o2ib [8/64/0/180] May 17 15:59:37 client-5 kernel: Lustre: Added LNI 10.10.4.5@tcp [8/256/0/180] May 17 15:59:37 client-5 kernel: Lustre: Accept all, port 7988 May 17 15:59:38 client-5 kernel: Lustre: Lustre OSC module (ffffffff88b3c6c0). May 17 15:59:38 client-5 kernel: Lustre: Lustre LOV module (ffffffff88bab540). May 17 15:59:38 client-5 kernel: Lustre: Lustre client module (ffffffff88c643a0). May 17 15:59:49 client-5 xinetd[3391]: START: shell pid=15932 from=192.168.4.5 May 17 15:59:49 client-5 rshd[15935]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark ost now in FULL state");echo XXRETCODE:$?' May 17 15:59:49 client-5 kernel: Lustre: DEBUG MARKER: ost now in FULL state May 17 15:59:49 client-5 xinetd[3391]: EXIT: shell status=0 pid=15932 duration=0(sec) May 17 16:00:01 client-5 xinetd[3391]: START: shell pid=16095 from=192.168.4.5 May 17 16:00:01 client-5 rshd[16096]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 16:00:01 client-5 xinetd[3391]: EXIT: shell status=0 pid=16095 duration=0(sec) May 17 16:00:01 client-5 xinetd[3391]: START: shell pid=16266 from=192.168.4.5 May 17 16:00:01 client-5 rshd[16268]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == conf-sanity test 19a: start\/stop MDS without OSTs ================================================= 16:00:01 \(1305673201\)");echo XXRETCODE:$?' May 17 16:00:01 client-5 kernel: Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ================================================= 16:00:01 (1305673201) May 17 16:00:01 client-5 xinetd[3391]: EXIT: shell status=0 pid=16266 duration=0(sec) May 17 16:00:09 client-5 xinetd[3391]: START: shell pid=16432 from=192.168.4.5 May 17 16:00:09 client-5 rshd[16435]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 16:00:09 client-5 xinetd[3391]: EXIT: shell status=0 pid=16432 duration=0(sec) May 17 16:00:09 client-5 xinetd[3391]: START: shell pid=16606 from=192.168.4.5 May 17 16:00:09 client-5 rshd[16607]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == conf-sanity test 19b: start\/stop OSTs without MDS ================================================= 16:00:09 \(1305673209\)");echo XXRETCODE:$?' May 17 16:00:09 client-5 kernel: Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ================================================= 16:00:09 (1305673209) May 17 16:00:09 client-5 xinetd[3391]: EXIT: shell status=0 pid=16606 duration=0(sec) May 17 16:00:34 client-5 xinetd[3391]: START: shell pid=16770 from=192.168.4.5 May 17 16:00:34 client-5 rshd[16771]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 16:00:34 client-5 xinetd[3391]: EXIT: shell status=0 pid=16770 duration=0(sec) May 17 16:00:34 client-5 xinetd[3391]: START: shell pid=16940 from=192.168.4.5 May 17 16:00:34 client-5 rshd[16943]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == conf-sanity test 20: remount ro,rw mounts work and doesn\'t break \/etc\/mtab ======================== 16:00:34 \(1305673234\)");echo XXRETCODE:$?' May 17 16:00:34 client-5 kernel: Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ======================== 16:00:34 (1305673234) May 17 16:00:34 client-5 xinetd[3391]: EXIT: shell status=0 pid=16940 duration=0(sec) May 17 16:00:36 client-5 kernel: Lustre: 17069:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 16:00:36 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 16:00:36 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 16:00:36 client-5 kernel: Lustre: 17069:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff81031486b800->192.168.4.128@o2ib netid 50000: select flavor null May 17 16:00:36 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 16:00:41 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 16:00:46 client-5 kernel: Lustre: 17069:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-OST0000-osc-ffff81031486b800->192.168.4.129@o2ib netid 50000: select flavor null May 17 16:00:46 client-5 kernel: Lustre: Client lustre-client has started May 17 16:00:46 client-5 kernel: Lustre: 17086:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:00:46 client-5 kernel: LustreError: 17126:0:(mdc_locks.c:722:mdc_enqueue()) ldlm_cli_enqueue: -30 May 17 16:00:46 client-5 kernel: LustreError: 17164:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:00:46 client-5 kernel: LustreError: 17164:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:00:46 client-5 kernel: LustreError: 17164:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:00:46 client-5 kernel: LustreError: 17164:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:00:46 client-5 kernel: Lustre: client ffff81031486b800 umount complete May 17 16:01:07 client-5 xinetd[3391]: START: shell pid=17295 from=192.168.4.5 May 17 16:01:07 client-5 rshd[17298]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 16:01:07 client-5 xinetd[3391]: EXIT: shell status=0 pid=17295 duration=0(sec) May 17 16:01:07 client-5 xinetd[3391]: START: shell pid=17469 from=192.168.4.5 May 17 16:01:07 client-5 rshd[17470]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == conf-sanity test 21a: start mds before ost, stop ost first ======================================== 16:01:07 \(1305673267\)");echo XXRETCODE:$?' May 17 16:01:07 client-5 kernel: Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ======================================== 16:01:07 (1305673267) May 17 16:01:07 client-5 xinetd[3391]: EXIT: shell status=0 pid=17469 duration=0(sec) May 17 16:01:18 client-5 xinetd[3391]: START: shell pid=17822 from=192.168.4.5 May 17 16:01:18 client-5 rshd[17823]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark ost now in FULL state");echo XXRETCODE:$?' May 17 16:01:18 client-5 kernel: Lustre: DEBUG MARKER: ost now in FULL state May 17 16:01:18 client-5 xinetd[3391]: EXIT: shell status=0 pid=17822 duration=0(sec) May 17 16:01:33 client-5 xinetd[3391]: START: shell pid=17979 from=192.168.4.5 May 17 16:01:33 client-5 rshd[17980]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 16:01:33 client-5 xinetd[3391]: EXIT: shell status=0 pid=17979 duration=0(sec) May 17 16:01:33 client-5 xinetd[3391]: START: shell pid=18151 from=192.168.4.5 May 17 16:01:33 client-5 rshd[18152]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == conf-sanity test 21b: start ost before mds, stop mds first ======================================== 16:01:33 \(1305673293\)");echo XXRETCODE:$?' May 17 16:01:33 client-5 kernel: Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ======================================== 16:01:33 (1305673293) May 17 16:01:33 client-5 xinetd[3391]: EXIT: shell status=0 pid=18151 duration=0(sec) May 17 16:01:57 client-5 xinetd[3391]: START: shell pid=18408 from=192.168.4.5 May 17 16:01:57 client-5 rshd[18409]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark ost now in FULL state");echo XXRETCODE:$?' May 17 16:01:57 client-5 kernel: Lustre: DEBUG MARKER: ost now in FULL state May 17 16:01:57 client-5 xinetd[3391]: EXIT: shell status=0 pid=18408 duration=0(sec) May 17 16:02:14 client-5 xinetd[3391]: START: shell pid=18563 from=192.168.4.5 May 17 16:02:14 client-5 rshd[18566]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 16:02:14 client-5 xinetd[3391]: EXIT: shell status=0 pid=18563 duration=0(sec) May 17 16:02:14 client-5 xinetd[3391]: START: shell pid=18735 from=192.168.4.5 May 17 16:02:14 client-5 rshd[18738]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == conf-sanity test 21c: start mds between two osts, stop mds last =================================== 16:02:14 \(1305673334\)");echo XXRETCODE:$?' May 17 16:02:14 client-5 kernel: Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last =================================== 16:02:14 (1305673334) May 17 16:02:14 client-5 xinetd[3391]: EXIT: shell status=0 pid=18735 duration=0(sec) May 17 16:02:42 client-5 xinetd[3391]: START: shell pid=19074 from=192.168.4.5 May 17 16:02:42 client-5 rshd[19075]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark ost2 now in FULL state");echo XXRETCODE:$?' May 17 16:02:42 client-5 kernel: Lustre: DEBUG MARKER: ost2 now in FULL state May 17 16:02:42 client-5 xinetd[3391]: EXIT: shell status=0 pid=19074 duration=0(sec) May 17 16:02:56 client-5 kernel: Lustre: 19211:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:02:56 client-5 kernel: Lustre: 19211:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 5 previous similar messages May 17 16:02:57 client-5 xinetd[3391]: START: shell pid=19421 from=192.168.4.5 May 17 16:02:57 client-5 rshd[19422]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 16:02:57 client-5 xinetd[3391]: EXIT: shell status=0 pid=19421 duration=0(sec) May 17 16:02:57 client-5 xinetd[3391]: START: shell pid=19593 from=192.168.4.5 May 17 16:02:57 client-5 rshd[19594]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == conf-sanity test 21d: start mgs then ost and then mds ============================================= 16:02:57 \(1305673377\)");echo XXRETCODE:$?' May 17 16:02:57 client-5 kernel: Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ============================================= 16:02:57 (1305673377) May 17 16:02:57 client-5 xinetd[3391]: EXIT: shell status=0 pid=19593 duration=0(sec) May 17 16:02:57 client-5 xinetd[3391]: START: shell pid=19686 from=192.168.4.5 May 17 16:02:57 client-5 rshd[19687]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark SKIP: conf-sanity test_21d need separate mgs device");echo XXRETCODE:$?' May 17 16:02:58 client-5 kernel: Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device May 17 16:02:58 client-5 xinetd[3391]: EXIT: shell status=0 pid=19686 duration=1(sec) May 17 16:02:58 client-5 xinetd[3391]: START: shell pid=19777 from=192.168.4.5 May 17 16:02:58 client-5 rshd[19778]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 16:02:58 client-5 xinetd[3391]: EXIT: shell status=0 pid=19777 duration=0(sec) May 17 16:02:58 client-5 xinetd[3391]: START: shell pid=19949 from=192.168.4.5 May 17 16:02:58 client-5 rshd[19950]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == conf-sanity test 22: start a client before osts \(should return errs\) ============================== 16:02:58 \(1305673378\)");echo XXRETCODE:$?' May 17 16:02:58 client-5 kernel: Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ============================== 16:02:58 (1305673378) May 17 16:02:58 client-5 xinetd[3391]: EXIT: shell status=0 pid=19949 duration=0(sec) May 17 16:03:05 client-5 xinetd[3391]: START: shell pid=20254 from=192.168.4.5 May 17 16:03:05 client-5 rshd[20255]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark ost now in FULL state");echo XXRETCODE:$?' May 17 16:03:05 client-5 kernel: Lustre: DEBUG MARKER: ost now in FULL state May 17 16:03:05 client-5 xinetd[3391]: EXIT: shell status=0 pid=20254 duration=0(sec) May 17 16:03:12 client-5 kernel: Lustre: 20331:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 16:03:12 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 16:03:12 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 16:03:12 client-5 kernel: Lustre: Client lustre-client has started May 17 16:03:12 client-5 kernel: Lustre: 20348:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:03:12 client-5 kernel: Lustre: 20348:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message May 17 16:03:17 client-5 kernel: Lustre: 15153:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369097557246018 sent from lustre-OST0000-osc-ffff8103172ee800 to NID 192.168.4.129@o2ib has timed out for slow reply: [sent 1305673392] [real_sent 1305673392] [current 1305673397] [deadline 5s] [delay 0s] req@ffff8102ff696800 x1369097557246018/t0(0) o-1->lustre-OST0000_UUID@192.168.4.129@o2ib:28/4 lens 368/392 e 0 to 1 dl 1305673397 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 16:03:17 client-5 kernel: LustreError: 20371:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:03:17 client-5 kernel: LustreError: 20371:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Skipped 1 previous similar message May 17 16:03:17 client-5 kernel: LustreError: 20371:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:03:17 client-5 kernel: LustreError: 20371:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) Skipped 1 previous similar message May 17 16:03:17 client-5 kernel: Lustre: client ffff8103172ee800 umount complete May 17 16:03:18 client-5 kernel: Lustre: 20423:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 16:03:18 client-5 kernel: Lustre: 20423:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 2 previous similar messages May 17 16:03:18 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 16:03:18 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 16:03:18 client-5 kernel: Lustre: Client lustre-client has started May 17 16:03:18 client-5 kernel: Lustre: 20440:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:03:18 client-5 kernel: Lustre: 20440:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message May 17 16:03:18 client-5 kernel: LustreError: 20482:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:03:18 client-5 kernel: LustreError: 20482:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Skipped 1 previous similar message May 17 16:03:18 client-5 kernel: LustreError: 20482:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:03:18 client-5 kernel: LustreError: 20482:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) Skipped 1 previous similar message May 17 16:03:18 client-5 kernel: Lustre: client ffff81030f77a400 umount complete May 17 16:03:33 client-5 kernel: Lustre: Acceptor stopping May 17 16:03:34 client-5 kernel: Lustre: Removed LNI 10.10.4.5@tcp May 17 16:03:36 client-5 kernel: Lustre: Removed LNI 192.168.4.5@o2ib May 17 16:03:37 client-5 xinetd[3391]: START: shell pid=20937 from=192.168.4.5 May 17 16:03:37 client-5 rshd[20939]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 16:03:37 client-5 xinetd[3391]: EXIT: shell status=0 pid=20937 duration=0(sec) May 17 16:03:38 client-5 kernel: Lustre: 21074:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:03:38 client-5 kernel: Lustre: 21075:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:03:38 client-5 kernel: Lustre: OBD class driver, http://www.lustre.org/ May 17 16:03:38 client-5 kernel: Lustre: Lustre Version: 2.0.60 May 17 16:03:38 client-5 kernel: Lustre: Build Version: jenkins-g985ac5b-PRISTINE-2.6.18-194.17.1.el5 May 17 16:03:38 client-5 kernel: Lustre: Lustre LU module (ffffffff888eb1c0). May 17 16:03:38 client-5 kernel: Lustre: Register global MR array, MR size: 0xffffffffffffffff, array size: 1 May 17 16:03:38 client-5 kernel: Lustre: Added LNI 192.168.4.5@o2ib [8/64/0/180] May 17 16:03:38 client-5 kernel: Lustre: Added LNI 10.10.4.5@tcp [8/256/0/180] May 17 16:03:38 client-5 kernel: Lustre: Accept all, port 7988 May 17 16:03:38 client-5 kernel: Lustre: Lustre OSC module (ffffffff88b3c6c0). May 17 16:03:38 client-5 kernel: Lustre: Lustre LOV module (ffffffff88bab540). May 17 16:03:38 client-5 kernel: Lustre: Lustre client module (ffffffff88c643a0). May 17 16:03:38 client-5 xinetd[3391]: START: shell pid=21471 from=192.168.4.5 May 17 16:03:38 client-5 rshd[21474]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == conf-sanity test 23a: interrupt client during recovery mount delay ================================ 16:03:38 \(1305673418\)");echo XXRETCODE:$?' May 17 16:03:38 client-5 kernel: Lustre: DEBUG MARKER: == conf-sanity test 23a: interrupt client during recovery mount delay ================================ 16:03:38 (1305673418) May 17 16:03:38 client-5 xinetd[3391]: EXIT: shell status=0 pid=21471 duration=0(sec) May 17 16:03:40 client-5 kernel: Lustre: 21608:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 16:03:40 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 16:03:40 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 16:03:40 client-5 kernel: Lustre: 21608:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff8102f640dc00->192.168.4.128@o2ib netid 50000: select flavor null May 17 16:03:40 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 16:03:45 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 16:03:50 client-5 kernel: Lustre: 21608:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-OST0000-osc-ffff8102f640dc00->192.168.4.129@o2ib netid 50000: select flavor null May 17 16:03:50 client-5 kernel: Lustre: Client lustre-client has started May 17 16:03:50 client-5 kernel: Lustre: 21625:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:03:56 client-5 kernel: Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request May 17 16:03:56 client-5 kernel: Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request May 17 16:03:57 client-5 kernel: LustreError: 21659:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:03:57 client-5 kernel: LustreError: 21659:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:04:02 client-5 kernel: Lustre: 21179:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369097809952787 sent from MGC192.168.4.128@o2ib to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305673435] [real_sent 1305673435] [current 1305673442] [deadline 7s] [delay 0s] req@ffff81031aef0400 x1369097809952787/t0(0) o-1->MGS@MGC192.168.4.128@o2ib_0:26/25 lens 192/192 e 0 to 1 dl 1305673442 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 16:04:03 client-5 kernel: Lustre: 21659:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369097809952791 sent from MGC192.168.4.128@o2ib to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305673437] [real_sent 1305673437] [current 1305673443] [deadline 6s] [delay 0s] req@ffff81031aef0000 x1369097809952791/t0(0) o-1->MGS@MGC192.168.4.128@o2ib_0:26/25 lens 192/192 e 0 to 1 dl 1305673443 ref 2 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 16:04:03 client-5 kernel: Lustre: client ffff8102f640dc00 umount complete May 17 16:04:03 client-5 kernel: Lustre: 21714:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 16:04:03 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 16:04:03 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 16:04:03 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -16 May 17 16:04:08 client-5 kernel: LustreError: 21714:0:(client.c:1896:ptlrpc_interrupted_set()) INTERRUPTED SET ffff810317269dc0 May 17 16:04:08 client-5 kernel: LustreError: 21714:0:(lmv_obd.c:1198:lmv_statfs()) can't stat MDS #0 (lustre-MDT0000-mdc-ffff810176bca400), error -16 May 17 16:04:08 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -16 May 17 16:04:08 client-5 kernel: LustreError: 21134:0:(lov_obd.c:924:lov_cleanup()) lov tgt 0 not cleaned! deathrow=0, lovrc=1 May 17 16:04:08 client-5 kernel: LustreError: 21714:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:04:08 client-5 kernel: LustreError: 21714:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:04:08 client-5 kernel: Lustre: client ffff810176bca400 umount complete May 17 16:04:09 client-5 kernel: LustreError: 21714:0:(obd_mount.c:2149:lustre_fill_super()) Unable to mount (-16) May 17 16:04:22 client-5 xinetd[3391]: START: shell pid=21869 from=192.168.4.5 May 17 16:04:23 client-5 rshd[21872]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 16:04:23 client-5 xinetd[3391]: EXIT: shell status=0 pid=21869 duration=1(sec) May 17 16:04:29 client-5 kernel: Lustre: Acceptor stopping May 17 16:04:30 client-5 kernel: Lustre: Removed LNI 10.10.4.5@tcp May 17 16:04:32 client-5 kernel: Lustre: Removed LNI 192.168.4.5@o2ib May 17 16:04:34 client-5 kernel: Lustre: 22391:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:04:34 client-5 kernel: Lustre: 22392:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:04:34 client-5 kernel: Lustre: OBD class driver, http://www.lustre.org/ May 17 16:04:34 client-5 kernel: Lustre: Lustre Version: 2.0.60 May 17 16:04:34 client-5 kernel: Lustre: Build Version: jenkins-g985ac5b-PRISTINE-2.6.18-194.17.1.el5 May 17 16:04:34 client-5 kernel: Lustre: Lustre LU module (ffffffff888eb1c0). May 17 16:04:34 client-5 kernel: Lustre: Register global MR array, MR size: 0xffffffffffffffff, array size: 1 May 17 16:04:34 client-5 kernel: Lustre: Added LNI 192.168.4.5@o2ib [8/64/0/180] May 17 16:04:34 client-5 kernel: Lustre: Added LNI 10.10.4.5@tcp [8/256/0/180] May 17 16:04:34 client-5 kernel: Lustre: Accept all, port 7988 May 17 16:04:34 client-5 kernel: Lustre: Lustre OSC module (ffffffff88b3c6c0). May 17 16:04:34 client-5 kernel: Lustre: Lustre LOV module (ffffffff88bab540). May 17 16:04:34 client-5 kernel: Lustre: Lustre client module (ffffffff88c643a0). May 17 16:04:34 client-5 xinetd[3391]: START: shell pid=22796 from=192.168.4.5 May 17 16:04:34 client-5 rshd[22797]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == conf-sanity test 23b: Simulate -EINTR during mount ================================================ 16:04:34 \(1305673474\)");echo XXRETCODE:$?' May 17 16:04:34 client-5 kernel: Lustre: DEBUG MARKER: == conf-sanity test 23b: Simulate -EINTR during mount ================================================ 16:04:34 (1305673474) May 17 16:04:34 client-5 xinetd[3391]: EXIT: shell status=0 pid=22796 duration=0(sec) May 17 16:04:36 client-5 kernel: Lustre: 22929:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 16:04:36 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 16:04:36 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 16:04:36 client-5 kernel: Lustre: 22929:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff8102fe9d0800->192.168.4.128@o2ib netid 50000: select flavor null May 17 16:04:36 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 16:04:41 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 16:04:46 client-5 kernel: Lustre: 22929:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-OST0000-osc-ffff8102fe9d0800->192.168.4.129@o2ib netid 50000: select flavor null May 17 16:04:46 client-5 kernel: LustreError: 22929:0:(obd_support.h:456:obd_fail_check_set()) *** obd_fail_loc=313 *** May 17 16:04:46 client-5 kernel: LustreError: 22929:0:(llite_lib.c:476:client_common_fill_super()) cannot start close thread: rc -4 May 17 16:04:46 client-5 kernel: LustreError: 22929:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:04:46 client-5 kernel: LustreError: 22929:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:04:46 client-5 kernel: Lustre: client ffff8102fe9d0800 umount complete May 17 16:04:46 client-5 kernel: LustreError: 22929:0:(obd_mount.c:2149:lustre_fill_super()) Unable to mount (-4) May 17 16:05:04 client-5 kernel: Lustre: Acceptor stopping May 17 16:05:05 client-5 kernel: Lustre: Removed LNI 10.10.4.5@tcp May 17 16:05:07 client-5 kernel: Lustre: Removed LNI 192.168.4.5@o2ib May 17 16:05:08 client-5 xinetd[3391]: START: shell pid=23402 from=192.168.4.5 May 17 16:05:08 client-5 rshd[23403]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 16:05:08 client-5 xinetd[3391]: EXIT: shell status=0 pid=23402 duration=0(sec) May 17 16:05:08 client-5 kernel: Lustre: 23537:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:05:08 client-5 kernel: Lustre: 23541:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:05:09 client-5 kernel: Lustre: OBD class driver, http://www.lustre.org/ May 17 16:05:09 client-5 kernel: Lustre: Lustre Version: 2.0.60 May 17 16:05:09 client-5 kernel: Lustre: Build Version: jenkins-g985ac5b-PRISTINE-2.6.18-194.17.1.el5 May 17 16:05:09 client-5 kernel: Lustre: Lustre LU module (ffffffff888eb1c0). May 17 16:05:09 client-5 kernel: Lustre: Register global MR array, MR size: 0xffffffffffffffff, array size: 1 May 17 16:05:09 client-5 kernel: Lustre: Added LNI 192.168.4.5@o2ib [8/64/0/180] May 17 16:05:09 client-5 kernel: Lustre: Added LNI 10.10.4.5@tcp [8/256/0/180] May 17 16:05:09 client-5 kernel: Lustre: Accept all, port 7988 May 17 16:05:09 client-5 kernel: Lustre: Lustre OSC module (ffffffff88b3c6c0). May 17 16:05:09 client-5 kernel: Lustre: Lustre LOV module (ffffffff88bab540). May 17 16:05:09 client-5 kernel: Lustre: Lustre client module (ffffffff88c643a0). May 17 16:05:09 client-5 xinetd[3391]: START: shell pid=23940 from=192.168.4.5 May 17 16:05:09 client-5 rshd[23941]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == conf-sanity test 24a: Multiple MDTs on a single node ============================================== 16:05:08 \(1305673508\)");echo XXRETCODE:$?' May 17 16:05:09 client-5 kernel: Lustre: DEBUG MARKER: == conf-sanity test 24a: Multiple MDTs on a single node ============================================== 16:05:08 (1305673508) May 17 16:05:09 client-5 xinetd[3391]: EXIT: shell status=0 pid=23940 duration=0(sec) May 17 16:05:09 client-5 xinetd[3391]: START: shell pid=24046 from=192.168.4.5 May 17 16:05:09 client-5 rshd[24048]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark SKIP: conf-sanity test_24a mixed loopback and real device not working");echo XXRETCODE:$?' May 17 16:05:09 client-5 kernel: Lustre: DEBUG MARKER: SKIP: conf-sanity test_24a mixed loopback and real device not working May 17 16:05:09 client-5 xinetd[3391]: EXIT: shell status=0 pid=24046 duration=0(sec) May 17 16:05:09 client-5 xinetd[3391]: START: shell pid=24138 from=192.168.4.5 May 17 16:05:09 client-5 rshd[24139]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 16:05:09 client-5 xinetd[3391]: EXIT: shell status=0 pid=24138 duration=0(sec) May 17 16:05:10 client-5 xinetd[3391]: START: shell pid=24310 from=192.168.4.5 May 17 16:05:10 client-5 rshd[24311]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == conf-sanity test 24b: Multiple MGSs on a single node \(should return err\) ========================== 16:05:10 \(1305673510\)");echo XXRETCODE:$?' May 17 16:05:10 client-5 kernel: Lustre: DEBUG MARKER: == conf-sanity test 24b: Multiple MGSs on a single node (should return err) ========================== 16:05:10 (1305673510) May 17 16:05:10 client-5 xinetd[3391]: EXIT: shell status=0 pid=24310 duration=0(sec) May 17 16:05:10 client-5 xinetd[3391]: START: shell pid=24412 from=192.168.4.5 May 17 16:05:10 client-5 rshd[24413]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark SKIP: conf-sanity test_24b mixed loopback and real device not working");echo XXRETCODE:$?' May 17 16:05:10 client-5 kernel: Lustre: DEBUG MARKER: SKIP: conf-sanity test_24b mixed loopback and real device not working May 17 16:05:10 client-5 xinetd[3391]: EXIT: shell status=0 pid=24412 duration=0(sec) May 17 16:05:10 client-5 xinetd[3391]: START: shell pid=24503 from=192.168.4.5 May 17 16:05:10 client-5 rshd[24504]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 16:05:10 client-5 xinetd[3391]: EXIT: shell status=0 pid=24503 duration=0(sec) May 17 16:05:10 client-5 xinetd[3391]: START: shell pid=24675 from=192.168.4.5 May 17 16:05:10 client-5 rshd[24676]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == conf-sanity test 25: Verify modules are referenced ================================================ 16:05:10 \(1305673510\)");echo XXRETCODE:$?' May 17 16:05:10 client-5 kernel: Lustre: DEBUG MARKER: == conf-sanity test 25: Verify modules are referenced ================================================ 16:05:10 (1305673510) May 17 16:05:10 client-5 xinetd[3391]: EXIT: shell status=0 pid=24675 duration=0(sec) May 17 16:05:12 client-5 kernel: Lustre: 24802:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 16:05:12 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 16:05:12 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 16:05:12 client-5 kernel: Lustre: 24802:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff8102f9a96000->192.168.4.128@o2ib netid 50000: select flavor null May 17 16:05:12 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 16:05:17 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 16:05:22 client-5 kernel: Lustre: 24802:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-OST0000-osc-ffff8102f9a96000->192.168.4.129@o2ib netid 50000: select flavor null May 17 16:05:22 client-5 kernel: Lustre: Client lustre-client has started May 17 16:05:22 client-5 kernel: Lustre: 24819:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:05:22 client-5 kernel: LustreError: 24917:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:05:22 client-5 kernel: LustreError: 24917:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:05:22 client-5 kernel: LustreError: 24917:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:05:22 client-5 kernel: LustreError: 24917:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:05:22 client-5 kernel: Lustre: client ffff8102f9a96000 umount complete May 17 16:05:44 client-5 kernel: Lustre: Acceptor stopping May 17 16:05:45 client-5 kernel: Lustre: Removed LNI 10.10.4.5@tcp May 17 16:05:47 client-5 kernel: Lustre: Removed LNI 192.168.4.5@o2ib May 17 16:05:48 client-5 xinetd[3391]: START: shell pid=25307 from=192.168.4.5 May 17 16:05:48 client-5 rshd[25310]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 16:05:48 client-5 xinetd[3391]: EXIT: shell status=0 pid=25307 duration=0(sec) May 17 16:05:48 client-5 kernel: Lustre: 25445:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:05:48 client-5 kernel: Lustre: 25449:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:05:49 client-5 kernel: Lustre: OBD class driver, http://www.lustre.org/ May 17 16:05:49 client-5 kernel: Lustre: Lustre Version: 2.0.60 May 17 16:05:49 client-5 kernel: Lustre: Build Version: jenkins-g985ac5b-PRISTINE-2.6.18-194.17.1.el5 May 17 16:05:49 client-5 kernel: Lustre: Lustre LU module (ffffffff888eb1c0). May 17 16:05:49 client-5 kernel: Lustre: Register global MR array, MR size: 0xffffffffffffffff, array size: 1 May 17 16:05:49 client-5 kernel: Lustre: Added LNI 192.168.4.5@o2ib [8/64/0/180] May 17 16:05:49 client-5 kernel: Lustre: Added LNI 10.10.4.5@tcp [8/256/0/180] May 17 16:05:49 client-5 kernel: Lustre: Accept all, port 7988 May 17 16:05:49 client-5 kernel: Lustre: Lustre OSC module (ffffffff88b3c6c0). May 17 16:05:49 client-5 kernel: Lustre: Lustre LOV module (ffffffff88bab540). May 17 16:05:49 client-5 kernel: Lustre: Lustre client module (ffffffff88c643a0). May 17 16:05:49 client-5 xinetd[3391]: START: shell pid=25847 from=192.168.4.5 May 17 16:05:49 client-5 rshd[25848]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == conf-sanity test 26: MDT startup failure cleans LOV \(should return errs\) ========================== 16:05:48 \(1305673548\)");echo XXRETCODE:$?' May 17 16:05:49 client-5 kernel: Lustre: DEBUG MARKER: == conf-sanity test 26: MDT startup failure cleans LOV (should return errs) ========================== 16:05:48 (1305673548) May 17 16:05:49 client-5 xinetd[3391]: EXIT: shell status=0 pid=25847 duration=0(sec) May 17 16:05:49 client-5 kernel: Lustre: 25891:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:06:04 client-5 kernel: Lustre: Acceptor stopping May 17 16:06:05 client-5 kernel: Lustre: Removed LNI 192.168.4.5@o2ib May 17 16:06:06 client-5 kernel: Lustre: Removed LNI 10.10.4.5@tcp May 17 16:06:07 client-5 xinetd[3391]: START: shell pid=26385 from=192.168.4.5 May 17 16:06:07 client-5 rshd[26388]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 16:06:07 client-5 xinetd[3391]: EXIT: shell status=0 pid=26385 duration=0(sec) May 17 16:06:07 client-5 kernel: Lustre: 26523:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:06:07 client-5 kernel: Lustre: 26524:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:06:07 client-5 kernel: Lustre: OBD class driver, http://www.lustre.org/ May 17 16:06:07 client-5 kernel: Lustre: Lustre Version: 2.0.60 May 17 16:06:07 client-5 kernel: Lustre: Build Version: jenkins-g985ac5b-PRISTINE-2.6.18-194.17.1.el5 May 17 16:06:07 client-5 kernel: Lustre: Lustre LU module (ffffffff888eb1c0). May 17 16:06:07 client-5 kernel: Lustre: Register global MR array, MR size: 0xffffffffffffffff, array size: 1 May 17 16:06:07 client-5 kernel: Lustre: Added LNI 192.168.4.5@o2ib [8/64/0/180] May 17 16:06:07 client-5 kernel: Lustre: Added LNI 10.10.4.5@tcp [8/256/0/180] May 17 16:06:07 client-5 kernel: Lustre: Accept all, port 7988 May 17 16:06:07 client-5 kernel: Lustre: Lustre OSC module (ffffffff88b3c6c0). May 17 16:06:07 client-5 kernel: Lustre: Lustre LOV module (ffffffff88bab540). May 17 16:06:08 client-5 kernel: Lustre: Lustre client module (ffffffff88c643a0). May 17 16:06:08 client-5 xinetd[3391]: START: shell pid=26926 from=192.168.4.5 May 17 16:06:08 client-5 rshd[26927]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == conf-sanity test 27a: Reacquire MGS lock if OST started first ===================================== 16:06:07 \(1305673567\)");echo XXRETCODE:$?' May 17 16:06:08 client-5 kernel: Lustre: DEBUG MARKER: == conf-sanity test 27a: Reacquire MGS lock if OST started first ===================================== 16:06:07 (1305673567) May 17 16:06:08 client-5 xinetd[3391]: EXIT: shell status=0 pid=26926 duration=0(sec) May 17 16:07:53 client-5 kernel: Lustre: Acceptor stopping May 17 16:07:54 client-5 kernel: Lustre: Removed LNI 192.168.4.5@o2ib May 17 16:07:55 client-5 kernel: Lustre: Removed LNI 10.10.4.5@tcp May 17 16:07:56 client-5 xinetd[3391]: START: shell pid=27620 from=192.168.4.5 May 17 16:07:56 client-5 rshd[27621]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 16:07:57 client-5 xinetd[3391]: EXIT: shell status=0 pid=27620 duration=1(sec) May 17 16:07:57 client-5 kernel: Lustre: 27755:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:07:57 client-5 kernel: Lustre: 27757:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:07:57 client-5 kernel: Lustre: OBD class driver, http://www.lustre.org/ May 17 16:07:57 client-5 kernel: Lustre: Lustre Version: 2.0.60 May 17 16:07:57 client-5 kernel: Lustre: Build Version: jenkins-g985ac5b-PRISTINE-2.6.18-194.17.1.el5 May 17 16:07:57 client-5 kernel: Lustre: Lustre LU module (ffffffff888eb1c0). May 17 16:07:57 client-5 kernel: Lustre: Register global MR array, MR size: 0xffffffffffffffff, array size: 1 May 17 16:07:57 client-5 kernel: Lustre: Added LNI 192.168.4.5@o2ib [8/64/0/180] May 17 16:07:57 client-5 kernel: Lustre: Added LNI 10.10.4.5@tcp [8/256/0/180] May 17 16:07:57 client-5 kernel: Lustre: Accept all, port 7988 May 17 16:07:57 client-5 kernel: Lustre: Lustre OSC module (ffffffff88b3c6c0). May 17 16:07:57 client-5 kernel: Lustre: Lustre LOV module (ffffffff88bab540). May 17 16:07:57 client-5 kernel: Lustre: Lustre client module (ffffffff88c643a0). May 17 16:07:57 client-5 xinetd[3391]: START: shell pid=28158 from=192.168.4.5 May 17 16:07:57 client-5 rshd[28159]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == conf-sanity test 27b: Reacquire MGS lock after failover =========================================== 16:07:57 \(1305673677\)");echo XXRETCODE:$?' May 17 16:07:57 client-5 kernel: Lustre: DEBUG MARKER: == conf-sanity test 27b: Reacquire MGS lock after failover =========================================== 16:07:57 (1305673677) May 17 16:07:57 client-5 xinetd[3391]: EXIT: shell status=0 pid=28158 duration=0(sec) May 17 16:07:59 client-5 kernel: Lustre: 28290:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 16:07:59 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 16:07:59 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 16:07:59 client-5 kernel: Lustre: 28290:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff81031d59ac00->192.168.4.128@o2ib netid 50000: select flavor null May 17 16:07:59 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 16:08:04 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 16:08:09 client-5 kernel: Lustre: 28290:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-OST0000-osc-ffff81031d59ac00->192.168.4.129@o2ib netid 50000: select flavor null May 17 16:08:09 client-5 kernel: Lustre: Client lustre-client has started May 17 16:08:09 client-5 kernel: Lustre: 28307:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:08:21 client-5 kernel: Lustre: 27862:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369098081533971 sent from MGC192.168.4.128@o2ib to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305673694] [real_sent 1305673694] [current 1305673701] [deadline 7s] [delay 0s] req@ffff8102ff15c800 x1369098081533971/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 192/192 e 0 to 1 dl 1305673701 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 16:08:21 client-5 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail. May 17 16:08:21 client-5 kernel: Lustre: 27862:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369098081533972 sent from lustre-MDT0000-mdc-ffff81031d59ac00 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305673694] [real_sent 1305673694] [current 1305673701] [deadline 7s] [delay 0s] req@ffff81031afc2c00 x1369098081533972/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 192/192 e 0 to 1 dl 1305673701 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 16:08:21 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff81031d59ac00: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 16:08:26 client-5 kernel: Lustre: 27862:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369098081533975 sent from lustre-MDT0000-mdc-ffff81031d59ac00 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305673699] [real_sent 1305673699] [current 1305673706] [deadline 7s] [delay 0s] req@ffff810308c84400 x1369098081533975/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 192/192 e 0 to 1 dl 1305673706 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 16:08:31 client-5 kernel: Lustre: 27864:0:(import.c:529:import_select_connection()) MGC192.168.4.128@o2ib: tried all connections, increasing latency to 6s May 17 16:08:31 client-5 kernel: Lustre: 27864:0:(import.c:529:import_select_connection()) lustre-MDT0000-mdc-ffff81031d59ac00: tried all connections, increasing latency to 6s May 17 16:08:31 client-5 kernel: Lustre: 27863:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74f2e8b5 to 0x62e5a8d74f2e98e May 17 16:08:31 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 16:08:31 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Connection restored to service MGS using nid 192.168.4.128@o2ib. May 17 16:08:31 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff81031d59ac00: Connection restored to service lustre-MDT0000 using nid 192.168.4.128@o2ib. May 17 16:08:37 client-5 kernel: LustreError: 28532:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:08:37 client-5 kernel: LustreError: 28532:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:08:37 client-5 kernel: LustreError: 28532:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:08:37 client-5 kernel: LustreError: 28532:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:08:37 client-5 kernel: Lustre: client ffff81031d59ac00 umount complete May 17 16:08:59 client-5 kernel: Lustre: Acceptor stopping May 17 16:09:00 client-5 kernel: Lustre: Removed LNI 10.10.4.5@tcp May 17 16:09:02 client-5 kernel: Lustre: Removed LNI 192.168.4.5@o2ib May 17 16:09:03 client-5 xinetd[3391]: START: shell pid=28984 from=192.168.4.5 May 17 16:09:03 client-5 rshd[28986]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 16:09:03 client-5 xinetd[3391]: EXIT: shell status=0 pid=28984 duration=0(sec) May 17 16:09:04 client-5 kernel: Lustre: 29120:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:09:04 client-5 kernel: Lustre: 29122:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:09:04 client-5 kernel: Lustre: OBD class driver, http://www.lustre.org/ May 17 16:09:04 client-5 kernel: Lustre: Lustre Version: 2.0.60 May 17 16:09:04 client-5 kernel: Lustre: Build Version: jenkins-g985ac5b-PRISTINE-2.6.18-194.17.1.el5 May 17 16:09:04 client-5 kernel: Lustre: Lustre LU module (ffffffff888eb1c0). May 17 16:09:04 client-5 kernel: Lustre: Register global MR array, MR size: 0xffffffffffffffff, array size: 1 May 17 16:09:04 client-5 kernel: Lustre: Added LNI 192.168.4.5@o2ib [8/64/0/180] May 17 16:09:04 client-5 kernel: Lustre: Added LNI 10.10.4.5@tcp [8/256/0/180] May 17 16:09:04 client-5 kernel: Lustre: Accept all, port 7988 May 17 16:09:04 client-5 kernel: Lustre: Lustre OSC module (ffffffff88b3c6c0). May 17 16:09:04 client-5 kernel: Lustre: Lustre LOV module (ffffffff88bab540). May 17 16:09:04 client-5 kernel: Lustre: Lustre client module (ffffffff88c643a0). May 17 16:09:04 client-5 xinetd[3391]: START: shell pid=29520 from=192.168.4.5 May 17 16:09:04 client-5 rshd[29523]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == conf-sanity test 28: permanent parameter setting ================================================== 16:09:04 \(1305673744\)");echo XXRETCODE:$?' May 17 16:09:04 client-5 kernel: Lustre: DEBUG MARKER: == conf-sanity test 28: permanent parameter setting ================================================== 16:09:04 (1305673744) May 17 16:09:04 client-5 xinetd[3391]: EXIT: shell status=0 pid=29520 duration=0(sec) May 17 16:09:06 client-5 kernel: Lustre: 29655:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 16:09:06 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 16:09:06 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 16:09:06 client-5 kernel: Lustre: 29655:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff8102f62b2800->192.168.4.128@o2ib netid 50000: select flavor null May 17 16:09:06 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 16:09:11 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 16:09:16 client-5 kernel: Lustre: 29655:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-OST0000-osc-ffff8102f62b2800->192.168.4.129@o2ib netid 50000: select flavor null May 17 16:09:16 client-5 kernel: Lustre: Client lustre-client has started May 17 16:09:16 client-5 kernel: Lustre: 29672:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:09:26 client-5 kernel: LustreError: 29754:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:09:26 client-5 kernel: LustreError: 29754:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:09:26 client-5 kernel: LustreError: 29754:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:09:26 client-5 kernel: LustreError: 29754:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:09:26 client-5 kernel: Lustre: client ffff8102f62b2800 umount complete May 17 16:09:26 client-5 kernel: Lustre: 29764:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 16:09:26 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 16:09:26 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 16:09:26 client-5 kernel: Lustre: Client lustre-client has started May 17 16:09:26 client-5 kernel: Lustre: 29781:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:09:26 client-5 kernel: Lustre: 29781:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message May 17 16:09:32 client-5 kernel: LustreError: 29834:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:09:32 client-5 kernel: LustreError: 29834:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:09:32 client-5 kernel: Lustre: client ffff81033f897000 umount complete May 17 16:09:47 client-5 kernel: Lustre: Acceptor stopping May 17 16:09:48 client-5 kernel: Lustre: Removed LNI 10.10.4.5@tcp May 17 16:09:50 client-5 kernel: Lustre: Removed LNI 192.168.4.5@o2ib May 17 16:09:52 client-5 xinetd[3391]: START: shell pid=30289 from=192.168.4.5 May 17 16:09:52 client-5 rshd[30290]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 16:09:52 client-5 xinetd[3391]: EXIT: shell status=0 pid=30289 duration=0(sec) May 17 16:09:52 client-5 kernel: Lustre: 30426:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:09:52 client-5 kernel: Lustre: 30427:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:09:52 client-5 kernel: Lustre: OBD class driver, http://www.lustre.org/ May 17 16:09:52 client-5 kernel: Lustre: Lustre Version: 2.0.60 May 17 16:09:52 client-5 kernel: Lustre: Build Version: jenkins-g985ac5b-PRISTINE-2.6.18-194.17.1.el5 May 17 16:09:52 client-5 kernel: Lustre: Lustre LU module (ffffffff888eb1c0). May 17 16:09:52 client-5 kernel: Lustre: Register global MR array, MR size: 0xffffffffffffffff, array size: 1 May 17 16:09:52 client-5 kernel: Lustre: Added LNI 192.168.4.5@o2ib [8/64/0/180] May 17 16:09:52 client-5 kernel: Lustre: Added LNI 10.10.4.5@tcp [8/256/0/180] May 17 16:09:52 client-5 kernel: Lustre: Accept all, port 7988 May 17 16:09:52 client-5 kernel: Lustre: Lustre OSC module (ffffffff88b3c6c0). May 17 16:09:52 client-5 kernel: Lustre: Lustre LOV module (ffffffff88bab540). May 17 16:09:52 client-5 kernel: Lustre: Lustre client module (ffffffff88c643a0). May 17 16:09:52 client-5 xinetd[3391]: START: shell pid=30827 from=192.168.4.5 May 17 16:09:53 client-5 rshd[30828]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == conf-sanity test 29: permanently remove an OST ==================================================== 16:09:52 \(1305673792\)");echo XXRETCODE:$?' May 17 16:09:53 client-5 kernel: Lustre: DEBUG MARKER: == conf-sanity test 29: permanently remove an OST ==================================================== 16:09:52 (1305673792) May 17 16:09:53 client-5 xinetd[3391]: EXIT: shell status=0 pid=30827 duration=1(sec) May 17 16:09:54 client-5 kernel: Lustre: 30960:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 16:09:54 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 16:09:54 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 16:09:54 client-5 kernel: Lustre: 30960:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff8102d45ae800->192.168.4.128@o2ib netid 50000: select flavor null May 17 16:09:54 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 16:09:59 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 16:10:04 client-5 kernel: Lustre: 30960:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-OST0000-osc-ffff8102d45ae800->192.168.4.129@o2ib netid 50000: select flavor null May 17 16:10:04 client-5 kernel: Lustre: Client lustre-client has started May 17 16:10:04 client-5 kernel: Lustre: 30977:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:10:09 client-5 kernel: Lustre: 31024:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-OST0001-osc-ffff8102d45ae800->192.168.4.129@o2ib netid 50000: select flavor null May 17 16:10:19 client-5 kernel: Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request May 17 16:10:26 client-5 kernel: LustreError: 31155:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:10:26 client-5 kernel: LustreError: 31155:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:10:26 client-5 kernel: LustreError: 31155:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:10:26 client-5 kernel: LustreError: 31155:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:10:26 client-5 kernel: Lustre: client ffff8102d45ae800 umount complete May 17 16:10:26 client-5 kernel: Lustre: 31165:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 16:10:26 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 16:10:26 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 16:10:26 client-5 kernel: Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request May 17 16:10:26 client-5 kernel: Lustre: Client lustre-client has started May 17 16:10:26 client-5 kernel: Lustre: 31182:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:10:26 client-5 kernel: Lustre: 31182:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message May 17 16:10:29 client-5 kernel: LustreError: 31214:0:(lov_obd.c:482:lov_notify()) event(11) of lustre-OST0001_UUID failed: -22 May 17 16:10:31 client-5 kernel: LustreError: 31238:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:10:31 client-5 kernel: LustreError: 31238:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:10:31 client-5 kernel: LustreError: 30485:0:(lov_obd.c:924:lov_cleanup()) lov tgt 1 not cleaned! deathrow=0, lovrc=1 May 17 16:10:31 client-5 kernel: Lustre: client ffff81031b694c00 umount complete May 17 16:10:48 client-5 kernel: Lustre: Acceptor stopping May 17 16:10:49 client-5 kernel: Lustre: Removed LNI 10.10.4.5@tcp May 17 16:10:51 client-5 kernel: Lustre: Removed LNI 192.168.4.5@o2ib May 17 16:10:52 client-5 kernel: Lustre: 31686:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:10:52 client-5 kernel: Lustre: 31687:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:10:52 client-5 kernel: Lustre: OBD class driver, http://www.lustre.org/ May 17 16:10:52 client-5 kernel: Lustre: Lustre Version: 2.0.60 May 17 16:10:52 client-5 kernel: Lustre: Build Version: jenkins-g985ac5b-PRISTINE-2.6.18-194.17.1.el5 May 17 16:10:52 client-5 kernel: Lustre: Lustre LU module (ffffffff888eb1c0). May 17 16:10:52 client-5 kernel: Lustre: Register global MR array, MR size: 0xffffffffffffffff, array size: 1 May 17 16:10:52 client-5 kernel: Lustre: Added LNI 192.168.4.5@o2ib [8/64/0/180] May 17 16:10:52 client-5 kernel: Lustre: Added LNI 10.10.4.5@tcp [8/256/0/180] May 17 16:10:52 client-5 kernel: Lustre: Accept all, port 7988 May 17 16:10:52 client-5 kernel: Lustre: Lustre OSC module (ffffffff88b3c6c0). May 17 16:10:52 client-5 kernel: Lustre: Lustre LOV module (ffffffff88bab540). May 17 16:10:53 client-5 kernel: Lustre: Lustre client module (ffffffff88c643a0). May 17 16:10:54 client-5 xinetd[3391]: START: shell pid=32189 from=192.168.4.5 May 17 16:10:54 client-5 rshd[32192]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 16:10:54 client-5 xinetd[3391]: EXIT: shell status=0 pid=32189 duration=0(sec) May 17 16:10:54 client-5 xinetd[3391]: START: shell pid=32363 from=192.168.4.5 May 17 16:10:54 client-5 rshd[32364]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == conf-sanity test 30a: Big config llog and conf_param deletion ===================================== 16:10:54 \(1305673854\)");echo XXRETCODE:$?' May 17 16:10:54 client-5 kernel: Lustre: DEBUG MARKER: == conf-sanity test 30a: Big config llog and conf_param deletion ===================================== 16:10:54 (1305673854) May 17 16:10:54 client-5 xinetd[3391]: EXIT: shell status=0 pid=32363 duration=0(sec) May 17 16:10:56 client-5 kernel: Lustre: 32490:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 16:10:56 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 16:10:56 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 16:10:56 client-5 kernel: Lustre: 32490:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff81031bf06800->192.168.4.128@o2ib netid 50000: select flavor null May 17 16:10:56 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 16:11:01 client-5 kernel: Lustre: 32490:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-OST0000-osc-ffff81031bf06800->192.168.4.129@o2ib netid 50000: select flavor null May 17 16:11:01 client-5 kernel: Lustre: Client lustre-client has started May 17 16:11:01 client-5 kernel: Lustre: 32507:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:12:49 client-5 kernel: LustreError: 709:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:12:49 client-5 kernel: LustreError: 709:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:12:49 client-5 kernel: LustreError: 709:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:12:49 client-5 kernel: LustreError: 709:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:12:49 client-5 kernel: Lustre: client ffff81031bf06800 umount complete May 17 16:12:49 client-5 kernel: Lustre: 719:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 16:12:49 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 16:12:49 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 16:12:49 client-5 kernel: Lustre: Client lustre-client has started May 17 16:12:49 client-5 kernel: Lustre: 736:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:12:49 client-5 kernel: Lustre: 736:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message May 17 16:12:49 client-5 kernel: Lustre: client ffff810325be1400 umount complete May 17 16:12:49 client-5 kernel: LustreError: 819:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:12:49 client-5 kernel: LustreError: 819:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Skipped 1 previous similar message May 17 16:12:49 client-5 kernel: LustreError: 819:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:12:49 client-5 kernel: LustreError: 819:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) Skipped 1 previous similar message May 17 16:13:07 client-5 kernel: Lustre: Acceptor stopping May 17 16:13:08 client-5 kernel: Lustre: Removed LNI 10.10.4.5@tcp May 17 16:13:10 client-5 kernel: Lustre: Removed LNI 192.168.4.5@o2ib May 17 16:13:11 client-5 xinetd[3391]: START: shell pid=1274 from=192.168.4.5 May 17 16:13:11 client-5 rshd[1275]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 16:13:12 client-5 xinetd[3391]: EXIT: shell status=0 pid=1274 duration=1(sec) May 17 16:13:12 client-5 kernel: Lustre: 1410:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:13:12 client-5 kernel: Lustre: 1413:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:13:12 client-5 kernel: Lustre: OBD class driver, http://www.lustre.org/ May 17 16:13:12 client-5 kernel: Lustre: Lustre Version: 2.0.60 May 17 16:13:12 client-5 kernel: Lustre: Build Version: jenkins-g985ac5b-PRISTINE-2.6.18-194.17.1.el5 May 17 16:13:12 client-5 kernel: Lustre: Lustre LU module (ffffffff888eb1c0). May 17 16:13:12 client-5 kernel: Lustre: Register global MR array, MR size: 0xffffffffffffffff, array size: 1 May 17 16:13:12 client-5 kernel: Lustre: Added LNI 192.168.4.5@o2ib [8/64/0/180] May 17 16:13:12 client-5 kernel: Lustre: Added LNI 10.10.4.5@tcp [8/256/0/180] May 17 16:13:12 client-5 kernel: Lustre: Accept all, port 7988 May 17 16:13:12 client-5 kernel: Lustre: Lustre OSC module (ffffffff88b3c6c0). May 17 16:13:12 client-5 kernel: Lustre: Lustre LOV module (ffffffff88bab540). May 17 16:13:12 client-5 kernel: Lustre: Lustre client module (ffffffff88c643a0). May 17 16:13:12 client-5 xinetd[3391]: START: shell pid=1815 from=192.168.4.5 May 17 16:13:12 client-5 rshd[1817]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == conf-sanity test 30b: Remove failover nids ======================================================== 16:13:12 \(1305673992\)");echo XXRETCODE:$?' May 17 16:13:12 client-5 kernel: Lustre: DEBUG MARKER: == conf-sanity test 30b: Remove failover nids ======================================================== 16:13:12 (1305673992) May 17 16:13:12 client-5 xinetd[3391]: EXIT: shell status=0 pid=1815 duration=0(sec) May 17 16:13:14 client-5 kernel: Lustre: 1953:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 16:13:14 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 16:13:14 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 16:13:14 client-5 kernel: Lustre: 1953:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff81030ac9b000->192.168.4.128@o2ib netid 50000: select flavor null May 17 16:13:14 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 16:13:19 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 16:13:24 client-5 kernel: Lustre: 1953:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-OST0000-osc-ffff81030ac9b000->192.168.4.129@o2ib netid 50000: select flavor null May 17 16:13:24 client-5 kernel: Lustre: Client lustre-client has started May 17 16:13:24 client-5 kernel: Lustre: 1970:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:13:30 client-5 kernel: LustreError: 2067:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:13:30 client-5 kernel: LustreError: 2067:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:13:30 client-5 kernel: Lustre: client ffff81030ac9b000 umount complete May 17 16:13:30 client-5 kernel: Lustre: 2077:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 16:13:30 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 16:13:30 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 16:13:30 client-5 kernel: Lustre: Client lustre-client has started May 17 16:13:30 client-5 kernel: Lustre: 2094:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:13:30 client-5 kernel: Lustre: 2094:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message May 17 16:13:30 client-5 kernel: LustreError: 2122:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:13:30 client-5 kernel: LustreError: 2122:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:13:30 client-5 kernel: Lustre: client ffff8102d7fad800 umount complete May 17 16:13:50 client-5 kernel: Lustre: Acceptor stopping May 17 16:13:51 client-5 kernel: Lustre: Removed LNI 10.10.4.5@tcp May 17 16:13:53 client-5 kernel: Lustre: Removed LNI 192.168.4.5@o2ib May 17 16:13:54 client-5 xinetd[3391]: START: shell pid=2605 from=192.168.4.5 May 17 16:13:54 client-5 rshd[2608]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 16:13:54 client-5 xinetd[3391]: EXIT: shell status=0 pid=2605 duration=0(sec) May 17 16:13:54 client-5 kernel: Lustre: 2743:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:13:54 client-5 kernel: Lustre: 2745:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:13:54 client-5 kernel: Lustre: OBD class driver, http://www.lustre.org/ May 17 16:13:54 client-5 kernel: Lustre: Lustre Version: 2.0.60 May 17 16:13:54 client-5 kernel: Lustre: Build Version: jenkins-g985ac5b-PRISTINE-2.6.18-194.17.1.el5 May 17 16:13:54 client-5 kernel: Lustre: Lustre LU module (ffffffff888eb1c0). May 17 16:13:54 client-5 kernel: Lustre: Register global MR array, MR size: 0xffffffffffffffff, array size: 1 May 17 16:13:54 client-5 kernel: Lustre: Added LNI 192.168.4.5@o2ib [8/64/0/180] May 17 16:13:54 client-5 kernel: Lustre: Added LNI 10.10.4.5@tcp [8/256/0/180] May 17 16:13:54 client-5 kernel: Lustre: Accept all, port 7988 May 17 16:13:55 client-5 kernel: Lustre: Lustre OSC module (ffffffff88b3c6c0). May 17 16:13:55 client-5 kernel: Lustre: Lustre LOV module (ffffffff88bab540). May 17 16:13:55 client-5 kernel: Lustre: Lustre client module (ffffffff88c643a0). May 17 16:13:55 client-5 xinetd[3391]: START: shell pid=3161 from=192.168.4.5 May 17 16:13:55 client-5 rshd[3162]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == conf-sanity test 31: Connect to non-existent node \(shouldn\'t crash\) =============================== 16:13:54 \(1305674034\)");echo XXRETCODE:$?' May 17 16:13:55 client-5 kernel: Lustre: DEBUG MARKER: == conf-sanity test 31: Connect to non-existent node (shouldn't crash) =============================== 16:13:54 (1305674034) May 17 16:13:55 client-5 xinetd[3391]: EXIT: shell status=0 pid=3161 duration=0(sec) May 17 16:13:55 client-5 kernel: Lustre: 3207:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC4.3.2.1@tcp->MGC4.3.2.1@tcp_0 netid 20000: select flavor null May 17 16:14:00 client-5 kernel: Lustre: 2851:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369098455875585 sent from MGC4.3.2.1@tcp to NID 4.3.2.1@tcp has timed out for sent delay: [sent 1305674035] [real_sent 0] [current 1305674040] [deadline 5s] [delay 0s] req@ffff810319eb9800 x1369098455875585/t0(0) o-1->MGS@MGC4.3.2.1@tcp_0:26/25 lens 368/392 e 0 to 1 dl 1305674040 ref 2 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 16:14:01 client-5 kernel: LustreError: 3207:0:(client.c:1046:ptlrpc_import_delay_req()) @@@ send limit expired req@ffff810319eb9400 x1369098455875586/t0(0) o-1->MGS@MGC4.3.2.1@tcp_0:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:W/ffffffff/ffffffff rc 0/-1 May 17 16:14:11 client-5 kernel: LustreError: 3218:0:(client.c:1046:ptlrpc_import_delay_req()) @@@ send limit expired req@ffff81031c880000 x1369098455875588/t0(0) o-1->MGS@MGC4.3.2.1@tcp_0:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:W/ffffffff/ffffffff rc 0/-1 May 17 16:14:20 client-5 kernel: Lustre: 2852:0:(import.c:529:import_select_connection()) MGC4.3.2.1@tcp: tried all connections, increasing latency to 5s May 17 16:14:20 client-5 kernel: LustreError: 3218:0:(client.c:1046:ptlrpc_import_delay_req()) @@@ send limit expired req@ffff81031c880000 x1369098455875589/t0(0) o-1->MGS@MGC4.3.2.1@tcp_0:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:W/ffffffff/ffffffff rc 0/-1 May 17 16:14:29 client-5 kernel: LustreError: 3218:0:(client.c:1046:ptlrpc_import_delay_req()) @@@ send limit expired req@ffff810319eb9000 x1369098455875591/t0(0) o-1->MGS@MGC4.3.2.1@tcp_0:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:W/ffffffff/ffffffff rc 0/-1 May 17 16:14:30 client-5 kernel: Lustre: 2851:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369098455875590 sent from MGC4.3.2.1@tcp to NID 4.3.2.1@tcp has timed out for sent delay: [sent 1305674060] [real_sent 0] [current 1305674070] [deadline 10s] [delay 0s] req@ffff810313571c00 x1369098455875590/t0(0) o-1->MGS@MGC4.3.2.1@tcp_0:26/25 lens 368/392 e 0 to 1 dl 1305674070 ref 2 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 16:14:40 client-5 kernel: LustreError: 3218:0:(client.c:1046:ptlrpc_import_delay_req()) @@@ send limit expired req@ffff810307997400 x1369098455875592/t0(0) o-1->MGS@MGC4.3.2.1@tcp_0:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:W/ffffffff/ffffffff rc 0/-1 May 17 16:14:50 client-5 kernel: LustreError: 3218:0:(client.c:1046:ptlrpc_import_delay_req()) @@@ send limit expired req@ffff810307997400 x1369098455875593/t0(0) o-1->MGS@MGC4.3.2.1@tcp_0:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:W/ffffffff/ffffffff rc 0/-1 May 17 16:14:52 client-5 kernel: LustreError: 156-2: The client profile 'lustre-client' could not be read from the MGS. Does that filesystem exist? May 17 16:14:52 client-5 kernel: Lustre: client ffff810324cba000 umount complete May 17 16:14:52 client-5 kernel: LustreError: 3207:0:(obd_mount.c:2149:lustre_fill_super()) Unable to mount (-22) May 17 16:15:30 client-5 kernel: Lustre: Acceptor stopping May 17 16:15:31 client-5 kernel: Lustre: Removed LNI 192.168.4.5@o2ib May 17 16:15:32 client-5 kernel: Lustre: 3579:0:(socklnd.c:2538:ksocknal_shutdown()) waiting for 1 peers to disconnect May 17 16:16:32 client-5 last message repeated 4 times May 17 16:17:05 client-5 kernel: Lustre: Removed LNI 10.10.4.5@tcp May 17 16:17:06 client-5 xinetd[3391]: START: shell pid=3759 from=192.168.4.5 May 17 16:17:06 client-5 rshd[3760]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 16:17:06 client-5 xinetd[3391]: EXIT: shell status=0 pid=3759 duration=0(sec) May 17 16:17:06 client-5 kernel: Lustre: 3895:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:17:06 client-5 kernel: Lustre: 3896:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:17:07 client-5 kernel: Lustre: OBD class driver, http://www.lustre.org/ May 17 16:17:07 client-5 kernel: Lustre: Lustre Version: 2.0.60 May 17 16:17:07 client-5 kernel: Lustre: Build Version: jenkins-g985ac5b-PRISTINE-2.6.18-194.17.1.el5 May 17 16:17:07 client-5 kernel: Lustre: Lustre LU module (ffffffff888eb1c0). May 17 16:17:07 client-5 kernel: Lustre: Register global MR array, MR size: 0xffffffffffffffff, array size: 1 May 17 16:17:07 client-5 kernel: Lustre: Added LNI 192.168.4.5@o2ib [8/64/0/180] May 17 16:17:07 client-5 kernel: Lustre: Added LNI 10.10.4.5@tcp [8/256/0/180] May 17 16:17:07 client-5 kernel: Lustre: Accept all, port 7988 May 17 16:17:07 client-5 kernel: Lustre: Lustre OSC module (ffffffff88b3c6c0). May 17 16:17:07 client-5 kernel: Lustre: Lustre LOV module (ffffffff88bab540). May 17 16:17:07 client-5 kernel: Lustre: Lustre client module (ffffffff88c643a0). May 17 16:17:07 client-5 xinetd[3391]: START: shell pid=4295 from=192.168.4.5 May 17 16:17:07 client-5 rshd[4297]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == conf-sanity test 32a: Upgrade from 1.8 \(not live\) ================================================= 16:17:06 \(1305674226\)");echo XXRETCODE:$?' May 17 16:17:07 client-5 kernel: Lustre: DEBUG MARKER: == conf-sanity test 32a: Upgrade from 1.8 (not live) ================================================= 16:17:06 (1305674226) May 17 16:17:07 client-5 xinetd[3391]: EXIT: shell status=0 pid=4295 duration=0(sec) May 17 16:17:07 client-5 xinetd[3391]: START: shell pid=4394 from=192.168.4.5 May 17 16:17:07 client-5 rshd[4395]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark SKIP: conf-sanity test_32a client only testing");echo XXRETCODE:$?' May 17 16:17:07 client-5 kernel: Lustre: DEBUG MARKER: SKIP: conf-sanity test_32a client only testing May 17 16:17:07 client-5 xinetd[3391]: EXIT: shell status=0 pid=4394 duration=0(sec) May 17 16:17:07 client-5 xinetd[3391]: START: shell pid=4486 from=192.168.4.5 May 17 16:17:07 client-5 rshd[4487]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 16:17:07 client-5 xinetd[3391]: EXIT: shell status=0 pid=4486 duration=0(sec) May 17 16:17:08 client-5 xinetd[3391]: START: shell pid=4658 from=192.168.4.5 May 17 16:17:08 client-5 rshd[4659]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == conf-sanity test 32b: Upgrade from 1.8 with writeconf ============================================= 16:17:08 \(1305674228\)");echo XXRETCODE:$?' May 17 16:17:08 client-5 kernel: Lustre: DEBUG MARKER: == conf-sanity test 32b: Upgrade from 1.8 with writeconf ============================================= 16:17:08 (1305674228) May 17 16:17:08 client-5 xinetd[3391]: EXIT: shell status=0 pid=4658 duration=0(sec) May 17 16:17:08 client-5 xinetd[3391]: START: shell pid=4751 from=192.168.4.5 May 17 16:17:08 client-5 rshd[4752]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark SKIP: conf-sanity test_32b client only testing");echo XXRETCODE:$?' May 17 16:17:08 client-5 kernel: Lustre: DEBUG MARKER: SKIP: conf-sanity test_32b client only testing May 17 16:17:08 client-5 xinetd[3391]: EXIT: shell status=0 pid=4751 duration=0(sec) May 17 16:17:08 client-5 xinetd[3391]: START: shell pid=4840 from=192.168.4.5 May 17 16:17:08 client-5 rshd[4843]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 16:17:08 client-5 xinetd[3391]: EXIT: shell status=0 pid=4840 duration=0(sec) May 17 16:17:08 client-5 xinetd[3391]: START: shell pid=5014 from=192.168.4.5 May 17 16:17:08 client-5 rshd[5015]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == conf-sanity test 33a: Mount ost with a large index number ========================================= 16:17:08 \(1305674228\)");echo XXRETCODE:$?' May 17 16:17:08 client-5 kernel: Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================= 16:17:08 (1305674228) May 17 16:17:08 client-5 xinetd[3391]: EXIT: shell status=0 pid=5014 duration=0(sec) May 17 16:17:09 client-5 xinetd[3391]: START: shell pid=5116 from=192.168.4.5 May 17 16:17:09 client-5 rshd[5117]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark SKIP: conf-sanity test_33a mixed loopback and real device not working");echo XXRETCODE:$?' May 17 16:17:09 client-5 kernel: Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working May 17 16:17:09 client-5 xinetd[3391]: EXIT: shell status=0 pid=5116 duration=0(sec) May 17 16:17:09 client-5 xinetd[3391]: START: shell pid=5207 from=192.168.4.5 May 17 16:17:09 client-5 rshd[5208]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 16:17:09 client-5 xinetd[3391]: EXIT: shell status=0 pid=5207 duration=0(sec) May 17 16:17:09 client-5 xinetd[3391]: START: shell pid=5379 from=192.168.4.5 May 17 16:17:09 client-5 rshd[5380]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == conf-sanity test 33b: Drop cancel during umount =================================================== 16:17:09 \(1305674229\)");echo XXRETCODE:$?' May 17 16:17:09 client-5 kernel: Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount =================================================== 16:17:09 (1305674229) May 17 16:17:09 client-5 xinetd[3391]: EXIT: shell status=0 pid=5379 duration=0(sec) May 17 16:17:11 client-5 kernel: Lustre: 5506:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 16:17:11 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 16:17:11 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 16:17:11 client-5 kernel: Lustre: 5506:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff8102f803e000->192.168.4.128@o2ib netid 50000: select flavor null May 17 16:17:11 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 16:17:16 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 16:17:21 client-5 kernel: Lustre: 5506:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-OST0000-osc-ffff8102f803e000->192.168.4.129@o2ib netid 50000: select flavor null May 17 16:17:21 client-5 kernel: Lustre: Client lustre-client has started May 17 16:17:21 client-5 kernel: Lustre: 5523:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:17:21 client-5 kernel: LustreError: 5556:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:17:21 client-5 kernel: LustreError: 5556:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:17:21 client-5 kernel: LustreError: 5556:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:17:21 client-5 kernel: LustreError: 5556:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:17:21 client-5 kernel: Lustre: client ffff8102f803e000 umount complete May 17 16:17:42 client-5 kernel: Lustre: Acceptor stopping May 17 16:17:43 client-5 kernel: Lustre: Removed LNI 10.10.4.5@tcp May 17 16:17:45 client-5 kernel: Lustre: Removed LNI 192.168.4.5@o2ib May 17 16:17:46 client-5 xinetd[3391]: START: shell pid=6018 from=192.168.4.5 May 17 16:17:46 client-5 rshd[6019]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 16:17:46 client-5 xinetd[3391]: EXIT: shell status=0 pid=6018 duration=0(sec) May 17 16:17:46 client-5 kernel: Lustre: 6154:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:17:46 client-5 kernel: Lustre: 6156:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:17:46 client-5 kernel: Lustre: OBD class driver, http://www.lustre.org/ May 17 16:17:46 client-5 kernel: Lustre: Lustre Version: 2.0.60 May 17 16:17:46 client-5 kernel: Lustre: Build Version: jenkins-g985ac5b-PRISTINE-2.6.18-194.17.1.el5 May 17 16:17:46 client-5 kernel: Lustre: Lustre LU module (ffffffff888eb1c0). May 17 16:17:46 client-5 kernel: Lustre: Register global MR array, MR size: 0xffffffffffffffff, array size: 1 May 17 16:17:46 client-5 kernel: Lustre: Added LNI 192.168.4.5@o2ib [8/64/0/180] May 17 16:17:46 client-5 kernel: Lustre: Added LNI 10.10.4.5@tcp [8/256/0/180] May 17 16:17:46 client-5 kernel: Lustre: Accept all, port 7988 May 17 16:17:47 client-5 kernel: Lustre: Lustre OSC module (ffffffff88b3c6c0). May 17 16:17:47 client-5 kernel: Lustre: Lustre LOV module (ffffffff88bab540). May 17 16:17:47 client-5 kernel: Lustre: Lustre client module (ffffffff88c643a0). May 17 16:17:47 client-5 xinetd[3391]: START: shell pid=6558 from=192.168.4.5 May 17 16:17:47 client-5 rshd[6559]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == conf-sanity test 34a: umount with opened file should be fail ====================================== 16:17:46 \(1305674266\)");echo XXRETCODE:$?' May 17 16:17:47 client-5 kernel: Lustre: DEBUG MARKER: == conf-sanity test 34a: umount with opened file should be fail ====================================== 16:17:46 (1305674266) May 17 16:17:47 client-5 xinetd[3391]: EXIT: shell status=0 pid=6558 duration=0(sec) May 17 16:17:48 client-5 kernel: Lustre: 6689:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 16:17:48 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 16:17:48 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 16:17:48 client-5 kernel: Lustre: 6689:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff8101d88dcc00->192.168.4.128@o2ib netid 50000: select flavor null May 17 16:17:48 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 16:17:53 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 16:17:58 client-5 kernel: Lustre: 6689:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-OST0000-osc-ffff8101d88dcc00->192.168.4.129@o2ib netid 50000: select flavor null May 17 16:17:58 client-5 kernel: Lustre: Client lustre-client has started May 17 16:17:58 client-5 kernel: Lustre: 6706:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:18:00 client-5 kernel: LustreError: 6752:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:18:00 client-5 kernel: LustreError: 6752:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:18:00 client-5 kernel: LustreError: 6752:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:18:00 client-5 kernel: LustreError: 6752:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:18:00 client-5 kernel: Lustre: client ffff8101d88dcc00 umount complete May 17 16:18:18 client-5 kernel: Lustre: Acceptor stopping May 17 16:18:19 client-5 kernel: Lustre: Removed LNI 10.10.4.5@tcp May 17 16:18:21 client-5 kernel: Lustre: Removed LNI 192.168.4.5@o2ib May 17 16:18:23 client-5 xinetd[3391]: START: shell pid=7202 from=192.168.4.5 May 17 16:18:23 client-5 rshd[7204]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 16:18:23 client-5 xinetd[3391]: EXIT: shell status=0 pid=7202 duration=0(sec) May 17 16:18:23 client-5 kernel: Lustre: 7339:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:18:23 client-5 kernel: Lustre: 7341:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:18:23 client-5 kernel: Lustre: OBD class driver, http://www.lustre.org/ May 17 16:18:23 client-5 kernel: Lustre: Lustre Version: 2.0.60 May 17 16:18:23 client-5 kernel: Lustre: Build Version: jenkins-g985ac5b-PRISTINE-2.6.18-194.17.1.el5 May 17 16:18:23 client-5 kernel: Lustre: Lustre LU module (ffffffff888eb1c0). May 17 16:18:23 client-5 kernel: Lustre: Register global MR array, MR size: 0xffffffffffffffff, array size: 1 May 17 16:18:23 client-5 kernel: Lustre: Added LNI 192.168.4.5@o2ib [8/64/0/180] May 17 16:18:23 client-5 kernel: Lustre: Added LNI 10.10.4.5@tcp [8/256/0/180] May 17 16:18:23 client-5 kernel: Lustre: Accept all, port 7988 May 17 16:18:23 client-5 kernel: Lustre: Lustre OSC module (ffffffff88b3c6c0). May 17 16:18:23 client-5 kernel: Lustre: Lustre LOV module (ffffffff88bab540). May 17 16:18:23 client-5 kernel: Lustre: Lustre client module (ffffffff88c643a0). May 17 16:18:23 client-5 xinetd[3391]: START: shell pid=7741 from=192.168.4.5 May 17 16:18:24 client-5 rshd[7743]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == conf-sanity test 34b: force umount with failed mds should be normal =============================== 16:18:23 \(1305674303\)");echo XXRETCODE:$?' May 17 16:18:24 client-5 kernel: Lustre: DEBUG MARKER: == conf-sanity test 34b: force umount with failed mds should be normal =============================== 16:18:23 (1305674303) May 17 16:18:24 client-5 xinetd[3391]: EXIT: shell status=0 pid=7741 duration=1(sec) May 17 16:18:25 client-5 kernel: Lustre: 7874:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 16:18:25 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 16:18:25 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 16:18:25 client-5 kernel: Lustre: 7874:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff810303621400->192.168.4.128@o2ib netid 50000: select flavor null May 17 16:18:25 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 16:18:30 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 16:18:35 client-5 kernel: Lustre: 7874:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-OST0000-osc-ffff810303621400->192.168.4.129@o2ib netid 50000: select flavor null May 17 16:18:35 client-5 kernel: Lustre: Client lustre-client has started May 17 16:18:35 client-5 kernel: Lustre: 7891:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:18:48 client-5 kernel: Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request May 17 16:18:48 client-5 kernel: Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request May 17 16:18:48 client-5 kernel: LustreError: 7937:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:18:48 client-5 kernel: LustreError: 7937:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:18:52 client-5 kernel: Lustre: 7447:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369098737942558 sent from MGC192.168.4.128@o2ib to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305674325] [real_sent 1305674325] [current 1305674332] [deadline 7s] [delay 0s] req@ffff8102f4218c00 x1369098737942558/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 192/192 e 0 to 1 dl 1305674332 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 16:18:54 client-5 kernel: Lustre: 7937:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369098737942562 sent from MGC192.168.4.128@o2ib to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305674328] [real_sent 1305674328] [current 1305674334] [deadline 6s] [delay 0s] req@ffff81031284b000 x1369098737942562/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 192/192 e 0 to 1 dl 1305674334 ref 2 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 16:18:54 client-5 kernel: Lustre: client ffff810303621400 umount complete May 17 16:19:05 client-5 kernel: Lustre: Acceptor stopping May 17 16:19:06 client-5 kernel: Lustre: Removed LNI 10.10.4.5@tcp May 17 16:19:08 client-5 kernel: Lustre: Removed LNI 192.168.4.5@o2ib May 17 16:19:09 client-5 xinetd[3391]: START: shell pid=8387 from=192.168.4.5 May 17 16:19:09 client-5 rshd[8388]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 16:19:10 client-5 xinetd[3391]: EXIT: shell status=0 pid=8387 duration=1(sec) May 17 16:19:10 client-5 kernel: Lustre: 8524:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:19:10 client-5 kernel: Lustre: 8527:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:19:10 client-5 kernel: Lustre: OBD class driver, http://www.lustre.org/ May 17 16:19:10 client-5 kernel: Lustre: Lustre Version: 2.0.60 May 17 16:19:10 client-5 kernel: Lustre: Build Version: jenkins-g985ac5b-PRISTINE-2.6.18-194.17.1.el5 May 17 16:19:10 client-5 kernel: Lustre: Lustre LU module (ffffffff888eb1c0). May 17 16:19:10 client-5 kernel: Lustre: Register global MR array, MR size: 0xffffffffffffffff, array size: 1 May 17 16:19:10 client-5 kernel: Lustre: Added LNI 192.168.4.5@o2ib [8/64/0/180] May 17 16:19:10 client-5 kernel: Lustre: Added LNI 10.10.4.5@tcp [8/256/0/180] May 17 16:19:10 client-5 kernel: Lustre: Accept all, port 7988 May 17 16:19:10 client-5 kernel: Lustre: Lustre OSC module (ffffffff88b3c6c0). May 17 16:19:10 client-5 kernel: Lustre: Lustre LOV module (ffffffff88bab540). May 17 16:19:10 client-5 kernel: Lustre: Lustre client module (ffffffff88c643a0). May 17 16:19:10 client-5 xinetd[3391]: START: shell pid=8922 from=192.168.4.5 May 17 16:19:10 client-5 rshd[8923]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == conf-sanity test 34c: force umount with failed ost should be normal =============================== 16:19:10 \(1305674350\)");echo XXRETCODE:$?' May 17 16:19:10 client-5 kernel: Lustre: DEBUG MARKER: == conf-sanity test 34c: force umount with failed ost should be normal =============================== 16:19:10 (1305674350) May 17 16:19:10 client-5 xinetd[3391]: EXIT: shell status=0 pid=8922 duration=0(sec) May 17 16:19:12 client-5 kernel: Lustre: 9058:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 16:19:12 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 16:19:12 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 16:19:12 client-5 kernel: Lustre: 9058:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff810320f92800->192.168.4.128@o2ib netid 50000: select flavor null May 17 16:19:12 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 16:19:17 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 16:19:22 client-5 kernel: Lustre: 9058:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-OST0000-osc-ffff810320f92800->192.168.4.129@o2ib netid 50000: select flavor null May 17 16:19:22 client-5 kernel: Lustre: Client lustre-client has started May 17 16:19:22 client-5 kernel: Lustre: 9075:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:19:24 client-5 kernel: Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request May 17 16:19:24 client-5 kernel: Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request May 17 16:19:24 client-5 kernel: LustreError: 9117:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:19:24 client-5 kernel: LustreError: 9117:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:19:24 client-5 kernel: Lustre: client ffff810320f92800 umount complete May 17 16:19:44 client-5 kernel: Lustre: Acceptor stopping May 17 16:19:45 client-5 kernel: Lustre: Removed LNI 10.10.4.5@tcp May 17 16:19:47 client-5 kernel: Lustre: Removed LNI 192.168.4.5@o2ib May 17 16:19:48 client-5 xinetd[3391]: START: shell pid=9566 from=192.168.4.5 May 17 16:19:48 client-5 rshd[9567]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 16:19:48 client-5 xinetd[3391]: EXIT: shell status=0 pid=9566 duration=0(sec) May 17 16:19:48 client-5 kernel: Lustre: 9705:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:19:48 client-5 kernel: Lustre: 9706:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:19:49 client-5 kernel: Lustre: OBD class driver, http://www.lustre.org/ May 17 16:19:49 client-5 kernel: Lustre: Lustre Version: 2.0.60 May 17 16:19:49 client-5 kernel: Lustre: Build Version: jenkins-g985ac5b-PRISTINE-2.6.18-194.17.1.el5 May 17 16:19:49 client-5 kernel: Lustre: Lustre LU module (ffffffff888eb1c0). May 17 16:19:49 client-5 kernel: Lustre: Register global MR array, MR size: 0xffffffffffffffff, array size: 1 May 17 16:19:49 client-5 kernel: Lustre: Added LNI 192.168.4.5@o2ib [8/64/0/180] May 17 16:19:49 client-5 kernel: Lustre: Added LNI 10.10.4.5@tcp [8/256/0/180] May 17 16:19:49 client-5 kernel: Lustre: Accept all, port 7988 May 17 16:19:49 client-5 kernel: Lustre: Lustre OSC module (ffffffff88b3c6c0). May 17 16:19:49 client-5 kernel: Lustre: Lustre LOV module (ffffffff88bab540). May 17 16:19:49 client-5 kernel: Lustre: Lustre client module (ffffffff88c643a0). May 17 16:19:49 client-5 xinetd[3391]: START: shell pid=10097 from=192.168.4.5 May 17 16:19:49 client-5 rshd[10099]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == conf-sanity test 35a: Reconnect to the last active server first =================================== 16:19:48 \(1305674388\)");echo XXRETCODE:$?' May 17 16:19:49 client-5 kernel: Lustre: DEBUG MARKER: == conf-sanity test 35a: Reconnect to the last active server first =================================== 16:19:48 (1305674388) May 17 16:19:49 client-5 xinetd[3391]: EXIT: shell status=0 pid=10097 duration=0(sec) May 17 16:19:51 client-5 kernel: Lustre: 10237:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 16:19:51 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 16:19:51 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 16:19:51 client-5 kernel: Lustre: 10237:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff810314828800->192.168.4.128@o2ib netid 50000: select flavor null May 17 16:19:51 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 16:19:56 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 16:20:01 client-5 kernel: Lustre: 10237:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-OST0000-osc-ffff810314828800->192.168.4.129@o2ib netid 50000: select flavor null May 17 16:20:01 client-5 kernel: Lustre: Client lustre-client has started May 17 16:20:01 client-5 kernel: Lustre: 10258:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:20:01 client-5 xinetd[3391]: START: shell pid=10321 from=192.168.4.5 May 17 16:20:01 client-5 rshd[10322]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark Set up a fake failnode for the MDS");echo XXRETCODE:$?' May 17 16:20:01 client-5 kernel: Lustre: DEBUG MARKER: Set up a fake failnode for the MDS May 17 16:20:01 client-5 xinetd[3391]: EXIT: shell status=0 pid=10321 duration=0(sec) May 17 16:20:01 client-5 xinetd[3391]: START: shell pid=10436 from=192.168.4.5 May 17 16:20:01 client-5 rshd[10437]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark Wait for RECONNECT_INTERVAL seconds \(10s\)");echo XXRETCODE:$?' May 17 16:20:01 client-5 kernel: Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) May 17 16:20:01 client-5 xinetd[3391]: EXIT: shell status=0 pid=10436 duration=0(sec) May 17 16:20:11 client-5 xinetd[3391]: START: shell pid=10534 from=192.168.4.5 May 17 16:20:11 client-5 rshd[10535]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark conf-sanity.sh test_35a 2011-05-1716h20m11s");echo XXRETCODE:$?' May 17 16:20:11 client-5 kernel: Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2011-05-1716h20m11s May 17 16:20:11 client-5 xinetd[3391]: EXIT: shell status=0 pid=10534 duration=0(sec) May 17 16:20:11 client-5 xinetd[3391]: START: shell pid=10627 from=192.168.4.5 May 17 16:20:11 client-5 rshd[10628]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark Stopping the MDT:");echo XXRETCODE:$?' May 17 16:20:12 client-5 kernel: Lustre: DEBUG MARKER: Stopping the MDT: May 17 16:20:12 client-5 xinetd[3391]: EXIT: shell status=0 pid=10627 duration=1(sec) May 17 16:20:18 client-5 xinetd[3391]: START: shell pid=10754 from=192.168.4.5 May 17 16:20:18 client-5 rshd[10755]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark Restarting the MDT:");echo XXRETCODE:$?' May 17 16:20:18 client-5 kernel: Lustre: DEBUG MARKER: Restarting the MDT: May 17 16:20:18 client-5 xinetd[3391]: EXIT: shell status=0 pid=10754 duration=0(sec) May 17 16:20:19 client-5 xinetd[3391]: START: shell pid=10890 from=192.168.4.5 May 17 16:20:19 client-5 rshd[10891]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark Wait for df \(10695\) ... ");echo XXRETCODE:$?' May 17 16:20:19 client-5 kernel: Lustre: DEBUG MARKER: Wait for df (10695) ... May 17 16:20:19 client-5 xinetd[3391]: EXIT: shell status=0 pid=10890 duration=0(sec) May 17 16:20:21 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The obd_ping operation failed with -107 May 17 16:20:21 client-5 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail. May 17 16:20:21 client-5 kernel: Lustre: 9809:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x62e5a8d74f2faac to 0x62e5a8d74f2fb9a May 17 16:20:21 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 16:20:21 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Connection restored to service MGS using nid 192.168.4.128@o2ib. May 17 16:20:23 client-5 kernel: Lustre: 9808:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369098828120099 sent from lustre-MDT0000-mdc-ffff810314828800 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305674416] [real_sent 1305674416] [current 1305674423] [deadline 7s] [delay 0s] req@ffff8102f07f9400 x1369098828120099/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 192/192 e 0 to 1 dl 1305674423 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 16:20:23 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff810314828800: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 16:20:23 client-5 kernel: LustreError: 167-0: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. May 17 16:20:23 client-5 kernel: LustreError: 10695:0:(mdc_locks.c:722:mdc_enqueue()) ldlm_cli_enqueue: -4 May 17 16:20:23 client-5 kernel: LustreError: 10695:0:(file.c:2161:ll_inode_revalidate_fini()) failure -4 inode 19 May 17 16:20:23 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff810314828800: Connection restored to service lustre-MDT0000 using nid 192.168.4.128@o2ib. May 17 16:20:23 client-5 xinetd[3391]: START: shell pid=10985 from=192.168.4.5 May 17 16:20:23 client-5 rshd[10986]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark done");echo XXRETCODE:$?' May 17 16:20:23 client-5 kernel: Lustre: DEBUG MARKER: done May 17 16:20:23 client-5 xinetd[3391]: EXIT: shell status=0 pid=10985 duration=0(sec) May 17 16:20:23 client-5 kernel: LustreError: 11041:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:20:23 client-5 kernel: LustreError: 11041:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:20:23 client-5 kernel: Lustre: client ffff810314828800 umount complete May 17 16:20:39 client-5 kernel: Lustre: Acceptor stopping May 17 16:20:40 client-5 kernel: Lustre: Removed LNI 10.10.4.5@tcp May 17 16:20:42 client-5 kernel: Lustre: Removed LNI 192.168.4.5@o2ib May 17 16:20:43 client-5 kernel: Lustre: 11456:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:20:43 client-5 kernel: Lustre: 11457:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:20:43 client-5 kernel: Lustre: OBD class driver, http://www.lustre.org/ May 17 16:20:43 client-5 kernel: Lustre: Lustre Version: 2.0.60 May 17 16:20:43 client-5 kernel: Lustre: Build Version: jenkins-g985ac5b-PRISTINE-2.6.18-194.17.1.el5 May 17 16:20:43 client-5 kernel: Lustre: Lustre LU module (ffffffff888eb1c0). May 17 16:20:43 client-5 kernel: Lustre: Register global MR array, MR size: 0xffffffffffffffff, array size: 1 May 17 16:20:43 client-5 kernel: Lustre: Added LNI 192.168.4.5@o2ib [8/64/0/180] May 17 16:20:43 client-5 kernel: Lustre: Added LNI 10.10.4.5@tcp [8/256/0/180] May 17 16:20:43 client-5 kernel: Lustre: Accept all, port 7988 May 17 16:20:43 client-5 kernel: Lustre: Lustre OSC module (ffffffff88b3c6c0). May 17 16:20:43 client-5 kernel: Lustre: Lustre LOV module (ffffffff88bab540). May 17 16:20:43 client-5 kernel: Lustre: Lustre client module (ffffffff88c643a0). May 17 16:20:45 client-5 xinetd[3391]: START: shell pid=11962 from=192.168.4.5 May 17 16:20:45 client-5 rshd[11963]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 16:20:45 client-5 xinetd[3391]: EXIT: shell status=0 pid=11962 duration=0(sec) May 17 16:20:45 client-5 xinetd[3391]: START: shell pid=12134 from=192.168.4.5 May 17 16:20:45 client-5 rshd[12135]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ================= 16:20:45 \(1305674445\)");echo XXRETCODE:$?' May 17 16:20:45 client-5 kernel: Lustre: DEBUG MARKER: == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ================= 16:20:45 (1305674445) May 17 16:20:45 client-5 xinetd[3391]: EXIT: shell status=0 pid=12134 duration=0(sec) May 17 16:20:46 client-5 kernel: Lustre: 12269:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 16:20:46 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 16:20:46 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 16:20:46 client-5 kernel: Lustre: 12269:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff810318d50000->192.168.4.128@o2ib netid 50000: select flavor null May 17 16:20:46 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 16:20:51 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 16:20:56 client-5 kernel: Lustre: 12269:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-OST0000-osc-ffff810318d50000->192.168.4.129@o2ib netid 50000: select flavor null May 17 16:20:56 client-5 kernel: Lustre: Client lustre-client has started May 17 16:20:56 client-5 kernel: Lustre: 12286:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:20:57 client-5 xinetd[3391]: START: shell pid=12351 from=192.168.4.5 May 17 16:20:57 client-5 rshd[12352]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark conf-sanity.sh test_35b 2011-05-1716h20m56s");echo XXRETCODE:$?' May 17 16:20:57 client-5 kernel: Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2011-05-1716h20m56s May 17 16:20:57 client-5 xinetd[3391]: EXIT: shell status=0 pid=12351 duration=0(sec) May 17 16:20:57 client-5 xinetd[3391]: START: shell pid=12444 from=192.168.4.5 May 17 16:20:57 client-5 rshd[12445]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark Set up a fake failnode for the MDS");echo XXRETCODE:$?' May 17 16:20:57 client-5 kernel: Lustre: DEBUG MARKER: Set up a fake failnode for the MDS May 17 16:20:57 client-5 xinetd[3391]: EXIT: shell status=0 pid=12444 duration=0(sec) May 17 16:20:57 client-5 xinetd[3391]: START: shell pid=12594 from=192.168.4.5 May 17 16:20:57 client-5 rshd[12595]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark Injecting EBUSY on MDS");echo XXRETCODE:$?' May 17 16:20:57 client-5 kernel: Lustre: DEBUG MARKER: Injecting EBUSY on MDS May 17 16:20:57 client-5 xinetd[3391]: EXIT: shell status=0 pid=12594 duration=0(sec) May 17 16:20:58 client-5 xinetd[3391]: START: shell pid=12693 from=192.168.4.5 May 17 16:20:58 client-5 rshd[12696]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark Stat on a test file");echo XXRETCODE:$?' May 17 16:20:58 client-5 kernel: Lustre: DEBUG MARKER: Stat on a test file May 17 16:20:58 client-5 xinetd[3391]: EXIT: shell status=0 pid=12693 duration=0(sec) May 17 16:21:07 client-5 xinetd[3391]: START: login pid=12737 from=10.10.0.1 May 17 16:21:18 client-5 kernel: Lustre: 12730:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369098884743200 sent from lustre-MDT0000-mdc-ffff810318d50000 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305674458] [real_sent 1305674458] [current 1305674478] [deadline 20s] [delay 0s] req@ffff8103201c3400 x1369098884743200/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 544/992 e 0 to 1 dl 1305674478 ref 2 fl Rpc:X/ffffffff/ffffffff rc 0/-1 May 17 16:21:18 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff810318d50000: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 16:21:18 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -16 May 17 16:21:19 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -16 May 17 16:21:19 client-5 kernel: LustreError: Skipped 4061 previous similar messages May 17 16:21:21 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -16 May 17 16:21:21 client-5 kernel: LustreError: Skipped 8649 previous similar messages May 17 16:21:25 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -16 May 17 16:21:25 client-5 kernel: LustreError: Skipped 16346 previous similar messages May 17 16:21:33 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -16 May 17 16:21:33 client-5 kernel: LustreError: Skipped 33831 previous similar messages May 17 16:21:38 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff810318d50000: Connection restored to service lustre-MDT0000 using nid 192.168.4.128@o2ib. May 17 16:21:38 client-5 xinetd[3391]: START: shell pid=12842 from=192.168.4.5 May 17 16:21:38 client-5 rshd[12843]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark Stop injecting EBUSY on MDS");echo XXRETCODE:$?' May 17 16:21:38 client-5 kernel: Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS May 17 16:21:38 client-5 xinetd[3391]: EXIT: shell status=0 pid=12842 duration=0(sec) May 17 16:21:38 client-5 xinetd[3391]: START: shell pid=12944 from=192.168.4.5 May 17 16:21:38 client-5 rshd[12945]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark done");echo XXRETCODE:$?' May 17 16:21:38 client-5 kernel: Lustre: DEBUG MARKER: done May 17 16:21:38 client-5 xinetd[3391]: EXIT: shell status=0 pid=12944 duration=0(sec) May 17 16:21:39 client-5 kernel: LustreError: 13014:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:21:39 client-5 kernel: LustreError: 13014:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:21:39 client-5 kernel: LustreError: 13014:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:21:39 client-5 kernel: LustreError: 13014:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:21:39 client-5 kernel: Lustre: client ffff810318d50000 umount complete May 17 16:21:58 client-5 kernel: Lustre: Acceptor stopping May 17 16:21:59 client-5 kernel: Lustre: Removed LNI 10.10.4.5@tcp May 17 16:22:01 client-5 kernel: Lustre: Removed LNI 192.168.4.5@o2ib May 17 16:22:02 client-5 kernel: Lustre: 13428:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:22:02 client-5 kernel: Lustre: 13429:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:22:02 client-5 kernel: Lustre: OBD class driver, http://www.lustre.org/ May 17 16:22:02 client-5 kernel: Lustre: Lustre Version: 2.0.60 May 17 16:22:02 client-5 kernel: Lustre: Build Version: jenkins-g985ac5b-PRISTINE-2.6.18-194.17.1.el5 May 17 16:22:02 client-5 kernel: Lustre: Lustre LU module (ffffffff888eb1c0). May 17 16:22:02 client-5 kernel: Lustre: Register global MR array, MR size: 0xffffffffffffffff, array size: 1 May 17 16:22:02 client-5 kernel: Lustre: Added LNI 192.168.4.5@o2ib [8/64/0/180] May 17 16:22:02 client-5 kernel: Lustre: Added LNI 10.10.4.5@tcp [8/256/0/180] May 17 16:22:02 client-5 kernel: Lustre: Accept all, port 7988 May 17 16:22:02 client-5 kernel: Lustre: Lustre OSC module (ffffffff88b3c6c0). May 17 16:22:02 client-5 kernel: Lustre: Lustre LOV module (ffffffff88bab540). May 17 16:22:02 client-5 kernel: Lustre: Lustre client module (ffffffff88c643a0). May 17 16:22:04 client-5 xinetd[3391]: START: shell pid=13934 from=192.168.4.5 May 17 16:22:04 client-5 rshd[13935]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 16:22:04 client-5 xinetd[3391]: EXIT: shell status=0 pid=13934 duration=0(sec) May 17 16:22:04 client-5 xinetd[3391]: START: shell pid=14106 from=192.168.4.5 May 17 16:22:04 client-5 rshd[14107]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == conf-sanity test 36: df report consistency on OSTs with different block size ====================== 16:22:04 \(1305674524\)");echo XXRETCODE:$?' May 17 16:22:04 client-5 kernel: Lustre: DEBUG MARKER: == conf-sanity test 36: df report consistency on OSTs with different block size ====================== 16:22:04 (1305674524) May 17 16:22:04 client-5 xinetd[3391]: EXIT: shell status=0 pid=14106 duration=0(sec) May 17 16:22:04 client-5 xinetd[3391]: START: shell pid=14201 from=192.168.4.5 May 17 16:22:04 client-5 rshd[14202]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark SKIP: conf-sanity test_36 remote OST");echo XXRETCODE:$?' May 17 16:22:04 client-5 kernel: Lustre: DEBUG MARKER: SKIP: conf-sanity test_36 remote OST May 17 16:22:04 client-5 xinetd[3391]: EXIT: shell status=0 pid=14201 duration=0(sec) May 17 16:22:04 client-5 xinetd[3391]: START: shell pid=14291 from=192.168.4.5 May 17 16:22:04 client-5 rshd[14293]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 16:22:04 client-5 xinetd[3391]: EXIT: shell status=0 pid=14291 duration=0(sec) May 17 16:22:04 client-5 xinetd[3391]: START: shell pid=14463 from=192.168.4.5 May 17 16:22:04 client-5 rshd[14465]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == conf-sanity test 37: verify set tunables works for symlink device ================================= 16:22:04 \(1305674524\)");echo XXRETCODE:$?' May 17 16:22:05 client-5 kernel: Lustre: DEBUG MARKER: == conf-sanity test 37: verify set tunables works for symlink device ================================= 16:22:04 (1305674524) May 17 16:22:05 client-5 xinetd[3391]: EXIT: shell status=0 pid=14463 duration=1(sec) May 17 16:22:12 client-5 xinetd[3391]: START: shell pid=14592 from=192.168.4.5 May 17 16:22:12 client-5 rshd[14594]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 16:22:12 client-5 xinetd[3391]: EXIT: shell status=0 pid=14592 duration=0(sec) May 17 16:22:12 client-5 xinetd[3391]: START: shell pid=14764 from=192.168.4.5 May 17 16:22:12 client-5 rshd[14766]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == conf-sanity test 38: MDS recreates missing lov_objid file from OST data =========================== 16:22:12 \(1305674532\)");echo XXRETCODE:$?' May 17 16:22:12 client-5 kernel: Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data =========================== 16:22:12 (1305674532) May 17 16:22:12 client-5 xinetd[3391]: EXIT: shell status=0 pid=14764 duration=0(sec) May 17 16:22:14 client-5 kernel: Lustre: 14892:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 16:22:14 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 16:22:14 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 16:22:14 client-5 kernel: Lustre: 14892:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff81030bdf7800->192.168.4.128@o2ib netid 50000: select flavor null May 17 16:22:14 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 16:22:19 client-5 kernel: Lustre: 14892:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-OST0000-osc-ffff81030bdf7800->192.168.4.129@o2ib netid 50000: select flavor null May 17 16:22:19 client-5 kernel: Lustre: Client lustre-client has started May 17 16:22:19 client-5 kernel: Lustre: 14909:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:22:19 client-5 xinetd[3391]: START: shell pid=14976 from=192.168.4.5 May 17 16:22:19 client-5 rshd[14977]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark copying 10 files to \/mnt\/lustre\/d0.conf-sanity\/d38");echo XXRETCODE:$?' May 17 16:22:19 client-5 kernel: Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d0.conf-sanity/d38 May 17 16:22:19 client-5 xinetd[3391]: EXIT: shell status=0 pid=14976 duration=0(sec) May 17 16:22:20 client-5 kernel: LustreError: 15033:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:22:20 client-5 kernel: LustreError: 15033:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:22:20 client-5 kernel: LustreError: 15033:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:22:20 client-5 kernel: LustreError: 15033:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:22:20 client-5 kernel: Lustre: client ffff81030bdf7800 umount complete May 17 16:22:27 client-5 xinetd[3391]: START: shell pid=15129 from=192.168.4.5 May 17 16:22:27 client-5 rshd[15130]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark rename lov_objid file on MDS");echo XXRETCODE:$?' May 17 16:22:27 client-5 kernel: Lustre: DEBUG MARKER: rename lov_objid file on MDS May 17 16:22:27 client-5 xinetd[3391]: EXIT: shell status=0 pid=15129 duration=0(sec) May 17 16:22:28 client-5 kernel: Lustre: 15242:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 16:22:28 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 16:22:28 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 16:22:28 client-5 kernel: Lustre: Client lustre-client has started May 17 16:22:28 client-5 kernel: Lustre: 15259:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:22:28 client-5 kernel: Lustre: 15259:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message May 17 16:22:28 client-5 kernel: LustreError: 15306:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:22:28 client-5 kernel: LustreError: 15306:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Skipped 3 previous similar messages May 17 16:22:28 client-5 kernel: LustreError: 15306:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:22:28 client-5 kernel: LustreError: 15306:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) Skipped 3 previous similar messages May 17 16:22:28 client-5 kernel: Lustre: client ffff81030bdf7000 umount complete May 17 16:22:36 client-5 kernel: Lustre: 15416:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 16:22:36 client-5 kernel: Lustre: 15416:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 2 previous similar messages May 17 16:22:36 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 16:22:36 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 16:22:36 client-5 kernel: Lustre: Client lustre-client has started May 17 16:22:36 client-5 kernel: Lustre: 15433:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:22:36 client-5 kernel: Lustre: 15433:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message May 17 16:22:37 client-5 kernel: LustreError: 15480:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:22:37 client-5 kernel: LustreError: 15480:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Skipped 15 previous similar messages May 17 16:22:37 client-5 kernel: LustreError: 15480:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:22:37 client-5 kernel: LustreError: 15480:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) Skipped 15 previous similar messages May 17 16:22:37 client-5 kernel: Lustre: client ffff8103266da400 umount complete May 17 16:22:44 client-5 xinetd[3391]: START: shell pid=15572 from=192.168.4.5 May 17 16:22:44 client-5 rshd[15573]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark files compared the same");echo XXRETCODE:$?' May 17 16:22:44 client-5 kernel: Lustre: DEBUG MARKER: files compared the same May 17 16:22:44 client-5 xinetd[3391]: EXIT: shell status=0 pid=15572 duration=0(sec) May 17 16:22:53 client-5 kernel: Lustre: Acceptor stopping May 17 16:22:54 client-5 kernel: Lustre: Removed LNI 10.10.4.5@tcp May 17 16:22:56 client-5 kernel: Lustre: Removed LNI 192.168.4.5@o2ib May 17 16:22:58 client-5 xinetd[3391]: START: shell pid=16055 from=192.168.4.5 May 17 16:22:58 client-5 rshd[16057]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 16:22:58 client-5 xinetd[3391]: EXIT: shell status=0 pid=16055 duration=0(sec) May 17 16:22:58 client-5 kernel: Lustre: 16191:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:22:58 client-5 kernel: Lustre: 16193:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:22:58 client-5 kernel: Lustre: OBD class driver, http://www.lustre.org/ May 17 16:22:58 client-5 kernel: Lustre: Lustre Version: 2.0.60 May 17 16:22:58 client-5 kernel: Lustre: Build Version: jenkins-g985ac5b-PRISTINE-2.6.18-194.17.1.el5 May 17 16:22:58 client-5 kernel: Lustre: Lustre LU module (ffffffff888eb1c0). May 17 16:22:58 client-5 kernel: Lustre: Register global MR array, MR size: 0xffffffffffffffff, array size: 1 May 17 16:22:58 client-5 kernel: Lustre: Added LNI 192.168.4.5@o2ib [8/64/0/180] May 17 16:22:58 client-5 kernel: Lustre: Added LNI 10.10.4.5@tcp [8/256/0/180] May 17 16:22:58 client-5 kernel: Lustre: Accept all, port 7988 May 17 16:22:58 client-5 kernel: Lustre: Lustre OSC module (ffffffff88b3c6c0). May 17 16:22:58 client-5 kernel: Lustre: Lustre LOV module (ffffffff88bab540). May 17 16:22:58 client-5 kernel: Lustre: Lustre client module (ffffffff88c643a0). May 17 16:22:58 client-5 xinetd[3391]: START: shell pid=16583 from=192.168.4.5 May 17 16:22:58 client-5 rshd[16585]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == conf-sanity test 39: leak_finder recognizes both LUSTRE and LNET malloc messages ================== 16:22:58 \(1305674578\)");echo XXRETCODE:$?' May 17 16:22:59 client-5 kernel: Lustre: DEBUG MARKER: == conf-sanity test 39: leak_finder recognizes both LUSTRE and LNET malloc messages ================== 16:22:58 (1305674578) May 17 16:22:59 client-5 xinetd[3391]: EXIT: shell status=0 pid=16583 duration=1(sec) May 17 16:23:00 client-5 kernel: Lustre: 16726:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 16:23:00 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 16:23:00 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 16:23:00 client-5 kernel: Lustre: 16726:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff810320559800->192.168.4.128@o2ib netid 50000: select flavor null May 17 16:23:00 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 16:23:05 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 16:23:10 client-5 kernel: Lustre: 16726:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-OST0000-osc-ffff810320559800->192.168.4.129@o2ib netid 50000: select flavor null May 17 16:23:10 client-5 kernel: Lustre: Client lustre-client has started May 17 16:23:10 client-5 kernel: Lustre: 16744:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:23:10 client-5 kernel: LustreError: 16764:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:23:10 client-5 kernel: LustreError: 16764:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:23:10 client-5 kernel: LustreError: 16764:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:23:10 client-5 kernel: LustreError: 16764:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:23:10 client-5 kernel: Lustre: client ffff810320559800 umount complete May 17 16:23:29 client-5 kernel: Lustre: Acceptor stopping May 17 16:23:30 client-5 kernel: Lustre: Removed LNI 10.10.4.5@tcp May 17 16:23:32 client-5 kernel: Lustre: Removed LNI 192.168.4.5@o2ib May 17 16:23:34 client-5 xinetd[3391]: START: shell pid=17220 from=192.168.4.5 May 17 16:23:34 client-5 rshd[17221]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 16:23:34 client-5 xinetd[3391]: EXIT: shell status=0 pid=17220 duration=0(sec) May 17 16:23:34 client-5 kernel: Lustre: 17355:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:23:34 client-5 kernel: Lustre: 17357:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:23:34 client-5 kernel: Lustre: OBD class driver, http://www.lustre.org/ May 17 16:23:34 client-5 kernel: Lustre: Lustre Version: 2.0.60 May 17 16:23:34 client-5 kernel: Lustre: Build Version: jenkins-g985ac5b-PRISTINE-2.6.18-194.17.1.el5 May 17 16:23:34 client-5 kernel: Lustre: Lustre LU module (ffffffff888eb1c0). May 17 16:23:34 client-5 kernel: Lustre: Register global MR array, MR size: 0xffffffffffffffff, array size: 1 May 17 16:23:34 client-5 kernel: Lustre: Added LNI 192.168.4.5@o2ib [8/64/0/180] May 17 16:23:34 client-5 kernel: Lustre: Added LNI 10.10.4.5@tcp [8/256/0/180] May 17 16:23:34 client-5 kernel: Lustre: Accept all, port 7988 May 17 16:23:34 client-5 kernel: Lustre: Lustre OSC module (ffffffff88b3c6c0). May 17 16:23:34 client-5 kernel: Lustre: Lustre LOV module (ffffffff88bab540). May 17 16:23:34 client-5 kernel: Lustre: Lustre client module (ffffffff88c643a0). May 17 16:23:34 client-5 xinetd[3391]: START: shell pid=17753 from=192.168.4.5 May 17 16:23:34 client-5 rshd[17756]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == conf-sanity test 40: race during service thread startup =========================================== 16:23:34 \(1305674614\)");echo XXRETCODE:$?' May 17 16:23:35 client-5 kernel: Lustre: DEBUG MARKER: == conf-sanity test 40: race during service thread startup =========================================== 16:23:34 (1305674614) May 17 16:23:35 client-5 xinetd[3391]: EXIT: shell status=0 pid=17753 duration=1(sec) May 17 16:24:21 client-5 kernel: Lustre: Acceptor stopping May 17 16:24:22 client-5 kernel: Lustre: Removed LNI 192.168.4.5@o2ib May 17 16:24:23 client-5 kernel: Lustre: Removed LNI 10.10.4.5@tcp May 17 16:24:24 client-5 xinetd[3391]: START: shell pid=18349 from=192.168.4.5 May 17 16:24:24 client-5 rshd[18350]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 16:24:24 client-5 xinetd[3391]: EXIT: shell status=0 pid=18349 duration=0(sec) May 17 16:24:25 client-5 kernel: Lustre: 18486:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:24:25 client-5 kernel: Lustre: 18487:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:24:25 client-5 kernel: Lustre: OBD class driver, http://www.lustre.org/ May 17 16:24:25 client-5 kernel: Lustre: Lustre Version: 2.0.60 May 17 16:24:25 client-5 kernel: Lustre: Build Version: jenkins-g985ac5b-PRISTINE-2.6.18-194.17.1.el5 May 17 16:24:25 client-5 kernel: Lustre: Lustre LU module (ffffffff888eb1c0). May 17 16:24:25 client-5 kernel: Lustre: Register global MR array, MR size: 0xffffffffffffffff, array size: 1 May 17 16:24:25 client-5 kernel: Lustre: Added LNI 192.168.4.5@o2ib [8/64/0/180] May 17 16:24:25 client-5 kernel: Lustre: Added LNI 10.10.4.5@tcp [8/256/0/180] May 17 16:24:25 client-5 kernel: Lustre: Accept all, port 7988 May 17 16:24:25 client-5 kernel: Lustre: Lustre OSC module (ffffffff88b3c6c0). May 17 16:24:25 client-5 kernel: Lustre: Lustre LOV module (ffffffff88bab540). May 17 16:24:25 client-5 kernel: Lustre: Lustre client module (ffffffff88c643a0). May 17 16:24:25 client-5 xinetd[3391]: START: shell pid=18887 from=192.168.4.5 May 17 16:24:25 client-5 rshd[18888]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == conf-sanity test 41a: mount mds with --nosvc and --nomgs ========================================== 16:24:24 \(1305674664\)");echo XXRETCODE:$?' May 17 16:24:25 client-5 kernel: Lustre: DEBUG MARKER: == conf-sanity test 41a: mount mds with --nosvc and --nomgs ========================================== 16:24:24 (1305674664) May 17 16:24:25 client-5 xinetd[3391]: EXIT: shell status=0 pid=18887 duration=0(sec) May 17 16:24:27 client-5 kernel: Lustre: 19050:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 16:24:27 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 16:24:27 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 16:24:27 client-5 kernel: Lustre: 19050:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff81033f76a000->192.168.4.128@o2ib netid 50000: select flavor null May 17 16:24:27 client-5 kernel: Lustre: Client lustre-client has started May 17 16:24:27 client-5 kernel: Lustre: 19067:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:24:32 client-5 kernel: LustreError: 19090:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:24:32 client-5 kernel: LustreError: 19090:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:24:32 client-5 kernel: LustreError: 19090:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:24:32 client-5 kernel: LustreError: 19090:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:24:33 client-5 kernel: Lustre: client ffff81033f76a000 umount complete May 17 16:24:53 client-5 kernel: Lustre: Acceptor stopping May 17 16:24:54 client-5 kernel: Lustre: Removed LNI 10.10.4.5@tcp May 17 16:24:56 client-5 kernel: Lustre: Removed LNI 192.168.4.5@o2ib May 17 16:24:57 client-5 xinetd[3391]: START: shell pid=19576 from=192.168.4.5 May 17 16:24:57 client-5 rshd[19577]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 16:24:58 client-5 xinetd[3391]: EXIT: shell status=0 pid=19576 duration=1(sec) May 17 16:24:58 client-5 kernel: Lustre: 19712:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:24:58 client-5 kernel: Lustre: 19713:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:24:58 client-5 kernel: Lustre: OBD class driver, http://www.lustre.org/ May 17 16:24:58 client-5 kernel: Lustre: Lustre Version: 2.0.60 May 17 16:24:58 client-5 kernel: Lustre: Build Version: jenkins-g985ac5b-PRISTINE-2.6.18-194.17.1.el5 May 17 16:24:58 client-5 kernel: Lustre: Lustre LU module (ffffffff888eb1c0). May 17 16:24:58 client-5 kernel: Lustre: Register global MR array, MR size: 0xffffffffffffffff, array size: 1 May 17 16:24:58 client-5 kernel: Lustre: Added LNI 192.168.4.5@o2ib [8/64/0/180] May 17 16:24:58 client-5 kernel: Lustre: Added LNI 10.10.4.5@tcp [8/256/0/180] May 17 16:24:58 client-5 kernel: Lustre: Accept all, port 7988 May 17 16:24:58 client-5 kernel: Lustre: Lustre OSC module (ffffffff88b3c6c0). May 17 16:24:58 client-5 kernel: Lustre: Lustre LOV module (ffffffff88bab540). May 17 16:24:58 client-5 kernel: Lustre: Lustre client module (ffffffff88c643a0). May 17 16:24:58 client-5 xinetd[3391]: START: shell pid=20109 from=192.168.4.5 May 17 16:24:58 client-5 rshd[20111]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount =========================== 16:24:58 \(1305674698\)");echo XXRETCODE:$?' May 17 16:24:58 client-5 kernel: Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount =========================== 16:24:58 (1305674698) May 17 16:24:58 client-5 xinetd[3391]: EXIT: shell status=0 pid=20109 duration=0(sec) May 17 16:24:58 client-5 xinetd[3391]: START: shell pid=20164 from=192.168.4.5 May 17 16:24:58 client-5 rshd[20165]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:; lsof -t /mnt/lustre || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre is still busy, wait one second && sleep 1; umount /mnt/lustre; fi fi");echo XXRETCODE:$?' May 17 16:24:58 client-5 xinetd[3391]: EXIT: shell status=0 pid=20164 duration=0(sec) May 17 16:24:58 client-5 xinetd[3391]: START: shell pid=20208 from=192.168.4.5 May 17 16:24:58 client-5 rshd[20209]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:; lsof -t /mnt/lustre2 || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre2 | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre2 is still busy, wait one second && sleep 1; umount /mnt/lustre2; fi fi");echo XXRETCODE:$?' May 17 16:24:59 client-5 xinetd[3391]: EXIT: shell status=0 pid=20208 duration=1(sec) May 17 16:25:00 client-5 xinetd[3391]: START: shell pid=20416 from=192.168.4.5 May 17 16:25:00 client-5 rshd[20417]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:; lsof -t /mnt/lustre || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre is still busy, wait one second && sleep 1; umount /mnt/lustre; fi fi");echo XXRETCODE:$?' May 17 16:25:00 client-5 xinetd[3391]: EXIT: shell status=0 pid=20416 duration=0(sec) May 17 16:25:00 client-5 xinetd[3391]: START: shell pid=20460 from=192.168.4.5 May 17 16:25:00 client-5 rshd[20461]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:; lsof -t /mnt/lustre2 || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre2 | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre2 is still busy, wait one second && sleep 1; umount /mnt/lustre2; fi fi");echo XXRETCODE:$?' May 17 16:25:00 client-5 xinetd[3391]: EXIT: shell status=0 pid=20460 duration=0(sec) May 17 16:25:02 client-5 kernel: Lustre: 20663:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:25:09 client-5 kernel: Lustre: 21112:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 16:25:09 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 16:25:09 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 16:25:09 client-5 kernel: Lustre: 21112:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff8102d407b400->192.168.4.128@o2ib netid 50000: select flavor null May 17 16:25:09 client-5 kernel: Lustre: Client lustre-client has started May 17 16:25:09 client-5 kernel: Lustre: 21129:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:25:09 client-5 kernel: Lustre: 21129:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message May 17 16:25:14 client-5 kernel: LustreError: 21152:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:25:14 client-5 kernel: LustreError: 21152:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:25:14 client-5 kernel: LustreError: 21152:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:25:14 client-5 kernel: LustreError: 21152:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:25:14 client-5 kernel: Lustre: client ffff8102d407b400 umount complete May 17 16:25:29 client-5 xinetd[3391]: START: shell pid=21308 from=192.168.4.5 May 17 16:25:29 client-5 rshd[21309]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 16:25:29 client-5 xinetd[3391]: EXIT: shell status=0 pid=21308 duration=0(sec) May 17 16:25:29 client-5 xinetd[3391]: START: shell pid=21480 from=192.168.4.5 May 17 16:25:29 client-5 rshd[21481]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == conf-sanity test 42: invalid config param should not prevent client from mounting ================= 16:25:29 \(1305674729\)");echo XXRETCODE:$?' May 17 16:25:29 client-5 kernel: Lustre: DEBUG MARKER: == conf-sanity test 42: invalid config param should not prevent client from mounting ================= 16:25:29 (1305674729) May 17 16:25:29 client-5 xinetd[3391]: EXIT: shell status=0 pid=21480 duration=0(sec) May 17 16:25:31 client-5 kernel: Lustre: 21607:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 16:25:31 client-5 kernel: Lustre: 21607:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 1 previous similar message May 17 16:25:31 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 16:25:31 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 16:25:31 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 16:25:36 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 16:25:41 client-5 kernel: Lustre: 21607:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-OST0000-osc-ffff8102fa6fbc00->192.168.4.129@o2ib netid 50000: select flavor null May 17 16:25:41 client-5 kernel: Lustre: 21607:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 1 previous similar message May 17 16:25:41 client-5 kernel: Lustre: Client lustre-client has started May 17 16:25:41 client-5 kernel: Lustre: 21624:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:25:41 client-5 kernel: Lustre: 21624:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message May 17 16:25:41 client-5 kernel: LustreError: 21674:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:25:41 client-5 kernel: LustreError: 21674:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Skipped 1 previous similar message May 17 16:25:41 client-5 kernel: LustreError: 21674:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:25:41 client-5 kernel: LustreError: 21674:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) Skipped 1 previous similar message May 17 16:25:41 client-5 kernel: Lustre: client ffff8102fa6fbc00 umount complete May 17 16:25:41 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 16:25:41 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 16:25:41 client-5 kernel: LustreError: 21695:0:(obd_config.c:1113:class_process_proc_param()) lustre-client-ffff8102f630dc00: unknown param some_wrong_param=10 May 17 16:25:41 client-5 kernel: Lustre: Client lustre-client has started May 17 16:25:58 client-5 kernel: Lustre: Acceptor stopping May 17 16:25:59 client-5 kernel: Lustre: Removed LNI 10.10.4.5@tcp May 17 16:26:01 client-5 kernel: Lustre: Removed LNI 192.168.4.5@o2ib May 17 16:26:03 client-5 xinetd[3391]: START: shell pid=22176 from=192.168.4.5 May 17 16:26:03 client-5 rshd[22177]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 16:26:03 client-5 xinetd[3391]: EXIT: shell status=0 pid=22176 duration=0(sec) May 17 16:26:03 client-5 kernel: Lustre: 22312:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:26:03 client-5 kernel: Lustre: 22313:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:26:03 client-5 kernel: Lustre: OBD class driver, http://www.lustre.org/ May 17 16:26:03 client-5 kernel: Lustre: Lustre Version: 2.0.60 May 17 16:26:03 client-5 kernel: Lustre: Build Version: jenkins-g985ac5b-PRISTINE-2.6.18-194.17.1.el5 May 17 16:26:03 client-5 kernel: Lustre: Lustre LU module (ffffffff888eb1c0). May 17 16:26:03 client-5 kernel: Lustre: Register global MR array, MR size: 0xffffffffffffffff, array size: 1 May 17 16:26:03 client-5 kernel: Lustre: Added LNI 192.168.4.5@o2ib [8/64/0/180] May 17 16:26:03 client-5 kernel: Lustre: Added LNI 10.10.4.5@tcp [8/256/0/180] May 17 16:26:03 client-5 kernel: Lustre: Accept all, port 7988 May 17 16:26:03 client-5 kernel: Lustre: Lustre OSC module (ffffffff88b3c6c0). May 17 16:26:03 client-5 kernel: Lustre: Lustre LOV module (ffffffff88bab540). May 17 16:26:03 client-5 kernel: Lustre: Lustre client module (ffffffff88c643a0). May 17 16:26:03 client-5 xinetd[3391]: START: shell pid=22716 from=192.168.4.5 May 17 16:26:03 client-5 rshd[22718]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == conf-sanity test 43: check root_squash and nosquash_nids ========================================== 16:26:03 \(1305674763\)");echo XXRETCODE:$?' May 17 16:26:04 client-5 kernel: Lustre: DEBUG MARKER: == conf-sanity test 43: check root_squash and nosquash_nids ========================================== 16:26:03 (1305674763) May 17 16:26:04 client-5 xinetd[3391]: EXIT: shell status=0 pid=22716 duration=1(sec) May 17 16:26:05 client-5 kernel: Lustre: 22846:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 16:26:05 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 16:26:05 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 16:26:05 client-5 kernel: LustreError: 22857:0:(obd_config.c:1113:class_process_proc_param()) lustre-client-ffff810321545000: unknown param some_wrong_param=10 May 17 16:26:05 client-5 kernel: Lustre: 22846:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff810321545000->192.168.4.128@o2ib netid 50000: select flavor null May 17 16:26:05 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 16:26:10 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 16:26:15 client-5 kernel: Lustre: 22846:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-OST0000-osc-ffff810321545000->192.168.4.129@o2ib netid 50000: select flavor null May 17 16:26:15 client-5 kernel: Lustre: Client lustre-client has started May 17 16:26:15 client-5 kernel: Lustre: 22867:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:26:27 client-5 xinetd[3391]: START: shell pid=23073 from=192.168.4.5 May 17 16:26:27 client-5 rshd[23074]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 16:26:27 client-5 xinetd[3391]: EXIT: shell status=0 pid=23073 duration=0(sec) May 17 16:26:27 client-5 kernel: LustreError: 23195:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:26:27 client-5 kernel: LustreError: 23195:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:26:27 client-5 kernel: LustreError: 23195:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:26:27 client-5 kernel: LustreError: 23195:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:26:27 client-5 kernel: Lustre: client ffff810321545000 umount complete May 17 16:26:44 client-5 kernel: Lustre: Acceptor stopping May 17 16:26:45 client-5 kernel: Lustre: Removed LNI 10.10.4.5@tcp May 17 16:26:47 client-5 kernel: Lustre: Removed LNI 192.168.4.5@o2ib May 17 16:26:48 client-5 kernel: Lustre: 23623:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:26:48 client-5 kernel: Lustre: 23624:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:26:48 client-5 kernel: Lustre: OBD class driver, http://www.lustre.org/ May 17 16:26:48 client-5 kernel: Lustre: Lustre Version: 2.0.60 May 17 16:26:48 client-5 kernel: Lustre: Build Version: jenkins-g985ac5b-PRISTINE-2.6.18-194.17.1.el5 May 17 16:26:48 client-5 kernel: Lustre: Lustre LU module (ffffffff888eb1c0). May 17 16:26:48 client-5 kernel: Lustre: Register global MR array, MR size: 0xffffffffffffffff, array size: 1 May 17 16:26:49 client-5 kernel: Lustre: Added LNI 192.168.4.5@o2ib [8/64/0/180] May 17 16:26:49 client-5 kernel: Lustre: Added LNI 10.10.4.5@tcp [8/256/0/180] May 17 16:26:49 client-5 kernel: Lustre: Accept all, port 7988 May 17 16:26:49 client-5 kernel: Lustre: Lustre OSC module (ffffffff88b3c6c0). May 17 16:26:49 client-5 kernel: Lustre: Lustre LOV module (ffffffff88bab540). May 17 16:26:49 client-5 kernel: Lustre: Lustre client module (ffffffff88c643a0). May 17 16:26:49 client-5 xinetd[3391]: START: shell pid=24028 from=192.168.4.5 May 17 16:26:49 client-5 rshd[24030]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == conf-sanity test 44: mounted client proc entry exists ============================================= 16:26:48 \(1305674808\)");echo XXRETCODE:$?' May 17 16:26:49 client-5 kernel: Lustre: DEBUG MARKER: == conf-sanity test 44: mounted client proc entry exists ============================================= 16:26:48 (1305674808) May 17 16:26:49 client-5 xinetd[3391]: EXIT: shell status=0 pid=24028 duration=0(sec) May 17 16:26:50 client-5 kernel: Lustre: 24159:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 16:26:50 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 16:26:50 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 16:26:50 client-5 kernel: LustreError: 24170:0:(obd_config.c:1113:class_process_proc_param()) lustre-client-ffff810317cad800: unknown param some_wrong_param=10 May 17 16:26:50 client-5 kernel: Lustre: 24159:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff810317cad800->192.168.4.128@o2ib netid 50000: select flavor null May 17 16:26:50 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 16:26:55 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 16:27:00 client-5 kernel: Lustre: 24159:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-OST0000-osc-ffff810317cad800->192.168.4.129@o2ib netid 50000: select flavor null May 17 16:27:00 client-5 kernel: Lustre: Client lustre-client has started May 17 16:27:00 client-5 kernel: Lustre: 24176:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:27:01 client-5 kernel: LustreError: 24234:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:27:01 client-5 kernel: LustreError: 24234:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:27:01 client-5 kernel: LustreError: 24234:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:27:01 client-5 kernel: LustreError: 24234:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:27:01 client-5 kernel: Lustre: client ffff810317cad800 umount complete May 17 16:27:23 client-5 kernel: Lustre: Acceptor stopping May 17 16:27:24 client-5 kernel: Lustre: Removed LNI 10.10.4.5@tcp May 17 16:27:26 client-5 kernel: Lustre: Removed LNI 192.168.4.5@o2ib May 17 16:27:28 client-5 xinetd[3391]: START: shell pid=24692 from=192.168.4.5 May 17 16:27:28 client-5 rshd[24693]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 16:27:28 client-5 xinetd[3391]: EXIT: shell status=0 pid=24692 duration=0(sec) May 17 16:27:28 client-5 kernel: Lustre: 24828:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:27:28 client-5 kernel: Lustre: 24831:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:27:28 client-5 kernel: Lustre: OBD class driver, http://www.lustre.org/ May 17 16:27:28 client-5 kernel: Lustre: Lustre Version: 2.0.60 May 17 16:27:28 client-5 kernel: Lustre: Build Version: jenkins-g985ac5b-PRISTINE-2.6.18-194.17.1.el5 May 17 16:27:28 client-5 kernel: Lustre: Lustre LU module (ffffffff888eb1c0). May 17 16:27:28 client-5 kernel: Lustre: Register global MR array, MR size: 0xffffffffffffffff, array size: 1 May 17 16:27:28 client-5 kernel: Lustre: Added LNI 192.168.4.5@o2ib [8/64/0/180] May 17 16:27:28 client-5 kernel: Lustre: Added LNI 10.10.4.5@tcp [8/256/0/180] May 17 16:27:28 client-5 kernel: Lustre: Accept all, port 7988 May 17 16:27:28 client-5 kernel: Lustre: Lustre OSC module (ffffffff88b3c6c0). May 17 16:27:28 client-5 kernel: Lustre: Lustre LOV module (ffffffff88bab540). May 17 16:27:28 client-5 kernel: Lustre: Lustre client module (ffffffff88c643a0). May 17 16:27:28 client-5 xinetd[3391]: START: shell pid=25229 from=192.168.4.5 May 17 16:27:28 client-5 rshd[25231]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == conf-sanity test 45: long unlink handling in ptlrpcd ============================================== 16:27:28 \(1305674848\)");echo XXRETCODE:$?' May 17 16:27:29 client-5 kernel: Lustre: DEBUG MARKER: == conf-sanity test 45: long unlink handling in ptlrpcd ============================================== 16:27:28 (1305674848) May 17 16:27:29 client-5 xinetd[3391]: EXIT: shell status=0 pid=25229 duration=1(sec) May 17 16:27:30 client-5 kernel: Lustre: 25362:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 16:27:30 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 16:27:30 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 16:27:30 client-5 kernel: LustreError: 25373:0:(obd_config.c:1113:class_process_proc_param()) lustre-client-ffff8102cd8e5c00: unknown param some_wrong_param=10 May 17 16:27:30 client-5 kernel: Lustre: 25362:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff8102cd8e5c00->192.168.4.128@o2ib netid 50000: select flavor null May 17 16:27:30 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 16:27:35 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 16:27:40 client-5 kernel: Lustre: 25362:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-OST0000-osc-ffff8102cd8e5c00->192.168.4.129@o2ib netid 50000: select flavor null May 17 16:27:40 client-5 kernel: Lustre: Client lustre-client has started May 17 16:27:40 client-5 kernel: Lustre: 25379:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:27:53 client-5 xinetd[3391]: START: shell pid=25493 from=192.168.4.5 May 17 16:27:53 client-5 rshd[25494]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark sleep 60 sec");echo XXRETCODE:$?' May 17 16:27:53 client-5 kernel: Lustre: DEBUG MARKER: sleep 60 sec May 17 16:27:53 client-5 xinetd[3391]: EXIT: shell status=0 pid=25493 duration=0(sec) May 17 16:27:57 client-5 kernel: Lustre: 24933:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369099309416483 sent from MGC192.168.4.128@o2ib to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305674870] [real_sent 1305674870] [current 1305674877] [deadline 7s] [delay 0s] req@ffff81031bb3ec00 x1369099309416483/t0(0) o-1->MGS@MGC192.168.4.128@o2ib_0:26/25 lens 192/192 e 0 to 1 dl 1305674877 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 16:27:57 client-5 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail. May 17 16:27:57 client-5 kernel: Lustre: 24933:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369099309416484 sent from lustre-MDT0000-mdc-ffff8102cd8e5c00 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305674870] [real_sent 1305674870] [current 1305674877] [deadline 7s] [delay 0s] req@ffff810327f42c00 x1369099309416484/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 192/192 e 0 to 1 dl 1305674877 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 16:27:57 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff8102cd8e5c00: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 16:28:00 client-5 kernel: Lustre: 25434:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369099309416486 sent from lustre-MDT0000-mdc-ffff8102cd8e5c00 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305674873] [real_sent 1305674873] [current 1305674880] [deadline 7s] [delay 0s] req@ffff81031bd57c00 x1369099309416486/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 544/992 e 0 to 1 dl 1305674880 ref 2 fl Rpc:X/ffffffff/ffffffff rc 0/-1 May 17 16:28:03 client-5 kernel: Lustre: 24934:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369099309416489 sent from MGC192.168.4.128@o2ib to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305674877] [real_sent 1305674877] [current 1305674883] [deadline 6s] [delay 0s] req@ffff8102fbe26000 x1369099309416489/t0(0) o-1->MGS@MGC192.168.4.128@o2ib_0:26/25 lens 368/392 e 0 to 1 dl 1305674883 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 16:28:07 client-5 kernel: Lustre: 24935:0:(import.c:529:import_select_connection()) MGC192.168.4.128@o2ib: tried all connections, increasing latency to 6s May 17 16:28:07 client-5 kernel: Lustre: 24935:0:(import.c:529:import_select_connection()) lustre-MDT0000-mdc-ffff8102cd8e5c00: tried all connections, increasing latency to 6s May 17 16:28:18 client-5 kernel: Lustre: 24934:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369099309416492 sent from MGC192.168.4.128@o2ib to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305674887] [real_sent 1305674887] [current 1305674898] [deadline 11s] [delay 0s] req@ffff81031bd57000 x1369099309416492/t0(0) o-1->MGS@MGC192.168.4.128@o2ib_0:26/25 lens 368/392 e 0 to 1 dl 1305674898 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 16:28:18 client-5 kernel: Lustre: 24934:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 1 previous similar message May 17 16:28:22 client-5 kernel: Lustre: 24935:0:(import.c:529:import_select_connection()) MGC192.168.4.128@o2ib: tried all connections, increasing latency to 11s May 17 16:28:38 client-5 kernel: Lustre: 24934:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369099309416496 sent from MGC192.168.4.128@o2ib to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305674902] [real_sent 1305674902] [current 1305674918] [deadline 16s] [delay 0s] req@ffff810314b3b400 x1369099309416496/t0(0) o-1->MGS@MGC192.168.4.128@o2ib_0:26/25 lens 368/392 e 0 to 1 dl 1305674918 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 16:28:38 client-5 kernel: Lustre: 24934:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 1 previous similar message May 17 16:28:42 client-5 kernel: Lustre: 24935:0:(import.c:529:import_select_connection()) MGC192.168.4.128@o2ib: tried all connections, increasing latency to 16s May 17 16:28:42 client-5 kernel: Lustre: 24935:0:(import.c:529:import_select_connection()) Skipped 1 previous similar message May 17 16:28:53 client-5 kernel: LustreError: 24934:0:(obd_support.h:456:obd_fail_check_set()) *** obd_fail_loc=50f *** May 17 16:28:53 client-5 kernel: LustreError: 24934:0:(obd_support.h:456:obd_fail_check_set()) *** obd_fail_loc=50f *** May 17 16:28:53 client-5 xinetd[3391]: START: shell pid=25597 from=192.168.4.5 May 17 16:28:53 client-5 rshd[25598]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark sleep 10 sec");echo XXRETCODE:$?' May 17 16:28:53 client-5 kernel: Lustre: DEBUG MARKER: sleep 10 sec May 17 16:28:53 client-5 xinetd[3391]: EXIT: shell status=0 pid=25597 duration=0(sec) May 17 16:28:54 client-5 kernel: LustreError: 25434:0:(obd_support.h:456:obd_fail_check_set()) *** obd_fail_loc=50f *** May 17 16:28:54 client-5 kernel: LustreError: 25434:0:(obd_support.h:456:obd_fail_check_set()) Skipped 6 previous similar messages May 17 16:28:55 client-5 kernel: LustreError: 25434:0:(obd_support.h:456:obd_fail_check_set()) *** obd_fail_loc=50f *** May 17 16:28:55 client-5 kernel: LustreError: 25434:0:(obd_support.h:456:obd_fail_check_set()) Skipped 11 previous similar messages May 17 16:28:57 client-5 kernel: LustreError: 25434:0:(obd_support.h:456:obd_fail_check_set()) *** obd_fail_loc=50f *** May 17 16:28:57 client-5 kernel: LustreError: 25434:0:(obd_support.h:456:obd_fail_check_set()) Skipped 19 previous similar messages May 17 16:29:01 client-5 kernel: LustreError: 25434:0:(obd_support.h:456:obd_fail_check_set()) *** obd_fail_loc=50f *** May 17 16:29:01 client-5 kernel: LustreError: 25434:0:(obd_support.h:456:obd_fail_check_set()) Skipped 52 previous similar messages May 17 16:29:03 client-5 kernel: Lustre: 24934:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369099309416501 sent from MGC192.168.4.128@o2ib to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305674922] [real_sent 1305674922] [current 1305674943] [deadline 21s] [delay 0s] req@ffff810314b3bc00 x1369099309416501/t0(0) o-1->MGS@MGC192.168.4.128@o2ib_0:26/25 lens 368/392 e 0 to 1 dl 1305674943 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 16:29:03 client-5 kernel: Lustre: 24934:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 1 previous similar message May 17 16:29:03 client-5 kernel: Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request May 17 16:29:03 client-5 kernel: LustreError: 25638:0:(obd_support.h:456:obd_fail_check_set()) *** obd_fail_loc=50f *** May 17 16:29:04 client-5 kernel: LustreError: 25638:0:(import.c:323:ptlrpc_invalidate_import()) lustre-MDT0000_UUID: rc = -110 waiting for callback (2 != 0) May 17 16:29:04 client-5 kernel: LustreError: 25638:0:(import.c:349:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff81031b717c00 x1369099309416502/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 368/392 e 0 to 1 dl 1305674943 ref 1 fl Unregistering:EXN/ffffffff/ffffffff rc -4/-1 May 17 16:29:04 client-5 kernel: LustreError: 25638:0:(import.c:357:ptlrpc_invalidate_import()) @@@ still on delayed list req@ffff81031bd57c00 x1369099309416486/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 544/992 e 0 to 1 dl 1305674880 ref 2 fl Unregistering:EX/ffffffff/ffffffff rc -4/-1 May 17 16:29:04 client-5 kernel: LustreError: 25638:0:(import.c:365:ptlrpc_invalidate_import()) lustre-MDT0000_UUID: RPCs in "Unregistering" phase found (2). Network is sluggish? Waiting them to error out. May 17 16:29:04 client-5 kernel: LustreError: 25638:0:(obd_support.h:456:obd_fail_check_set()) *** obd_fail_loc=50f *** May 17 16:29:05 client-5 kernel: LustreError: 25638:0:(import.c:323:ptlrpc_invalidate_import()) lustre-MDT0000_UUID: rc = -110 waiting for callback (2 != 0) May 17 16:29:05 client-5 kernel: LustreError: 25638:0:(import.c:349:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff81031b717c00 x1369099309416502/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 368/392 e 0 to 1 dl 1305674943 ref 1 fl Unregistering:EXN/ffffffff/ffffffff rc -4/-1 May 17 16:29:06 client-5 kernel: LustreError: 25638:0:(import.c:357:ptlrpc_invalidate_import()) @@@ still on delayed list req@ffff81031bd57c00 x1369099309416486/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 544/992 e 0 to 1 dl 1305674880 ref 2 fl Unregistering:EX/ffffffff/ffffffff rc -4/-1 May 17 16:29:06 client-5 kernel: LustreError: 25638:0:(import.c:365:ptlrpc_invalidate_import()) lustre-MDT0000_UUID: RPCs in "Unregistering" phase found (2). Network is sluggish? Waiting them to error out. May 17 16:29:06 client-5 kernel: LustreError: 25638:0:(obd_support.h:456:obd_fail_check_set()) *** obd_fail_loc=50f *** May 17 16:29:07 client-5 kernel: LustreError: 25638:0:(import.c:323:ptlrpc_invalidate_import()) lustre-MDT0000_UUID: rc = -110 waiting for callback (2 != 0) May 17 16:29:07 client-5 kernel: LustreError: 25638:0:(import.c:349:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff81031b717c00 x1369099309416502/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 368/392 e 0 to 1 dl 1305674943 ref 1 fl Unregistering:EXN/ffffffff/ffffffff rc -4/-1 May 17 16:29:07 client-5 kernel: LustreError: 25638:0:(import.c:357:ptlrpc_invalidate_import()) @@@ still on delayed list req@ffff81031bd57c00 x1369099309416486/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 544/992 e 0 to 1 dl 1305674880 ref 2 fl Unregistering:EX/ffffffff/ffffffff rc -4/-1 May 17 16:29:07 client-5 kernel: LustreError: 25638:0:(import.c:365:ptlrpc_invalidate_import()) lustre-MDT0000_UUID: RPCs in "Unregistering" phase found (2). Network is sluggish? Waiting them to error out. May 17 16:29:07 client-5 kernel: LustreError: 25638:0:(obd_support.h:456:obd_fail_check_set()) *** obd_fail_loc=50f *** May 17 16:29:08 client-5 kernel: LustreError: 25638:0:(import.c:323:ptlrpc_invalidate_import()) lustre-MDT0000_UUID: rc = -110 waiting for callback (2 != 0) May 17 16:29:08 client-5 kernel: LustreError: 25638:0:(import.c:349:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff81031b717c00 x1369099309416502/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 368/392 e 0 to 1 dl 1305674943 ref 1 fl Unregistering:EXN/ffffffff/ffffffff rc -4/-1 May 17 16:29:08 client-5 kernel: LustreError: 25638:0:(import.c:357:ptlrpc_invalidate_import()) @@@ still on delayed list req@ffff81031bd57c00 x1369099309416486/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 544/992 e 0 to 1 dl 1305674880 ref 2 fl Unregistering:EX/ffffffff/ffffffff rc -4/-1 May 17 16:29:08 client-5 kernel: LustreError: 25638:0:(import.c:365:ptlrpc_invalidate_import()) lustre-MDT0000_UUID: RPCs in "Unregistering" phase found (2). Network is sluggish? Waiting them to error out. May 17 16:29:09 client-5 kernel: LustreError: 25638:0:(obd_support.h:456:obd_fail_check_set()) *** obd_fail_loc=50f *** May 17 16:29:09 client-5 kernel: LustreError: 25638:0:(obd_support.h:456:obd_fail_check_set()) Skipped 1 previous similar message May 17 16:29:09 client-5 kernel: LustreError: 25434:0:(obd_support.h:456:obd_fail_check_set()) *** obd_fail_loc=50f *** May 17 16:29:09 client-5 kernel: LustreError: 25434:0:(obd_support.h:456:obd_fail_check_set()) Skipped 108 previous similar messages May 17 16:29:10 client-5 kernel: LustreError: 25638:0:(import.c:323:ptlrpc_invalidate_import()) lustre-MDT0000_UUID: rc = -110 waiting for callback (2 != 0) May 17 16:29:10 client-5 kernel: LustreError: 25638:0:(import.c:323:ptlrpc_invalidate_import()) Skipped 1 previous similar message May 17 16:29:10 client-5 kernel: LustreError: 25638:0:(import.c:349:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff81031b717c00 x1369099309416502/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 368/392 e 0 to 1 dl 1305674943 ref 1 fl Unregistering:EXN/ffffffff/ffffffff rc -4/-1 May 17 16:29:10 client-5 kernel: LustreError: 25638:0:(import.c:349:ptlrpc_invalidate_import()) Skipped 1 previous similar message May 17 16:29:10 client-5 kernel: LustreError: 25638:0:(import.c:357:ptlrpc_invalidate_import()) @@@ still on delayed list req@ffff81031bd57c00 x1369099309416486/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 544/992 e 0 to 1 dl 1305674880 ref 2 fl Unregistering:EX/ffffffff/ffffffff rc -4/-1 May 17 16:29:10 client-5 kernel: LustreError: 25638:0:(import.c:357:ptlrpc_invalidate_import()) Skipped 1 previous similar message May 17 16:29:10 client-5 kernel: LustreError: 25638:0:(import.c:365:ptlrpc_invalidate_import()) lustre-MDT0000_UUID: RPCs in "Unregistering" phase found (2). Network is sluggish? Waiting them to error out. May 17 16:29:10 client-5 kernel: LustreError: 25638:0:(import.c:365:ptlrpc_invalidate_import()) Skipped 1 previous similar message May 17 16:29:13 client-5 kernel: LustreError: 25638:0:(obd_support.h:456:obd_fail_check_set()) *** obd_fail_loc=50f *** May 17 16:29:13 client-5 kernel: LustreError: 25638:0:(obd_support.h:456:obd_fail_check_set()) Skipped 3 previous similar messages May 17 16:29:14 client-5 kernel: LustreError: 25638:0:(import.c:323:ptlrpc_invalidate_import()) lustre-MDT0000_UUID: rc = -110 waiting for callback (2 != 0) May 17 16:29:14 client-5 kernel: LustreError: 25638:0:(import.c:323:ptlrpc_invalidate_import()) Skipped 3 previous similar messages May 17 16:29:14 client-5 kernel: LustreError: 25638:0:(import.c:349:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff81031b717c00 x1369099309416502/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 368/392 e 0 to 1 dl 1305674943 ref 1 fl Unregistering:EXN/ffffffff/ffffffff rc -4/-1 May 17 16:29:14 client-5 kernel: LustreError: 25638:0:(import.c:349:ptlrpc_invalidate_import()) Skipped 3 previous similar messages May 17 16:29:14 client-5 kernel: LustreError: 25638:0:(import.c:357:ptlrpc_invalidate_import()) @@@ still on delayed list req@ffff81031bd57c00 x1369099309416486/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 544/992 e 0 to 1 dl 1305674880 ref 2 fl Unregistering:EX/ffffffff/ffffffff rc -4/-1 May 17 16:29:14 client-5 kernel: LustreError: 25638:0:(import.c:357:ptlrpc_invalidate_import()) Skipped 3 previous similar messages May 17 16:29:14 client-5 kernel: LustreError: 25638:0:(import.c:365:ptlrpc_invalidate_import()) lustre-MDT0000_UUID: RPCs in "Unregistering" phase found (2). Network is sluggish? Waiting them to error out. May 17 16:29:14 client-5 kernel: LustreError: 25638:0:(import.c:365:ptlrpc_invalidate_import()) Skipped 3 previous similar messages May 17 16:29:21 client-5 kernel: LustreError: 25638:0:(obd_support.h:456:obd_fail_check_set()) *** obd_fail_loc=50f *** May 17 16:29:21 client-5 kernel: LustreError: 25638:0:(obd_support.h:456:obd_fail_check_set()) Skipped 7 previous similar messages May 17 16:29:22 client-5 kernel: LustreError: 25638:0:(import.c:323:ptlrpc_invalidate_import()) lustre-MDT0000_UUID: rc = -110 waiting for callback (2 != 0) May 17 16:29:22 client-5 kernel: LustreError: 25638:0:(import.c:323:ptlrpc_invalidate_import()) Skipped 7 previous similar messages May 17 16:29:22 client-5 kernel: LustreError: 25638:0:(import.c:349:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff81031b717c00 x1369099309416502/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 368/392 e 0 to 1 dl 1305674943 ref 1 fl Unregistering:EXN/ffffffff/ffffffff rc -4/-1 May 17 16:29:22 client-5 kernel: LustreError: 25638:0:(import.c:349:ptlrpc_invalidate_import()) Skipped 7 previous similar messages May 17 16:29:22 client-5 kernel: LustreError: 25638:0:(import.c:357:ptlrpc_invalidate_import()) @@@ still on delayed list req@ffff81031bd57c00 x1369099309416486/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 544/992 e 0 to 1 dl 1305674880 ref 2 fl Unregistering:EX/ffffffff/ffffffff rc -4/-1 May 17 16:29:22 client-5 kernel: LustreError: 25638:0:(import.c:357:ptlrpc_invalidate_import()) Skipped 7 previous similar messages May 17 16:29:22 client-5 kernel: LustreError: 25638:0:(import.c:365:ptlrpc_invalidate_import()) lustre-MDT0000_UUID: RPCs in "Unregistering" phase found (2). Network is sluggish? Waiting them to error out. May 17 16:29:22 client-5 kernel: LustreError: 25638:0:(import.c:365:ptlrpc_invalidate_import()) Skipped 7 previous similar messages May 17 16:29:25 client-5 kernel: LustreError: 25434:0:(obd_support.h:456:obd_fail_check_set()) *** obd_fail_loc=50f *** May 17 16:29:25 client-5 kernel: LustreError: 25434:0:(obd_support.h:456:obd_fail_check_set()) Skipped 194 previous similar messages May 17 16:29:37 client-5 kernel: LustreError: 25638:0:(obd_support.h:456:obd_fail_check_set()) *** obd_fail_loc=50f *** May 17 16:29:37 client-5 kernel: LustreError: 25638:0:(obd_support.h:456:obd_fail_check_set()) Skipped 15 previous similar messages May 17 16:29:38 client-5 kernel: LustreError: 25638:0:(import.c:323:ptlrpc_invalidate_import()) lustre-MDT0000_UUID: rc = -110 waiting for callback (2 != 0) May 17 16:29:38 client-5 kernel: LustreError: 25638:0:(import.c:323:ptlrpc_invalidate_import()) Skipped 15 previous similar messages May 17 16:29:38 client-5 kernel: LustreError: 25638:0:(import.c:349:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff81031b717c00 x1369099309416502/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 368/392 e 0 to 1 dl 1305674943 ref 1 fl Unregistering:EXN/ffffffff/ffffffff rc -4/-1 May 17 16:29:38 client-5 kernel: LustreError: 25638:0:(import.c:349:ptlrpc_invalidate_import()) Skipped 15 previous similar messages May 17 16:29:38 client-5 kernel: LustreError: 25638:0:(import.c:357:ptlrpc_invalidate_import()) @@@ still on delayed list req@ffff81031bd57c00 x1369099309416486/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 544/992 e 0 to 1 dl 1305674880 ref 2 fl Unregistering:EX/ffffffff/ffffffff rc -4/-1 May 17 16:29:38 client-5 kernel: LustreError: 25638:0:(import.c:357:ptlrpc_invalidate_import()) Skipped 15 previous similar messages May 17 16:29:38 client-5 kernel: LustreError: 25638:0:(import.c:365:ptlrpc_invalidate_import()) lustre-MDT0000_UUID: RPCs in "Unregistering" phase found (2). Network is sluggish? Waiting them to error out. May 17 16:29:38 client-5 kernel: LustreError: 25638:0:(import.c:365:ptlrpc_invalidate_import()) Skipped 15 previous similar messages May 17 16:29:57 client-5 kernel: LustreError: 25434:0:(obd_support.h:456:obd_fail_check_set()) *** obd_fail_loc=50f *** May 17 16:29:57 client-5 kernel: LustreError: 25434:0:(obd_support.h:456:obd_fail_check_set()) Skipped 524 previous similar messages May 17 16:30:05 client-5 xinetd[3391]: EXIT: login status=0 pid=12737 duration=538(sec) May 17 16:30:09 client-5 kernel: LustreError: 25638:0:(obd_support.h:456:obd_fail_check_set()) *** obd_fail_loc=50f *** May 17 16:30:09 client-5 kernel: LustreError: 25638:0:(obd_support.h:456:obd_fail_check_set()) Skipped 31 previous similar messages May 17 16:30:10 client-5 kernel: LustreError: 25638:0:(import.c:323:ptlrpc_invalidate_import()) lustre-MDT0000_UUID: rc = -110 waiting for callback (2 != 0) May 17 16:30:10 client-5 kernel: LustreError: 25638:0:(import.c:323:ptlrpc_invalidate_import()) Skipped 31 previous similar messages May 17 16:30:10 client-5 kernel: LustreError: 25638:0:(import.c:349:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff81031b717c00 x1369099309416502/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 368/392 e 0 to 1 dl 1305674943 ref 1 fl Unregistering:EXN/ffffffff/ffffffff rc -4/-1 May 17 16:30:10 client-5 kernel: LustreError: 25638:0:(import.c:349:ptlrpc_invalidate_import()) Skipped 31 previous similar messages May 17 16:30:10 client-5 kernel: LustreError: 25638:0:(import.c:357:ptlrpc_invalidate_import()) @@@ still on delayed list req@ffff81031bd57c00 x1369099309416486/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 544/992 e 0 to 1 dl 1305674880 ref 2 fl Unregistering:EX/ffffffff/ffffffff rc -4/-1 May 17 16:30:10 client-5 kernel: LustreError: 25638:0:(import.c:357:ptlrpc_invalidate_import()) Skipped 31 previous similar messages May 17 16:30:10 client-5 kernel: LustreError: 25638:0:(import.c:365:ptlrpc_invalidate_import()) lustre-MDT0000_UUID: RPCs in "Unregistering" phase found (2). Network is sluggish? Waiting them to error out. May 17 16:30:10 client-5 kernel: LustreError: 25638:0:(import.c:365:ptlrpc_invalidate_import()) Skipped 31 previous similar messages May 17 16:31:01 client-5 kernel: LustreError: 25434:0:(obd_support.h:456:obd_fail_check_set()) *** obd_fail_loc=50f *** May 17 16:31:01 client-5 kernel: LustreError: 25434:0:(obd_support.h:456:obd_fail_check_set()) Skipped 1722 previous similar messages May 17 16:31:13 client-5 kernel: LustreError: 25638:0:(obd_support.h:456:obd_fail_check_set()) *** obd_fail_loc=50f *** May 17 16:31:13 client-5 kernel: LustreError: 25638:0:(obd_support.h:456:obd_fail_check_set()) Skipped 63 previous similar messages May 17 16:31:14 client-5 kernel: LustreError: 25638:0:(import.c:323:ptlrpc_invalidate_import()) lustre-MDT0000_UUID: rc = -110 waiting for callback (2 != 0) May 17 16:31:14 client-5 kernel: LustreError: 25638:0:(import.c:323:ptlrpc_invalidate_import()) Skipped 63 previous similar messages May 17 16:31:14 client-5 kernel: LustreError: 25638:0:(import.c:349:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff81031b717c00 x1369099309416502/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 368/392 e 0 to 1 dl 1305674943 ref 1 fl Unregistering:EXN/ffffffff/ffffffff rc -4/-1 May 17 16:31:14 client-5 kernel: LustreError: 25638:0:(import.c:349:ptlrpc_invalidate_import()) Skipped 63 previous similar messages May 17 16:31:14 client-5 kernel: LustreError: 25638:0:(import.c:357:ptlrpc_invalidate_import()) @@@ still on delayed list req@ffff81031bd57c00 x1369099309416486/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 544/992 e 0 to 1 dl 1305674880 ref 2 fl Unregistering:EX/ffffffff/ffffffff rc -4/-1 May 17 16:31:14 client-5 kernel: LustreError: 25638:0:(import.c:357:ptlrpc_invalidate_import()) Skipped 63 previous similar messages May 17 16:31:14 client-5 kernel: LustreError: 25638:0:(import.c:365:ptlrpc_invalidate_import()) lustre-MDT0000_UUID: RPCs in "Unregistering" phase found (2). Network is sluggish? Waiting them to error out. May 17 16:31:14 client-5 kernel: LustreError: 25638:0:(import.c:365:ptlrpc_invalidate_import()) Skipped 63 previous similar messages May 17 16:33:09 client-5 kernel: LustreError: 25434:0:(obd_support.h:456:obd_fail_check_set()) *** obd_fail_loc=50f *** May 17 16:33:09 client-5 kernel: LustreError: 25434:0:(obd_support.h:456:obd_fail_check_set()) Skipped 5980 previous similar messages May 17 16:33:21 client-5 kernel: LustreError: 25638:0:(obd_support.h:456:obd_fail_check_set()) *** obd_fail_loc=50f *** May 17 16:33:21 client-5 kernel: LustreError: 25638:0:(obd_support.h:456:obd_fail_check_set()) Skipped 127 previous similar messages May 17 16:33:22 client-5 kernel: LustreError: 25638:0:(import.c:323:ptlrpc_invalidate_import()) lustre-MDT0000_UUID: rc = -110 waiting for callback (2 != 0) May 17 16:33:22 client-5 kernel: LustreError: 25638:0:(import.c:323:ptlrpc_invalidate_import()) Skipped 127 previous similar messages May 17 16:33:23 client-5 kernel: LustreError: 25638:0:(import.c:349:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff81031b717c00 x1369099309416502/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 368/392 e 0 to 1 dl 1305674943 ref 1 fl Unregistering:EXN/ffffffff/ffffffff rc -4/-1 May 17 16:33:23 client-5 kernel: LustreError: 25638:0:(import.c:349:ptlrpc_invalidate_import()) Skipped 127 previous similar messages May 17 16:33:23 client-5 kernel: LustreError: 25638:0:(import.c:357:ptlrpc_invalidate_import()) @@@ still on delayed list req@ffff81031bd57c00 x1369099309416486/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 544/992 e 0 to 1 dl 1305674880 ref 2 fl Unregistering:EX/ffffffff/ffffffff rc -4/-1 May 17 16:33:23 client-5 kernel: LustreError: 25638:0:(import.c:357:ptlrpc_invalidate_import()) Skipped 127 previous similar messages May 17 16:33:23 client-5 kernel: LustreError: 25638:0:(import.c:365:ptlrpc_invalidate_import()) lustre-MDT0000_UUID: RPCs in "Unregistering" phase found (2). Network is sluggish? Waiting them to error out. May 17 16:33:23 client-5 kernel: LustreError: 25638:0:(import.c:365:ptlrpc_invalidate_import()) Skipped 127 previous similar messages May 17 16:33:51 client-5 xinetd[3391]: START: login pid=25656 from=10.10.0.1 May 17 16:33:54 client-5 kernel: LustreError: 25434:0:(mdc_locks.c:722:mdc_enqueue()) ldlm_cli_enqueue: -4 May 17 16:33:54 client-5 kernel: LustreError: 25434:0:(file.c:2161:ll_inode_revalidate_fini()) failure -4 inode 20 May 17 16:33:54 client-5 kernel: LustreError: 25434:0:(mdc_locks.c:722:mdc_enqueue()) ldlm_cli_enqueue: -108 May 17 16:33:54 client-5 kernel: LustreError: 25434:0:(file.c:2161:ll_inode_revalidate_fini()) failure -108 inode 20 May 17 16:33:54 client-5 kernel: Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request May 17 16:33:54 client-5 kernel: Lustre: client ffff8102cd8e5c00 umount complete May 17 16:33:55 client-5 kernel: Lustre: 25760:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 16:33:55 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 16:33:55 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 16:33:55 client-5 kernel: LustreError: 25771:0:(obd_config.c:1113:class_process_proc_param()) lustre-client-ffff8102ff41cc00: unknown param some_wrong_param=10 May 17 16:33:55 client-5 kernel: Lustre: Client lustre-client has started May 17 16:33:55 client-5 kernel: Lustre: 25777:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:33:55 client-5 kernel: Lustre: 25777:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message May 17 16:33:56 client-5 kernel: LustreError: 25798:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:33:56 client-5 kernel: LustreError: 25798:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:33:56 client-5 kernel: LustreError: 25798:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:33:56 client-5 kernel: LustreError: 25798:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:33:56 client-5 kernel: Lustre: client ffff8102ff41cc00 umount complete May 17 16:34:15 client-5 kernel: Lustre: Acceptor stopping May 17 16:34:16 client-5 kernel: Lustre: Removed LNI 10.10.4.5@tcp May 17 16:34:18 client-5 kernel: Lustre: Removed LNI 192.168.4.5@o2ib May 17 16:34:20 client-5 xinetd[3391]: START: shell pid=26277 from=192.168.4.5 May 17 16:34:20 client-5 rshd[26278]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 16:34:20 client-5 xinetd[3391]: EXIT: shell status=0 pid=26277 duration=0(sec) May 17 16:34:20 client-5 kernel: Lustre: 26412:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:34:20 client-5 kernel: Lustre: 26415:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:34:20 client-5 kernel: Lustre: OBD class driver, http://www.lustre.org/ May 17 16:34:20 client-5 kernel: Lustre: Lustre Version: 2.0.60 May 17 16:34:20 client-5 kernel: Lustre: Build Version: jenkins-g985ac5b-PRISTINE-2.6.18-194.17.1.el5 May 17 16:34:20 client-5 kernel: Lustre: Lustre LU module (ffffffff888eb1c0). May 17 16:34:20 client-5 kernel: Lustre: Register global MR array, MR size: 0xffffffffffffffff, array size: 1 May 17 16:34:20 client-5 kernel: Lustre: Added LNI 192.168.4.5@o2ib [8/64/0/180] May 17 16:34:20 client-5 kernel: Lustre: Added LNI 10.10.4.5@tcp [8/256/0/180] May 17 16:34:20 client-5 kernel: Lustre: Accept all, port 7988 May 17 16:34:20 client-5 kernel: Lustre: Lustre OSC module (ffffffff88b3c6c0). May 17 16:34:20 client-5 kernel: Lustre: Lustre LOV module (ffffffff88bab540). May 17 16:34:20 client-5 kernel: Lustre: Lustre client module (ffffffff88c643a0). May 17 16:34:20 client-5 xinetd[3391]: START: shell pid=26812 from=192.168.4.5 May 17 16:34:20 client-5 rshd[26814]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == conf-sanity test 46a: handle ost additional - wide striped file =================================== 16:34:20 \(1305675260\)");echo XXRETCODE:$?' May 17 16:34:21 client-5 kernel: Lustre: DEBUG MARKER: == conf-sanity test 46a: handle ost additional - wide striped file =================================== 16:34:20 (1305675260) May 17 16:34:21 client-5 xinetd[3391]: EXIT: shell status=0 pid=26812 duration=1(sec) May 17 16:34:21 client-5 xinetd[3391]: START: shell pid=26863 from=192.168.4.5 May 17 16:34:21 client-5 rshd[26864]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:; lsof -t /mnt/lustre || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre is still busy, wait one second && sleep 1; umount /mnt/lustre; fi fi");echo XXRETCODE:$?' May 17 16:34:21 client-5 xinetd[3391]: EXIT: shell status=0 pid=26863 duration=0(sec) May 17 16:34:21 client-5 xinetd[3391]: START: shell pid=26908 from=192.168.4.5 May 17 16:34:21 client-5 rshd[26909]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:; lsof -t /mnt/lustre2 || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre2 | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre2 is still busy, wait one second && sleep 1; umount /mnt/lustre2; fi fi");echo XXRETCODE:$?' May 17 16:34:21 client-5 xinetd[3391]: EXIT: shell status=0 pid=26908 duration=0(sec) May 17 16:34:22 client-5 kernel: Lustre: 27111:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:34:33 client-5 xinetd[3391]: START: shell pid=27695 from=192.168.4.5 May 17 16:34:33 client-5 rshd[27696]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark ost now in FULL state");echo XXRETCODE:$?' May 17 16:34:33 client-5 kernel: Lustre: DEBUG MARKER: ost now in FULL state May 17 16:34:33 client-5 xinetd[3391]: EXIT: shell status=0 pid=27695 duration=0(sec) May 17 16:34:58 client-5 xinetd[3391]: START: shell pid=28117 from=192.168.4.5 May 17 16:34:58 client-5 rshd[28119]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark ost now in FULL state");echo XXRETCODE:$?' May 17 16:34:58 client-5 kernel: Lustre: DEBUG MARKER: ost now in FULL state May 17 16:34:58 client-5 xinetd[3391]: EXIT: shell status=0 pid=28117 duration=0(sec) May 17 16:34:58 client-5 kernel: Lustre: 28162:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 16:34:58 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 16:34:58 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 16:34:58 client-5 kernel: Lustre: 28162:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff8102e4462c00->192.168.4.128@o2ib netid 50000: select flavor null May 17 16:34:58 client-5 kernel: Lustre: Client lustre-client has started May 17 16:34:58 client-5 kernel: Lustre: 28179:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:34:58 client-5 kernel: Lustre: 28179:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message May 17 16:35:03 client-5 xinetd[3391]: START: shell pid=28519 from=192.168.4.5 May 17 16:35:03 client-5 rshd[28521]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark ost2 now in FULL state");echo XXRETCODE:$?' May 17 16:35:03 client-5 kernel: Lustre: DEBUG MARKER: ost2 now in FULL state May 17 16:35:03 client-5 xinetd[3391]: EXIT: shell status=0 pid=28519 duration=0(sec) May 17 16:35:03 client-5 xinetd[3391]: START: shell pid=28693 from=192.168.4.5 May 17 16:35:03 client-5 rshd[28695]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark ost3 now in FULL state");echo XXRETCODE:$?' May 17 16:35:03 client-5 kernel: Lustre: DEBUG MARKER: ost3 now in FULL state May 17 16:35:03 client-5 xinetd[3391]: EXIT: shell status=0 pid=28693 duration=0(sec) May 17 16:35:04 client-5 kernel: Lustre: 28776:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-OST0001-osc-ffff8102e4462c00->192.168.4.129@o2ib netid 50000: select flavor null May 17 16:35:04 client-5 kernel: Lustre: 28776:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 1 previous similar message May 17 16:35:04 client-5 xinetd[3391]: START: shell pid=28870 from=192.168.4.5 May 17 16:35:04 client-5 rshd[28871]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark ost4 now in FULL state");echo XXRETCODE:$?' May 17 16:35:04 client-5 kernel: Lustre: DEBUG MARKER: ost4 now in FULL state May 17 16:35:04 client-5 xinetd[3391]: EXIT: shell status=0 pid=28870 duration=0(sec) May 17 16:35:05 client-5 xinetd[3391]: START: shell pid=29044 from=192.168.4.5 May 17 16:35:05 client-5 rshd[29045]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark ost5 now in FULL state");echo XXRETCODE:$?' May 17 16:35:05 client-5 kernel: Lustre: DEBUG MARKER: ost5 now in FULL state May 17 16:35:05 client-5 xinetd[3391]: EXIT: shell status=0 pid=29044 duration=0(sec) May 17 16:35:06 client-5 xinetd[3391]: START: shell pid=29218 from=192.168.4.5 May 17 16:35:06 client-5 rshd[29219]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark ost6 now in FULL state");echo XXRETCODE:$?' May 17 16:35:06 client-5 kernel: Lustre: DEBUG MARKER: ost6 now in FULL state May 17 16:35:06 client-5 xinetd[3391]: EXIT: shell status=0 pid=29218 duration=0(sec) May 17 16:35:06 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 16:35:06 client-5 kernel: Lustre: 29262:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff810314d37400->192.168.4.128@o2ib netid 50000: select flavor null May 17 16:35:06 client-5 kernel: Lustre: 29262:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 4 previous similar messages May 17 16:35:06 client-5 kernel: Lustre: Client lustre-client has started May 17 16:35:06 client-5 kernel: Lustre: 29269:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:35:06 client-5 kernel: Lustre: 29269:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message May 17 16:35:07 client-5 kernel: LustreError: 29321:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:35:07 client-5 kernel: LustreError: 29321:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:35:07 client-5 kernel: Lustre: client ffff810314d37400 umount complete May 17 16:35:07 client-5 kernel: LustreError: 29340:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:35:07 client-5 kernel: LustreError: 29340:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:35:07 client-5 kernel: Lustre: client ffff8102e4462c00 umount complete May 17 16:35:33 client-5 kernel: Lustre: Acceptor stopping May 17 16:35:34 client-5 kernel: Lustre: Removed LNI 10.10.4.5@tcp May 17 16:35:36 client-5 kernel: Lustre: Removed LNI 192.168.4.5@o2ib May 17 16:35:37 client-5 kernel: Lustre: 29952:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:35:37 client-5 kernel: Lustre: 29953:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:35:37 client-5 kernel: Lustre: OBD class driver, http://www.lustre.org/ May 17 16:35:37 client-5 kernel: Lustre: Lustre Version: 2.0.60 May 17 16:35:37 client-5 kernel: Lustre: Build Version: jenkins-g985ac5b-PRISTINE-2.6.18-194.17.1.el5 May 17 16:35:37 client-5 kernel: Lustre: Lustre LU module (ffffffff888eb1c0). May 17 16:35:37 client-5 kernel: Lustre: Register global MR array, MR size: 0xffffffffffffffff, array size: 1 May 17 16:35:37 client-5 kernel: Lustre: Added LNI 192.168.4.5@o2ib [8/64/0/180] May 17 16:35:37 client-5 kernel: Lustre: Added LNI 10.10.4.5@tcp [8/256/0/180] May 17 16:35:37 client-5 kernel: Lustre: Accept all, port 7988 May 17 16:35:38 client-5 kernel: Lustre: Lustre OSC module (ffffffff88b3c6c0). May 17 16:35:38 client-5 kernel: Lustre: Lustre LOV module (ffffffff88bab540). May 17 16:35:38 client-5 kernel: Lustre: Lustre client module (ffffffff88c643a0). May 17 16:35:39 client-5 xinetd[3391]: START: shell pid=30457 from=192.168.4.5 May 17 16:35:39 client-5 rshd[30458]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 16:35:39 client-5 xinetd[3391]: EXIT: shell status=0 pid=30457 duration=0(sec) May 17 16:35:39 client-5 xinetd[3391]: START: shell pid=30629 from=192.168.4.5 May 17 16:35:39 client-5 rshd[30630]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == conf-sanity test 47: server restart does not make client loss lru_resize settings ================= 16:35:39 \(1305675339\)");echo XXRETCODE:$?' May 17 16:35:39 client-5 kernel: Lustre: DEBUG MARKER: == conf-sanity test 47: server restart does not make client loss lru_resize settings ================= 16:35:39 (1305675339) May 17 16:35:39 client-5 xinetd[3391]: EXIT: shell status=0 pid=30629 duration=0(sec) May 17 16:35:39 client-5 xinetd[3391]: START: shell pid=30672 from=192.168.4.5 May 17 16:35:39 client-5 rshd[30673]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:; lsof -t /mnt/lustre || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre is still busy, wait one second && sleep 1; umount /mnt/lustre; fi fi");echo XXRETCODE:$?' May 17 16:35:39 client-5 xinetd[3391]: EXIT: shell status=0 pid=30672 duration=0(sec) May 17 16:35:39 client-5 xinetd[3391]: START: shell pid=30716 from=192.168.4.5 May 17 16:35:39 client-5 rshd[30717]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:; lsof -t /mnt/lustre2 || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre2 | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre2 is still busy, wait one second && sleep 1; umount /mnt/lustre2; fi fi");echo XXRETCODE:$?' May 17 16:35:40 client-5 xinetd[3391]: EXIT: shell status=0 pid=30716 duration=1(sec) May 17 16:35:41 client-5 kernel: Lustre: 30919:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:35:47 client-5 kernel: Lustre: 31337:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 16:35:47 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 16:35:47 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 16:35:47 client-5 kernel: Lustre: 31337:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff8102f473ec00->192.168.4.128@o2ib netid 50000: select flavor null May 17 16:35:47 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 16:35:52 client-5 kernel: Lustre: 31337:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-OST0000-osc-ffff8102f473ec00->192.168.4.129@o2ib netid 50000: select flavor null May 17 16:35:52 client-5 kernel: Lustre: Client lustre-client has started May 17 16:35:52 client-5 kernel: Lustre: 31354:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:35:52 client-5 kernel: Lustre: 31354:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message May 17 16:36:07 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.129@o2ib. The obd_ping operation failed with -107 May 17 16:36:07 client-5 kernel: Lustre: lustre-OST0000-osc-ffff8102f473ec00: Connection to service lustre-OST0000 via nid 192.168.4.129@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 16:36:07 client-5 kernel: Lustre: lustre-OST0000-osc-ffff8102f473ec00: Connection restored to service lustre-OST0000 using nid 192.168.4.129@o2ib. May 17 16:36:09 client-5 kernel: Lustre: 30060:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369099822170147 sent from lustre-OST0000-osc-ffff8102f473ec00 to NID 192.168.4.129@o2ib has timed out for slow reply: [sent 1305675362] [real_sent 1305675362] [current 1305675369] [deadline 7s] [delay 0s] req@ffff81033f770000 x1369099822170147/t0(0) o-1->lustre-OST0000_UUID@192.168.4.129@o2ib:28/4 lens 192/192 e 0 to 1 dl 1305675369 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 16:36:14 client-5 kernel: Lustre: 30060:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369099822170148 sent from MGC192.168.4.128@o2ib to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305675367] [real_sent 1305675367] [current 1305675374] [deadline 7s] [delay 0s] req@ffff810305b03000 x1369099822170148/t0(0) o-1->MGS@MGC192.168.4.128@o2ib_0:26/25 lens 192/192 e 0 to 1 dl 1305675374 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 16:36:14 client-5 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail. May 17 16:36:14 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff8102f473ec00: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 16:36:21 client-5 kernel: Lustre: 30060:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369099822170154 sent from lustre-MDT0000-mdc-ffff8102f473ec00 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305675372] [real_sent 1305675372] [current 1305675380] [deadline 7s] [delay 1s] req@ffff810320990800 x1369099822170154/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 192/192 e 0 to 1 dl 1305675379 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 16:36:21 client-5 kernel: Lustre: 30060:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 3 previous similar messages May 17 16:36:24 client-5 kernel: Lustre: 30062:0:(import.c:529:import_select_connection()) MGC192.168.4.128@o2ib: tried all connections, increasing latency to 6s May 17 16:36:25 client-5 kernel: Lustre: 30062:0:(import.c:529:import_select_connection()) lustre-MDT0000-mdc-ffff8102f473ec00: tried all connections, increasing latency to 6s May 17 16:36:25 client-5 kernel: Lustre: 30061:0:(import.c:885:ptlrpc_connect_interpret()) MGS@MGC192.168.4.128@o2ib_0 changed server handle from 0x62e5a8d74f31f8f to 0x62e5a8d74f32117 May 17 16:36:25 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 16:36:25 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Connection restored to service MGS using nid 192.168.4.128@o2ib. May 17 16:36:25 client-5 kernel: LustreError: 31639:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:36:25 client-5 kernel: LustreError: 31639:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:36:25 client-5 kernel: Lustre: client ffff8102f473ec00 umount complete May 17 16:36:41 client-5 kernel: Lustre: Acceptor stopping May 17 16:36:42 client-5 kernel: Lustre: Removed LNI 10.10.4.5@tcp May 17 16:36:44 client-5 kernel: Lustre: Removed LNI 192.168.4.5@o2ib May 17 16:36:45 client-5 xinetd[3391]: START: shell pid=32091 from=192.168.4.5 May 17 16:36:45 client-5 rshd[32093]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 16:36:45 client-5 xinetd[3391]: EXIT: shell status=0 pid=32091 duration=0(sec) May 17 16:36:45 client-5 kernel: Lustre: 32229:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:36:45 client-5 kernel: Lustre: 32231:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:36:45 client-5 kernel: Lustre: OBD class driver, http://www.lustre.org/ May 17 16:36:45 client-5 kernel: Lustre: Lustre Version: 2.0.60 May 17 16:36:45 client-5 kernel: Lustre: Build Version: jenkins-g985ac5b-PRISTINE-2.6.18-194.17.1.el5 May 17 16:36:45 client-5 kernel: Lustre: Lustre LU module (ffffffff888eb1c0). May 17 16:36:45 client-5 kernel: Lustre: Register global MR array, MR size: 0xffffffffffffffff, array size: 1 May 17 16:36:45 client-5 kernel: Lustre: Added LNI 192.168.4.5@o2ib [8/64/0/180] May 17 16:36:45 client-5 kernel: Lustre: Added LNI 10.10.4.5@tcp [8/256/0/180] May 17 16:36:45 client-5 kernel: Lustre: Accept all, port 7988 May 17 16:36:45 client-5 kernel: Lustre: Lustre OSC module (ffffffff88b3c6c0). May 17 16:36:45 client-5 kernel: Lustre: Lustre LOV module (ffffffff88bab540). May 17 16:36:46 client-5 kernel: Lustre: Lustre client module (ffffffff88c643a0). May 17 16:36:46 client-5 xinetd[3391]: START: shell pid=32634 from=192.168.4.5 May 17 16:36:46 client-5 rshd[32636]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == conf-sanity test 48: too many acls on file ======================================================== 16:36:45 \(1305675405\)");echo XXRETCODE:$?' May 17 16:36:46 client-5 kernel: Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ======================================================== 16:36:45 (1305675405) May 17 16:36:46 client-5 xinetd[3391]: EXIT: shell status=0 pid=32634 duration=0(sec) May 17 16:36:46 client-5 xinetd[3391]: START: shell pid=32679 from=192.168.4.5 May 17 16:36:46 client-5 rshd[32680]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:; lsof -t /mnt/lustre || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre is still busy, wait one second && sleep 1; umount /mnt/lustre; fi fi");echo XXRETCODE:$?' May 17 16:36:46 client-5 xinetd[3391]: EXIT: shell status=0 pid=32679 duration=0(sec) May 17 16:36:46 client-5 xinetd[3391]: START: shell pid=32722 from=192.168.4.5 May 17 16:36:46 client-5 rshd[32724]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:; lsof -t /mnt/lustre2 || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre2 | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre2 is still busy, wait one second && sleep 1; umount /mnt/lustre2; fi fi");echo XXRETCODE:$?' May 17 16:36:46 client-5 xinetd[3391]: EXIT: shell status=0 pid=32722 duration=0(sec) May 17 16:36:47 client-5 kernel: Lustre: 468:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:36:54 client-5 kernel: Lustre: 902:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 16:36:54 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 16:36:54 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 16:36:54 client-5 kernel: Lustre: 902:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff8103131dc400->192.168.4.128@o2ib netid 50000: select flavor null May 17 16:36:54 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 16:36:59 client-5 kernel: Lustre: 902:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-OST0000-osc-ffff8103131dc400->192.168.4.129@o2ib netid 50000: select flavor null May 17 16:36:59 client-5 kernel: Lustre: Client lustre-client has started May 17 16:36:59 client-5 kernel: Lustre: 919:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:36:59 client-5 kernel: Lustre: 919:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message May 17 16:37:00 client-5 xinetd[3391]: START: shell pid=1050 from=192.168.4.5 May 17 16:37:00 client-5 rshd[1051]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:; lsof -t /mnt/lustre || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre is still busy, wait one second && sleep 1; umount /mnt/lustre; fi fi");echo XXRETCODE:$?' May 17 16:37:00 client-5 kernel: LustreError: 1092:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:37:00 client-5 kernel: LustreError: 1092:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:37:00 client-5 kernel: LustreError: 1092:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:37:00 client-5 kernel: LustreError: 1092:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:37:00 client-5 kernel: Lustre: client ffff8103131dc400 umount complete May 17 16:37:00 client-5 xinetd[3391]: EXIT: shell status=0 pid=1050 duration=0(sec) May 17 16:37:00 client-5 xinetd[3391]: START: shell pid=1101 from=192.168.4.5 May 17 16:37:00 client-5 rshd[1102]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:; lsof -t /mnt/lustre2 || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre2 | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre2 is still busy, wait one second && sleep 1; umount /mnt/lustre2; fi fi");echo XXRETCODE:$?' May 17 16:37:00 client-5 xinetd[3391]: EXIT: shell status=0 pid=1101 duration=0(sec) May 17 16:37:18 client-5 kernel: Lustre: 1320:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:37:18 client-5 kernel: Lustre: 1320:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message May 17 16:37:30 client-5 xinetd[3391]: START: shell pid=1929 from=192.168.4.5 May 17 16:37:30 client-5 rshd[1930]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark ost now in FULL state");echo XXRETCODE:$?' May 17 16:37:30 client-5 kernel: Lustre: DEBUG MARKER: ost now in FULL state May 17 16:37:30 client-5 xinetd[3391]: EXIT: shell status=0 pid=1929 duration=0(sec) May 17 16:37:42 client-5 xinetd[3391]: START: shell pid=2087 from=192.168.4.5 May 17 16:37:42 client-5 rshd[2088]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 16:37:42 client-5 xinetd[3391]: EXIT: shell status=0 pid=2087 duration=0(sec) May 17 16:37:42 client-5 xinetd[3391]: START: shell pid=2271 from=192.168.4.5 May 17 16:37:42 client-5 rshd[2274]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == conf-sanity test 49: check PARAM_SYS_LDLM_TIMEOUT option of MKFS.LUSTRE =========================== 16:37:42 \(1305675462\)");echo XXRETCODE:$?' May 17 16:37:42 client-5 kernel: Lustre: DEBUG MARKER: == conf-sanity test 49: check PARAM_SYS_LDLM_TIMEOUT option of MKFS.LUSTRE =========================== 16:37:42 (1305675462) May 17 16:37:42 client-5 xinetd[3391]: EXIT: shell status=0 pid=2271 duration=0(sec) May 17 16:37:42 client-5 xinetd[3391]: START: shell pid=2316 from=192.168.4.5 May 17 16:37:42 client-5 rshd[2317]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:; lsof -t /mnt/lustre || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre is still busy, wait one second && sleep 1; umount /mnt/lustre; fi fi");echo XXRETCODE:$?' May 17 16:37:42 client-5 xinetd[3391]: EXIT: shell status=0 pid=2316 duration=0(sec) May 17 16:37:42 client-5 xinetd[3391]: START: shell pid=2360 from=192.168.4.5 May 17 16:37:42 client-5 rshd[2361]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:; lsof -t /mnt/lustre2 || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre2 | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre2 is still busy, wait one second && sleep 1; umount /mnt/lustre2; fi fi");echo XXRETCODE:$?' May 17 16:37:42 client-5 xinetd[3391]: EXIT: shell status=0 pid=2360 duration=0(sec) May 17 16:37:44 client-5 kernel: Lustre: 2580:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:37:44 client-5 kernel: Lustre: 2580:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message May 17 16:37:50 client-5 kernel: Lustre: 3002:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 16:37:50 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 16:37:50 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 16:37:50 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 16:37:55 client-5 kernel: Lustre: 3002:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-OST0000-osc-ffff8103159bac00->192.168.4.129@o2ib netid 50000: select flavor null May 17 16:37:55 client-5 kernel: Lustre: 3002:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 1 previous similar message May 17 16:37:55 client-5 kernel: Lustre: Client lustre-client has started May 17 16:37:55 client-5 kernel: Lustre: 3021:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:37:55 client-5 kernel: Lustre: 3021:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message May 17 16:37:56 client-5 kernel: LustreError: 3091:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:37:56 client-5 kernel: LustreError: 3091:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:37:56 client-5 kernel: Lustre: client ffff8103159bac00 umount complete May 17 16:38:07 client-5 xinetd[3391]: START: shell pid=3173 from=192.168.4.5 May 17 16:38:07 client-5 rshd[3174]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:; lsof -t /mnt/lustre || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre is still busy, wait one second && sleep 1; umount /mnt/lustre; fi fi");echo XXRETCODE:$?' May 17 16:38:07 client-5 xinetd[3391]: EXIT: shell status=0 pid=3173 duration=0(sec) May 17 16:38:07 client-5 xinetd[3391]: START: shell pid=3220 from=192.168.4.5 May 17 16:38:07 client-5 rshd[3222]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:; lsof -t /mnt/lustre2 || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre2 | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre2 is still busy, wait one second && sleep 1; umount /mnt/lustre2; fi fi");echo XXRETCODE:$?' May 17 16:38:07 client-5 xinetd[3391]: EXIT: shell status=0 pid=3220 duration=0(sec) May 17 16:38:15 client-5 kernel: Lustre: 3884:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 16:38:15 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 16:38:15 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 16:38:15 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 16:38:20 client-5 kernel: Lustre: Client lustre-client has started May 17 16:38:20 client-5 kernel: Lustre: 3901:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:38:20 client-5 kernel: Lustre: 3901:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 3 previous similar messages May 17 16:38:20 client-5 kernel: LustreError: 3966:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:38:20 client-5 kernel: LustreError: 3966:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Skipped 1 previous similar message May 17 16:38:20 client-5 kernel: LustreError: 3966:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:38:20 client-5 kernel: LustreError: 3966:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) Skipped 1 previous similar message May 17 16:38:21 client-5 kernel: Lustre: client ffff81033f7c0400 umount complete May 17 16:38:38 client-5 kernel: Lustre: Acceptor stopping May 17 16:38:39 client-5 kernel: Lustre: Removed LNI 10.10.4.5@tcp May 17 16:38:41 client-5 kernel: Lustre: Removed LNI 192.168.4.5@o2ib May 17 16:38:42 client-5 xinetd[3391]: START: shell pid=4423 from=192.168.4.5 May 17 16:38:42 client-5 rshd[4424]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 16:38:42 client-5 xinetd[3391]: EXIT: shell status=0 pid=4423 duration=0(sec) May 17 16:38:43 client-5 kernel: Lustre: 4558:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:38:43 client-5 kernel: Lustre: 4560:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:38:43 client-5 kernel: Lustre: OBD class driver, http://www.lustre.org/ May 17 16:38:43 client-5 kernel: Lustre: Lustre Version: 2.0.60 May 17 16:38:43 client-5 kernel: Lustre: Build Version: jenkins-g985ac5b-PRISTINE-2.6.18-194.17.1.el5 May 17 16:38:43 client-5 kernel: Lustre: Lustre LU module (ffffffff888eb1c0). May 17 16:38:43 client-5 kernel: Lustre: Register global MR array, MR size: 0xffffffffffffffff, array size: 1 May 17 16:38:43 client-5 kernel: Lustre: Added LNI 192.168.4.5@o2ib [8/64/0/180] May 17 16:38:43 client-5 kernel: Lustre: Added LNI 10.10.4.5@tcp [8/256/0/180] May 17 16:38:43 client-5 kernel: Lustre: Accept all, port 7988 May 17 16:38:43 client-5 kernel: Lustre: Lustre OSC module (ffffffff88b3c6c0). May 17 16:38:43 client-5 kernel: Lustre: Lustre LOV module (ffffffff88bab540). May 17 16:38:43 client-5 kernel: Lustre: Lustre client module (ffffffff88c643a0). May 17 16:38:43 client-5 xinetd[3391]: START: shell pid=4964 from=192.168.4.5 May 17 16:38:43 client-5 rshd[4967]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == conf-sanity test 50a: lazystatfs all servers available ============================================ 16:38:43 \(1305675523\)");echo XXRETCODE:$?' May 17 16:38:43 client-5 kernel: Lustre: DEBUG MARKER: == conf-sanity test 50a: lazystatfs all servers available ============================================ 16:38:43 (1305675523) May 17 16:38:43 client-5 xinetd[3391]: EXIT: shell status=0 pid=4964 duration=0(sec) May 17 16:38:45 client-5 kernel: Lustre: 5093:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 16:38:45 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 16:38:45 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 16:38:45 client-5 kernel: Lustre: 5093:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff81033f76a000->192.168.4.128@o2ib netid 50000: select flavor null May 17 16:38:45 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 16:38:50 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 16:38:55 client-5 kernel: Lustre: 5093:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-OST0000-osc-ffff81033f76a000->192.168.4.129@o2ib netid 50000: select flavor null May 17 16:38:55 client-5 kernel: Lustre: Client lustre-client has started May 17 16:38:55 client-5 kernel: Lustre: 5110:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:39:00 client-5 kernel: LustreError: 5139:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:39:00 client-5 kernel: LustreError: 5139:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:39:00 client-5 kernel: LustreError: 5139:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:39:00 client-5 kernel: LustreError: 5139:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:39:00 client-5 kernel: Lustre: client ffff81033f76a000 umount complete May 17 16:39:16 client-5 kernel: Lustre: Acceptor stopping May 17 16:39:17 client-5 kernel: Lustre: Removed LNI 10.10.4.5@tcp May 17 16:39:19 client-5 kernel: Lustre: Removed LNI 192.168.4.5@o2ib May 17 16:39:20 client-5 xinetd[3391]: START: shell pid=5590 from=192.168.4.5 May 17 16:39:20 client-5 rshd[5593]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 16:39:20 client-5 xinetd[3391]: EXIT: shell status=0 pid=5590 duration=0(sec) May 17 16:39:20 client-5 kernel: Lustre: 5730:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:39:20 client-5 kernel: Lustre: 5731:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:39:20 client-5 kernel: Lustre: OBD class driver, http://www.lustre.org/ May 17 16:39:20 client-5 kernel: Lustre: Lustre Version: 2.0.60 May 17 16:39:20 client-5 kernel: Lustre: Build Version: jenkins-g985ac5b-PRISTINE-2.6.18-194.17.1.el5 May 17 16:39:20 client-5 kernel: Lustre: Lustre LU module (ffffffff888eb1c0). May 17 16:39:20 client-5 kernel: Lustre: Register global MR array, MR size: 0xffffffffffffffff, array size: 1 May 17 16:39:20 client-5 kernel: Lustre: Added LNI 192.168.4.5@o2ib [8/64/0/180] May 17 16:39:20 client-5 kernel: Lustre: Added LNI 10.10.4.5@tcp [8/256/0/180] May 17 16:39:20 client-5 kernel: Lustre: Accept all, port 7988 May 17 16:39:21 client-5 kernel: Lustre: Lustre OSC module (ffffffff88b3c6c0). May 17 16:39:21 client-5 kernel: Lustre: Lustre LOV module (ffffffff88bab540). May 17 16:39:21 client-5 kernel: Lustre: Lustre client module (ffffffff88c643a0). May 17 16:39:21 client-5 xinetd[3391]: START: shell pid=6131 from=192.168.4.5 May 17 16:39:21 client-5 rshd[6132]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == conf-sanity test 50b: lazystatfs all servers down ================================================= 16:39:20 \(1305675560\)");echo XXRETCODE:$?' May 17 16:39:21 client-5 kernel: Lustre: DEBUG MARKER: == conf-sanity test 50b: lazystatfs all servers down ================================================= 16:39:20 (1305675560) May 17 16:39:21 client-5 xinetd[3391]: EXIT: shell status=0 pid=6131 duration=0(sec) May 17 16:39:22 client-5 kernel: Lustre: 6262:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 16:39:22 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 16:39:22 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 16:39:22 client-5 kernel: Lustre: 6262:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff81032e017400->192.168.4.128@o2ib netid 50000: select flavor null May 17 16:39:22 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 16:39:27 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 16:39:32 client-5 kernel: Lustre: 6262:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-OST0000-osc-ffff81032e017400->192.168.4.129@o2ib netid 50000: select flavor null May 17 16:39:32 client-5 kernel: Lustre: Client lustre-client has started May 17 16:39:32 client-5 kernel: Lustre: 6279:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:39:49 client-5 kernel: Lustre: 5833:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369100056002590 sent from lustre-OST0000-osc-ffff81032e017400 to NID 192.168.4.129@o2ib has timed out for slow reply: [sent 1305675582] [real_sent 1305675582] [current 1305675589] [deadline 7s] [delay 0s] req@ffff8103399af800 x1369100056002590/t0(0) o-1->lustre-OST0000_UUID@192.168.4.129@o2ib:28/4 lens 192/192 e 0 to 1 dl 1305675589 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 16:39:49 client-5 kernel: Lustre: lustre-OST0000-osc-ffff81032e017400: Connection to service lustre-OST0000 via nid 192.168.4.129@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 16:39:54 client-5 kernel: Lustre: 5833:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369100056002593 sent from lustre-OST0000-osc-ffff81032e017400 to NID 192.168.4.129@o2ib has timed out for slow reply: [sent 1305675587] [real_sent 1305675587] [current 1305675594] [deadline 7s] [delay 0s] req@ffff8102e76bc800 x1369100056002593/t0(0) o-1->lustre-OST0000_UUID@192.168.4.129@o2ib:28/4 lens 192/192 e 0 to 1 dl 1305675594 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 16:39:55 client-5 kernel: Lustre: 5834:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369100056002594 sent from lustre-OST0000-osc-ffff81032e017400 to NID 192.168.4.129@o2ib has timed out for slow reply: [sent 1305675589] [real_sent 1305675589] [current 1305675595] [deadline 6s] [delay 0s] req@ffff8102e76bc000 x1369100056002594/t0(0) o-1->lustre-OST0000_UUID@192.168.4.129@o2ib:28/4 lens 368/392 e 0 to 1 dl 1305675595 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 16:39:58 client-5 kernel: LustreError: 6618:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:39:58 client-5 kernel: LustreError: 6618:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:39:58 client-5 kernel: LustreError: 6618:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:39:58 client-5 kernel: LustreError: 6618:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:39:58 client-5 kernel: Lustre: client ffff81032e017400 umount complete May 17 16:40:05 client-5 xinetd[3391]: START: shell pid=6706 from=192.168.4.5 May 17 16:40:05 client-5 rshd[6709]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 16:40:05 client-5 xinetd[3391]: EXIT: shell status=0 pid=6706 duration=0(sec) May 17 16:40:05 client-5 xinetd[3391]: START: shell pid=6880 from=192.168.4.5 May 17 16:40:05 client-5 rshd[6881]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == conf-sanity test 50c: lazystatfs one server down ================================================== 16:40:05 \(1305675605\)");echo XXRETCODE:$?' May 17 16:40:05 client-5 kernel: Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ================================================== 16:40:05 (1305675605) May 17 16:40:05 client-5 xinetd[3391]: EXIT: shell status=0 pid=6880 duration=0(sec) May 17 16:40:07 client-5 kernel: Lustre: 7047:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 16:40:07 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 16:40:07 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 16:40:07 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 16:40:12 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 16:40:17 client-5 kernel: Lustre: 7047:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-OST0000-osc-ffff81031b758800->192.168.4.129@o2ib netid 50000: select flavor null May 17 16:40:17 client-5 kernel: Lustre: 7047:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 1 previous similar message May 17 16:40:17 client-5 kernel: Lustre: Client lustre-client has started May 17 16:40:17 client-5 kernel: Lustre: 7064:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:40:17 client-5 kernel: Lustre: 7064:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message May 17 16:40:24 client-5 xinetd[3391]: START: shell pid=7277 from=192.168.4.5 May 17 16:40:24 client-5 rshd[7278]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark ost now in DISCONN state");echo XXRETCODE:$?' May 17 16:40:24 client-5 kernel: Lustre: DEBUG MARKER: ost now in DISCONN state May 17 16:40:24 client-5 xinetd[3391]: EXIT: shell status=0 pid=7277 duration=0(sec) May 17 16:40:24 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.129@o2ib. The ost_statfs operation failed with -107 May 17 16:40:24 client-5 kernel: Lustre: lustre-OST0000-osc-ffff81031b758800: Connection to service lustre-OST0000 via nid 192.168.4.129@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 16:40:29 client-5 kernel: LustreError: 7336:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:40:29 client-5 kernel: LustreError: 7336:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:40:29 client-5 kernel: Lustre: client ffff81031b758800 umount complete May 17 16:40:38 client-5 xinetd[3391]: START: shell pid=7459 from=192.168.4.5 May 17 16:40:38 client-5 rshd[7460]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 16:40:38 client-5 xinetd[3391]: EXIT: shell status=0 pid=7459 duration=0(sec) May 17 16:40:38 client-5 xinetd[3391]: START: shell pid=7631 from=192.168.4.5 May 17 16:40:38 client-5 rshd[7632]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == conf-sanity test 50d: lazystatfs client\/server conn race ========================================== 16:40:38 \(1305675638\)");echo XXRETCODE:$?' May 17 16:40:38 client-5 kernel: Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================== 16:40:38 (1305675638) May 17 16:40:38 client-5 xinetd[3391]: EXIT: shell status=0 pid=7631 duration=0(sec) May 17 16:40:40 client-5 kernel: Lustre: 7798:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 16:40:40 client-5 kernel: Lustre: 7798:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 1 previous similar message May 17 16:40:40 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 16:40:40 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 16:40:40 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 16:40:40 client-5 kernel: LustreError: Skipped 1 previous similar message May 17 16:40:50 client-5 kernel: Lustre: 7798:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-OST0000-osc-ffff810316b0cc00->192.168.4.129@o2ib netid 50000: select flavor null May 17 16:40:50 client-5 kernel: Lustre: 7798:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 1 previous similar message May 17 16:40:50 client-5 kernel: Lustre: Client lustre-client has started May 17 16:40:50 client-5 kernel: Lustre: 7815:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:40:50 client-5 kernel: Lustre: 7815:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message May 17 16:40:52 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.129@o2ib. The ost_statfs operation failed with -107 May 17 16:40:52 client-5 kernel: LustreError: Skipped 1 previous similar message May 17 16:40:52 client-5 kernel: Lustre: lustre-OST0000-osc-ffff810316b0cc00: Connection to service lustre-OST0000 via nid 192.168.4.129@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 16:40:57 client-5 kernel: LustreError: 7877:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:40:57 client-5 kernel: LustreError: 7877:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Skipped 1 previous similar message May 17 16:40:57 client-5 kernel: LustreError: 7877:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:40:57 client-5 kernel: LustreError: 7877:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) Skipped 1 previous similar message May 17 16:40:57 client-5 kernel: Lustre: client ffff810316b0cc00 umount complete May 17 16:41:06 client-5 xinetd[3391]: START: shell pid=8000 from=192.168.4.5 May 17 16:41:06 client-5 rshd[8001]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 16:41:06 client-5 xinetd[3391]: EXIT: shell status=0 pid=8000 duration=0(sec) May 17 16:41:07 client-5 xinetd[3391]: START: shell pid=8176 from=192.168.4.5 May 17 16:41:07 client-5 rshd[8177]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == conf-sanity test 50e: normal statfs all servers down ============================================== 16:41:06 \(1305675666\)");echo XXRETCODE:$?' May 17 16:41:07 client-5 kernel: Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down ============================================== 16:41:06 (1305675666) May 17 16:41:07 client-5 xinetd[3391]: EXIT: shell status=0 pid=8176 duration=0(sec) May 17 16:41:07 client-5 xinetd[3391]: START: shell pid=8219 from=192.168.4.5 May 17 16:41:07 client-5 rshd[8220]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:; lsof -t /mnt/lustre || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre is still busy, wait one second && sleep 1; umount /mnt/lustre; fi fi");echo XXRETCODE:$?' May 17 16:41:07 client-5 xinetd[3391]: EXIT: shell status=0 pid=8219 duration=0(sec) May 17 16:41:07 client-5 xinetd[3391]: START: shell pid=8263 from=192.168.4.5 May 17 16:41:07 client-5 rshd[8264]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:; lsof -t /mnt/lustre2 || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre2 | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre2 is still busy, wait one second && sleep 1; umount /mnt/lustre2; fi fi");echo XXRETCODE:$?' May 17 16:41:07 client-5 xinetd[3391]: EXIT: shell status=0 pid=8263 duration=0(sec) May 17 16:41:08 client-5 kernel: Lustre: 8466:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:41:08 client-5 kernel: Lustre: 8466:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message May 17 16:41:20 client-5 xinetd[3391]: START: shell pid=9062 from=192.168.4.5 May 17 16:41:20 client-5 rshd[9063]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark ost now in FULL state");echo XXRETCODE:$?' May 17 16:41:20 client-5 kernel: Lustre: DEBUG MARKER: ost now in FULL state May 17 16:41:20 client-5 xinetd[3391]: EXIT: shell status=0 pid=9062 duration=0(sec) May 17 16:41:42 client-5 xinetd[3391]: START: shell pid=9481 from=192.168.4.5 May 17 16:41:42 client-5 rshd[9482]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark ost now in FULL state");echo XXRETCODE:$?' May 17 16:41:42 client-5 kernel: Lustre: DEBUG MARKER: ost now in FULL state May 17 16:41:42 client-5 xinetd[3391]: EXIT: shell status=0 pid=9481 duration=0(sec) May 17 16:41:48 client-5 xinetd[3391]: START: shell pid=9686 from=192.168.4.5 May 17 16:41:48 client-5 rshd[9689]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark ost now in DISCONN state");echo XXRETCODE:$?' May 17 16:41:48 client-5 kernel: Lustre: DEBUG MARKER: ost now in DISCONN state May 17 16:41:48 client-5 xinetd[3391]: EXIT: shell status=0 pid=9686 duration=0(sec) May 17 16:41:48 client-5 kernel: Lustre: 9732:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 16:41:48 client-5 kernel: Lustre: 9732:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 1 previous similar message May 17 16:41:48 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 16:41:48 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 16:41:48 client-5 kernel: Lustre: Client lustre-client has started May 17 16:41:48 client-5 kernel: Lustre: 9749:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:41:48 client-5 kernel: Lustre: 9749:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message May 17 16:41:53 client-5 kernel: Lustre: 5834:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369100056002693 sent from lustre-OST0000-osc-ffff8102eff76800 to NID 192.168.4.129@o2ib has timed out for slow reply: [sent 1305675708] [real_sent 1305675708] [current 1305675713] [deadline 5s] [delay 0s] req@ffff81033eb8bc00 x1369100056002693/t0(0) o-1->lustre-OST0000_UUID@192.168.4.129@o2ib:28/4 lens 368/392 e 0 to 1 dl 1305675713 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 16:41:58 client-5 kernel: Lustre: 5835:0:(import.c:529:import_select_connection()) lustre-OST0000-osc-ffff8102eff76800: tried all connections, increasing latency to 5s May 17 16:42:09 client-5 kernel: Lustre: 5834:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369100056002702 sent from lustre-OST0000-osc-ffff8102eff76800 to NID 192.168.4.129@o2ib has timed out for slow reply: [sent 1305675718] [real_sent 1305675718] [current 1305675728] [deadline 10s] [delay 0s] req@ffff8102e7caf400 x1369100056002702/t0(0) o-1->lustre-OST0000_UUID@192.168.4.129@o2ib:28/4 lens 368/392 e 0 to 1 dl 1305675728 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 16:42:13 client-5 kernel: Lustre: 5835:0:(import.c:529:import_select_connection()) lustre-OST0000-osc-ffff8102eff76800: tried all connections, increasing latency to 10s May 17 16:42:14 client-5 kernel: LustreError: 9815:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:42:14 client-5 kernel: LustreError: 9815:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Skipped 1 previous similar message May 17 16:42:14 client-5 kernel: LustreError: 9815:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:42:14 client-5 kernel: LustreError: 9815:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) Skipped 1 previous similar message May 17 16:42:14 client-5 kernel: Lustre: client ffff8102eff76800 umount complete May 17 16:42:26 client-5 xinetd[3391]: START: shell pid=9942 from=192.168.4.5 May 17 16:42:26 client-5 rshd[9943]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 16:42:26 client-5 xinetd[3391]: EXIT: shell status=0 pid=9942 duration=0(sec) May 17 16:42:26 client-5 xinetd[3391]: START: shell pid=10114 from=192.168.4.5 May 17 16:42:26 client-5 rshd[10115]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == conf-sanity test 50f: normal statfs one server in down ============================================ 16:42:26 \(1305675746\)");echo XXRETCODE:$?' May 17 16:42:26 client-5 kernel: Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ============================================ 16:42:26 (1305675746) May 17 16:42:26 client-5 xinetd[3391]: EXIT: shell status=0 pid=10114 duration=0(sec) May 17 16:42:37 client-5 xinetd[3391]: START: shell pid=10467 from=192.168.4.5 May 17 16:42:37 client-5 rshd[10468]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark ost now in FULL state");echo XXRETCODE:$?' May 17 16:42:37 client-5 kernel: Lustre: DEBUG MARKER: ost now in FULL state May 17 16:42:37 client-5 xinetd[3391]: EXIT: shell status=0 pid=10467 duration=0(sec) May 17 16:42:43 client-5 xinetd[3391]: START: shell pid=10729 from=192.168.4.5 May 17 16:42:43 client-5 rshd[10730]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark ost2 now in FULL state");echo XXRETCODE:$?' May 17 16:42:43 client-5 kernel: Lustre: DEBUG MARKER: ost2 now in FULL state May 17 16:42:43 client-5 xinetd[3391]: EXIT: shell status=0 pid=10729 duration=0(sec) May 17 16:42:48 client-5 xinetd[3391]: START: shell pid=10936 from=192.168.4.5 May 17 16:42:48 client-5 rshd[10937]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark ost2 now in DISCONN state");echo XXRETCODE:$?' May 17 16:42:48 client-5 kernel: Lustre: DEBUG MARKER: ost2 now in DISCONN state May 17 16:42:48 client-5 xinetd[3391]: EXIT: shell status=0 pid=10936 duration=0(sec) May 17 16:42:48 client-5 kernel: Lustre: 10980:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 16:42:48 client-5 kernel: Lustre: 10980:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 2 previous similar messages May 17 16:42:49 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 16:42:49 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 16:42:49 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.129@o2ib. The ost_connect operation failed with -19 May 17 16:42:49 client-5 kernel: LustreError: Skipped 1 previous similar message May 17 16:42:49 client-5 kernel: Lustre: Client lustre-client has started May 17 16:42:49 client-5 kernel: Lustre: 10997:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:42:49 client-5 kernel: Lustre: 10997:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message May 17 16:42:54 client-5 kernel: Lustre: 5835:0:(import.c:529:import_select_connection()) lustre-OST0001-osc-ffff81031ab0dc00: tried all connections, increasing latency to 6s May 17 16:43:04 client-5 kernel: Lustre: 5835:0:(import.c:529:import_select_connection()) lustre-OST0001-osc-ffff81031ab0dc00: tried all connections, increasing latency to 11s May 17 16:43:14 client-5 kernel: Lustre: 5835:0:(import.c:529:import_select_connection()) lustre-OST0001-osc-ffff81031ab0dc00: tried all connections, increasing latency to 16s May 17 16:43:18 client-5 kernel: LustreError: 11096:0:(lov_obd.c:334:lov_disconnect_obd()) Target lustre-OST0001_UUID disconnect error -107 May 17 16:43:18 client-5 kernel: LustreError: 11096:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:43:18 client-5 kernel: LustreError: 11096:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Skipped 1 previous similar message May 17 16:43:18 client-5 kernel: LustreError: 11096:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:43:18 client-5 kernel: LustreError: 11096:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) Skipped 1 previous similar message May 17 16:43:18 client-5 kernel: Lustre: client ffff81031ab0dc00 umount complete May 17 16:43:27 client-5 kernel: Lustre: 11166:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:43:27 client-5 kernel: Lustre: 11166:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message May 17 16:43:29 client-5 xinetd[3391]: START: shell pid=11375 from=192.168.4.5 May 17 16:43:29 client-5 rshd[11377]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 16:43:29 client-5 xinetd[3391]: EXIT: shell status=0 pid=11375 duration=0(sec) May 17 16:43:29 client-5 xinetd[3391]: START: shell pid=11548 from=192.168.4.5 May 17 16:43:29 client-5 rshd[11549]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == conf-sanity test 50g: deactivated OST should not cause panic======================================= 16:43:29 \(1305675809\)");echo XXRETCODE:$?' May 17 16:43:29 client-5 kernel: Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic======================================= 16:43:29 (1305675809) May 17 16:43:29 client-5 xinetd[3391]: EXIT: shell status=0 pid=11548 duration=0(sec) May 17 16:43:30 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 16:43:30 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 16:43:30 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 16:43:30 client-5 kernel: LustreError: Skipped 3 previous similar messages May 17 16:43:35 client-5 kernel: Lustre: Client lustre-client has started May 17 16:43:41 client-5 xinetd[3391]: START: shell pid=11930 from=192.168.4.5 May 17 16:43:41 client-5 rshd[11931]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark ost2 now in FULL state");echo XXRETCODE:$?' May 17 16:43:41 client-5 kernel: Lustre: DEBUG MARKER: ost2 now in FULL state May 17 16:43:41 client-5 xinetd[3391]: EXIT: shell status=0 pid=11930 duration=0(sec) May 17 16:43:41 client-5 xinetd[3391]: START: shell pid=12034 from=192.168.4.5 May 17 16:43:41 client-5 rshd[12035]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark ost2 now in FULL state");echo XXRETCODE:$?' May 17 16:43:41 client-5 kernel: Lustre: DEBUG MARKER: ost2 now in FULL state May 17 16:43:41 client-5 xinetd[3391]: EXIT: shell status=0 pid=12034 duration=0(sec) May 17 16:43:42 client-5 kernel: LustreError: 12096:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:43:42 client-5 kernel: LustreError: 12096:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Skipped 1 previous similar message May 17 16:43:42 client-5 kernel: LustreError: 12096:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:43:42 client-5 kernel: LustreError: 12096:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) Skipped 1 previous similar message May 17 16:43:42 client-5 kernel: Lustre: client ffff810313d92000 umount complete May 17 16:43:42 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 16:43:42 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 16:43:42 client-5 kernel: Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request May 17 16:43:42 client-5 kernel: LustreError: 12137:0:(obd_class.h:1502:obd_find_cbdata()) obd_find_cbdata: NULL export May 17 16:43:42 client-5 kernel: LustreError: 5788:0:(lov_obd.c:924:lov_cleanup()) lov tgt 1 not cleaned! deathrow=0, lovrc=1 May 17 16:43:56 client-5 xinetd[3391]: START: shell pid=12468 from=192.168.4.5 May 17 16:43:56 client-5 rshd[12470]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 16:43:56 client-5 xinetd[3391]: EXIT: shell status=0 pid=12468 duration=0(sec) May 17 16:43:57 client-5 xinetd[3391]: START: shell pid=12641 from=192.168.4.5 May 17 16:43:57 client-5 rshd[12642]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ====== 16:43:57 \(1305675837\)");echo XXRETCODE:$?' May 17 16:43:57 client-5 kernel: Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ====== 16:43:57 (1305675837) May 17 16:43:57 client-5 xinetd[3391]: EXIT: shell status=0 pid=12641 duration=0(sec) May 17 16:43:57 client-5 xinetd[3391]: START: shell pid=12684 from=192.168.4.5 May 17 16:43:57 client-5 rshd[12685]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:; lsof -t /mnt/lustre || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre is still busy, wait one second && sleep 1; umount /mnt/lustre; fi fi");echo XXRETCODE:$?' May 17 16:43:57 client-5 xinetd[3391]: EXIT: shell status=0 pid=12684 duration=0(sec) May 17 16:43:57 client-5 xinetd[3391]: START: shell pid=12728 from=192.168.4.5 May 17 16:43:57 client-5 rshd[12729]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:; lsof -t /mnt/lustre2 || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre2 | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre2 is still busy, wait one second && sleep 1; umount /mnt/lustre2; fi fi");echo XXRETCODE:$?' May 17 16:43:57 client-5 xinetd[3391]: EXIT: shell status=0 pid=12728 duration=0(sec) May 17 16:44:05 client-5 kernel: Lustre: 13349:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 16:44:05 client-5 kernel: Lustre: 13349:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 10 previous similar messages May 17 16:44:05 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 16:44:05 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 16:44:10 client-5 kernel: Lustre: Client lustre-client has started May 17 16:44:10 client-5 kernel: Lustre: Skipped 1 previous similar message May 17 16:44:38 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.129@o2ib. The obd_ping operation failed with -107 May 17 16:44:38 client-5 kernel: LustreError: Skipped 1 previous similar message May 17 16:44:38 client-5 kernel: Lustre: lustre-OST0001-osc-ffff8102ff802800: Connection to service lustre-OST0001 via nid 192.168.4.129@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 16:44:38 client-5 kernel: LustreError: 13495:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:44:38 client-5 kernel: LustreError: 13495:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Skipped 1 previous similar message May 17 16:44:38 client-5 kernel: LustreError: 13495:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:44:38 client-5 kernel: LustreError: 13495:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) Skipped 1 previous similar message May 17 16:44:38 client-5 kernel: Lustre: client ffff8102ff802800 umount complete May 17 16:44:38 client-5 kernel: Lustre: Skipped 1 previous similar message May 17 16:44:53 client-5 kernel: Lustre: Acceptor stopping May 17 16:44:54 client-5 kernel: Lustre: Removed LNI 10.10.4.5@tcp May 17 16:44:56 client-5 kernel: Lustre: Removed LNI 192.168.4.5@o2ib May 17 16:44:57 client-5 xinetd[3391]: START: shell pid=13951 from=192.168.4.5 May 17 16:44:57 client-5 rshd[13953]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 16:44:57 client-5 xinetd[3391]: EXIT: shell status=0 pid=13951 duration=0(sec) May 17 16:44:57 client-5 kernel: Lustre: 14088:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:44:57 client-5 kernel: Lustre: 14089:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:44:57 client-5 kernel: Lustre: OBD class driver, http://www.lustre.org/ May 17 16:44:57 client-5 kernel: Lustre: Lustre Version: 2.0.60 May 17 16:44:57 client-5 kernel: Lustre: Build Version: jenkins-g985ac5b-PRISTINE-2.6.18-194.17.1.el5 May 17 16:44:57 client-5 kernel: Lustre: Lustre LU module (ffffffff888eb1c0). May 17 16:44:57 client-5 kernel: Lustre: Register global MR array, MR size: 0xffffffffffffffff, array size: 1 May 17 16:44:57 client-5 kernel: Lustre: Added LNI 192.168.4.5@o2ib [8/64/0/180] May 17 16:44:57 client-5 kernel: Lustre: Added LNI 10.10.4.5@tcp [8/256/0/180] May 17 16:44:57 client-5 kernel: Lustre: Accept all, port 7988 May 17 16:44:58 client-5 kernel: Lustre: Lustre OSC module (ffffffff88b3c6c0). May 17 16:44:58 client-5 kernel: Lustre: Lustre LOV module (ffffffff88bab540). May 17 16:44:58 client-5 kernel: Lustre: Lustre client module (ffffffff88c643a0). May 17 16:44:58 client-5 xinetd[3391]: START: shell pid=14488 from=192.168.4.5 May 17 16:44:58 client-5 rshd[14491]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == conf-sanity test 52: check recovering objects from lost+found ===================================== 16:44:57 \(1305675897\)");echo XXRETCODE:$?' May 17 16:44:58 client-5 kernel: Lustre: DEBUG MARKER: == conf-sanity test 52: check recovering objects from lost+found ===================================== 16:44:57 (1305675897) May 17 16:44:58 client-5 xinetd[3391]: EXIT: shell status=0 pid=14488 duration=0(sec) May 17 16:44:59 client-5 kernel: Lustre: 14622:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 16:44:59 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 16:44:59 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 16:44:59 client-5 kernel: Lustre: 14622:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff8102eb5a4000->192.168.4.128@o2ib netid 50000: select flavor null May 17 16:44:59 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 16:45:04 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 16:45:09 client-5 kernel: Lustre: 14622:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-OST0000-osc-ffff8102eb5a4000->192.168.4.129@o2ib netid 50000: select flavor null May 17 16:45:09 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.129@o2ib. The ost_connect operation failed with -19 May 17 16:45:09 client-5 kernel: Lustre: Client lustre-client has started May 17 16:45:09 client-5 kernel: Lustre: 14639:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:45:14 client-5 kernel: LustreError: 14698:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:45:14 client-5 kernel: LustreError: 14698:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:45:14 client-5 kernel: LustreError: 14698:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:45:14 client-5 kernel: LustreError: 14698:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:45:14 client-5 kernel: Lustre: client ffff8102eb5a4000 umount complete May 17 16:45:18 client-5 kernel: Lustre: 14852:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 16:45:18 client-5 kernel: Lustre: 14852:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 1 previous similar message May 17 16:45:18 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 16:45:18 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 16:45:18 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.129@o2ib. The ost_connect operation failed with -19 May 17 16:45:18 client-5 kernel: Lustre: Client lustre-client has started May 17 16:45:19 client-5 kernel: Lustre: 14869:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:45:19 client-5 kernel: Lustre: 14869:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message May 17 16:45:22 client-5 kernel: LustreError: 14924:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:45:22 client-5 kernel: LustreError: 14924:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Skipped 10 previous similar messages May 17 16:45:22 client-5 kernel: LustreError: 14924:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:45:22 client-5 kernel: LustreError: 14924:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) Skipped 10 previous similar messages May 17 16:45:22 client-5 kernel: Lustre: client ffff81030f420400 umount complete May 17 16:45:36 client-5 kernel: Lustre: Acceptor stopping May 17 16:45:37 client-5 kernel: Lustre: Removed LNI 10.10.4.5@tcp May 17 16:45:39 client-5 kernel: Lustre: Removed LNI 192.168.4.5@o2ib May 17 16:45:41 client-5 xinetd[3391]: START: shell pid=15375 from=192.168.4.5 May 17 16:45:41 client-5 rshd[15377]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 16:45:41 client-5 xinetd[3391]: EXIT: shell status=0 pid=15375 duration=0(sec) May 17 16:45:41 client-5 kernel: Lustre: 15512:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:45:41 client-5 kernel: Lustre: 15513:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:45:41 client-5 kernel: Lustre: OBD class driver, http://www.lustre.org/ May 17 16:45:41 client-5 kernel: Lustre: Lustre Version: 2.0.60 May 17 16:45:41 client-5 kernel: Lustre: Build Version: jenkins-g985ac5b-PRISTINE-2.6.18-194.17.1.el5 May 17 16:45:41 client-5 kernel: Lustre: Lustre LU module (ffffffff888eb1c0). May 17 16:45:41 client-5 kernel: Lustre: Register global MR array, MR size: 0xffffffffffffffff, array size: 1 May 17 16:45:41 client-5 kernel: Lustre: Added LNI 192.168.4.5@o2ib [8/64/0/180] May 17 16:45:41 client-5 kernel: Lustre: Added LNI 10.10.4.5@tcp [8/256/0/180] May 17 16:45:41 client-5 kernel: Lustre: Accept all, port 7988 May 17 16:45:41 client-5 kernel: Lustre: Lustre OSC module (ffffffff88b3c6c0). May 17 16:45:42 client-5 kernel: Lustre: Lustre LOV module (ffffffff88bab540). May 17 16:45:42 client-5 kernel: Lustre: Lustre client module (ffffffff88c643a0). May 17 16:45:42 client-5 xinetd[3391]: START: shell pid=15918 from=192.168.4.5 May 17 16:45:42 client-5 rshd[15919]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == conf-sanity test 53a: check OSS thread count params =============================================== 16:45:41 \(1305675941\)");echo XXRETCODE:$?' May 17 16:45:42 client-5 kernel: Lustre: DEBUG MARKER: == conf-sanity test 53a: check OSS thread count params =============================================== 16:45:41 (1305675941) May 17 16:45:42 client-5 xinetd[3391]: EXIT: shell status=0 pid=15918 duration=0(sec) May 17 16:45:43 client-5 kernel: Lustre: 16046:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 16:45:43 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 16:45:43 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 16:45:43 client-5 kernel: Lustre: 16046:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff81032b300000->192.168.4.128@o2ib netid 50000: select flavor null May 17 16:45:43 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 16:45:48 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 16:45:53 client-5 kernel: Lustre: 16046:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-OST0000-osc-ffff81032b300000->192.168.4.129@o2ib netid 50000: select flavor null May 17 16:45:53 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.129@o2ib. The ost_connect operation failed with -19 May 17 16:45:53 client-5 kernel: Lustre: Client lustre-client has started May 17 16:45:53 client-5 kernel: Lustre: 16063:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:45:55 client-5 kernel: LustreError: 16241:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:45:55 client-5 kernel: LustreError: 16241:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:45:55 client-5 kernel: LustreError: 16241:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:45:55 client-5 kernel: LustreError: 16241:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:45:55 client-5 kernel: Lustre: client ffff81032b300000 umount complete May 17 16:46:13 client-5 kernel: Lustre: Acceptor stopping May 17 16:46:14 client-5 kernel: Lustre: Removed LNI 10.10.4.5@tcp May 17 16:46:16 client-5 kernel: Lustre: Removed LNI 192.168.4.5@o2ib May 17 16:46:30 client-5 kernel: Lustre: 16726:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:46:30 client-5 kernel: Lustre: 16730:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:46:30 client-5 kernel: Lustre: OBD class driver, http://www.lustre.org/ May 17 16:46:30 client-5 kernel: Lustre: Lustre Version: 2.0.60 May 17 16:46:30 client-5 kernel: Lustre: Build Version: jenkins-g985ac5b-PRISTINE-2.6.18-194.17.1.el5 May 17 16:46:30 client-5 kernel: Lustre: Lustre LU module (ffffffff888eb1c0). May 17 16:46:30 client-5 kernel: Lustre: Register global MR array, MR size: 0xffffffffffffffff, array size: 1 May 17 16:46:30 client-5 kernel: Lustre: Added LNI 192.168.4.5@o2ib [8/64/0/180] May 17 16:46:30 client-5 kernel: Lustre: Added LNI 10.10.4.5@tcp [8/256/0/180] May 17 16:46:30 client-5 kernel: Lustre: Accept all, port 7988 May 17 16:46:30 client-5 kernel: Lustre: Lustre OSC module (ffffffff88b3c6c0). May 17 16:46:30 client-5 kernel: Lustre: Lustre LOV module (ffffffff88bab540). May 17 16:46:30 client-5 kernel: Lustre: Lustre client module (ffffffff88c643a0). May 17 16:46:34 client-5 kernel: Lustre: 17228:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 16:46:34 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 16:46:34 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 16:46:34 client-5 kernel: Lustre: 17228:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff8102f99c7c00->192.168.4.128@o2ib netid 50000: select flavor null May 17 16:46:34 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 16:46:54 client-5 last message repeated 4 times May 17 16:46:59 client-5 kernel: Lustre: 17228:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-OST0000-osc-ffff8102f99c7c00->192.168.4.129@o2ib netid 50000: select flavor null May 17 16:46:59 client-5 kernel: Lustre: Client lustre-client has started May 17 16:46:59 client-5 kernel: Lustre: 17245:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:46:59 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.129@o2ib. The ost_connect operation failed with -19 May 17 16:47:00 client-5 kernel: LustreError: 17313:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:47:00 client-5 kernel: LustreError: 17313:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:47:00 client-5 kernel: LustreError: 17313:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:47:00 client-5 kernel: LustreError: 17313:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:47:00 client-5 kernel: Lustre: client ffff8102f99c7c00 umount complete May 17 16:47:21 client-5 kernel: Lustre: Acceptor stopping May 17 16:47:22 client-5 kernel: Lustre: Removed LNI 10.10.4.5@tcp May 17 16:47:24 client-5 kernel: Lustre: Removed LNI 192.168.4.5@o2ib May 17 16:47:37 client-5 kernel: Lustre: 17799:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:47:37 client-5 kernel: Lustre: 17800:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:47:37 client-5 kernel: Lustre: OBD class driver, http://www.lustre.org/ May 17 16:47:37 client-5 kernel: Lustre: Lustre Version: 2.0.60 May 17 16:47:37 client-5 kernel: Lustre: Build Version: jenkins-g985ac5b-PRISTINE-2.6.18-194.17.1.el5 May 17 16:47:37 client-5 kernel: Lustre: Lustre LU module (ffffffff888eb1c0). May 17 16:47:37 client-5 kernel: Lustre: Register global MR array, MR size: 0xffffffffffffffff, array size: 1 May 17 16:47:37 client-5 kernel: Lustre: Added LNI 192.168.4.5@o2ib [8/64/0/180] May 17 16:47:37 client-5 kernel: Lustre: Added LNI 10.10.4.5@tcp [8/256/0/180] May 17 16:47:37 client-5 kernel: Lustre: Accept all, port 7988 May 17 16:47:37 client-5 kernel: Lustre: Lustre OSC module (ffffffff88b3c6c0). May 17 16:47:37 client-5 kernel: Lustre: Lustre LOV module (ffffffff88bab540). May 17 16:47:38 client-5 kernel: Lustre: Lustre client module (ffffffff88c643a0). May 17 16:48:01 client-5 kernel: Lustre: 18242:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 16:48:01 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 16:48:01 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 16:48:01 client-5 kernel: Lustre: 18242:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff81031f09e800->192.168.4.128@o2ib netid 50000: select flavor null May 17 16:48:01 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 16:48:06 client-5 kernel: Lustre: 18242:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-OST0000-osc-ffff81031f09e800->192.168.4.129@o2ib netid 50000: select flavor null May 17 16:48:06 client-5 kernel: Lustre: Client lustre-client has started May 17 16:48:06 client-5 kernel: Lustre: 18259:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:48:06 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.129@o2ib. The ost_connect operation failed with -19 May 17 16:48:06 client-5 kernel: LustreError: 18280:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:48:06 client-5 kernel: LustreError: 18280:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:48:06 client-5 kernel: LustreError: 18280:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:48:06 client-5 kernel: LustreError: 18280:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:48:06 client-5 kernel: Lustre: client ffff81031f09e800 umount complete May 17 16:48:24 client-5 kernel: Lustre: Acceptor stopping May 17 16:48:25 client-5 kernel: Lustre: Removed LNI 10.10.4.5@tcp May 17 16:48:27 client-5 kernel: Lustre: Removed LNI 192.168.4.5@o2ib May 17 16:48:28 client-5 xinetd[3391]: START: shell pid=18733 from=192.168.4.5 May 17 16:48:28 client-5 rshd[18734]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 16:48:28 client-5 xinetd[3391]: EXIT: shell status=0 pid=18733 duration=0(sec) May 17 16:48:29 client-5 kernel: Lustre: 18870:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:48:29 client-5 kernel: Lustre: 18871:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:48:29 client-5 kernel: Lustre: OBD class driver, http://www.lustre.org/ May 17 16:48:29 client-5 kernel: Lustre: Lustre Version: 2.0.60 May 17 16:48:29 client-5 kernel: Lustre: Build Version: jenkins-g985ac5b-PRISTINE-2.6.18-194.17.1.el5 May 17 16:48:29 client-5 kernel: Lustre: Lustre LU module (ffffffff888eb1c0). May 17 16:48:29 client-5 kernel: Lustre: Register global MR array, MR size: 0xffffffffffffffff, array size: 1 May 17 16:48:29 client-5 kernel: Lustre: Added LNI 192.168.4.5@o2ib [8/64/0/180] May 17 16:48:29 client-5 kernel: Lustre: Added LNI 10.10.4.5@tcp [8/256/0/180] May 17 16:48:29 client-5 kernel: Lustre: Accept all, port 7988 May 17 16:48:29 client-5 kernel: Lustre: Lustre OSC module (ffffffff88b3c6c0). May 17 16:48:29 client-5 kernel: Lustre: Lustre LOV module (ffffffff88bab540). May 17 16:48:29 client-5 kernel: Lustre: Lustre client module (ffffffff88c643a0). May 17 16:48:29 client-5 xinetd[3391]: START: shell pid=19274 from=192.168.4.5 May 17 16:48:29 client-5 rshd[19275]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == conf-sanity test 53b: check MDT thread count params =============================================== 16:48:29 \(1305676109\)");echo XXRETCODE:$?' May 17 16:48:29 client-5 kernel: Lustre: DEBUG MARKER: == conf-sanity test 53b: check MDT thread count params =============================================== 16:48:29 (1305676109) May 17 16:48:29 client-5 xinetd[3391]: EXIT: shell status=0 pid=19274 duration=0(sec) May 17 16:48:31 client-5 kernel: Lustre: 19403:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 16:48:31 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 16:48:31 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 16:48:31 client-5 kernel: Lustre: 19403:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff81033f40cc00->192.168.4.128@o2ib netid 50000: select flavor null May 17 16:48:31 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 16:48:36 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 16:48:41 client-5 kernel: Lustre: 19403:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-OST0000-osc-ffff81033f40cc00->192.168.4.129@o2ib netid 50000: select flavor null May 17 16:48:41 client-5 kernel: Lustre: Client lustre-client has started May 17 16:48:41 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.129@o2ib. The ost_connect operation failed with -19 May 17 16:48:41 client-5 kernel: Lustre: 19424:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:48:43 client-5 kernel: LustreError: 19602:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:48:43 client-5 kernel: LustreError: 19602:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:48:43 client-5 kernel: LustreError: 19602:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:48:43 client-5 kernel: LustreError: 19602:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:48:43 client-5 kernel: Lustre: client ffff81033f40cc00 umount complete May 17 16:49:01 client-5 kernel: Lustre: Acceptor stopping May 17 16:49:02 client-5 kernel: Lustre: Removed LNI 10.10.4.5@tcp May 17 16:49:04 client-5 kernel: Lustre: Removed LNI 192.168.4.5@o2ib May 17 16:49:16 client-5 kernel: Lustre: 20083:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:49:16 client-5 kernel: Lustre: 20086:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:49:16 client-5 kernel: Lustre: OBD class driver, http://www.lustre.org/ May 17 16:49:16 client-5 kernel: Lustre: Lustre Version: 2.0.60 May 17 16:49:16 client-5 kernel: Lustre: Build Version: jenkins-g985ac5b-PRISTINE-2.6.18-194.17.1.el5 May 17 16:49:16 client-5 kernel: Lustre: Lustre LU module (ffffffff888eb1c0). May 17 16:49:16 client-5 kernel: Lustre: Register global MR array, MR size: 0xffffffffffffffff, array size: 1 May 17 16:49:16 client-5 kernel: Lustre: Added LNI 192.168.4.5@o2ib [8/64/0/180] May 17 16:49:16 client-5 kernel: Lustre: Added LNI 10.10.4.5@tcp [8/256/0/180] May 17 16:49:16 client-5 kernel: Lustre: Accept all, port 7988 May 17 16:49:16 client-5 kernel: Lustre: Lustre OSC module (ffffffff88b3c6c0). May 17 16:49:16 client-5 kernel: Lustre: Lustre LOV module (ffffffff88bab540). May 17 16:49:16 client-5 kernel: Lustre: Lustre client module (ffffffff88c643a0). May 17 16:49:19 client-5 kernel: Lustre: 20585:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 16:49:19 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 16:49:19 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 16:49:19 client-5 kernel: Lustre: 20585:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff810329f77c00->192.168.4.128@o2ib netid 50000: select flavor null May 17 16:49:19 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 16:49:39 client-5 last message repeated 4 times May 17 16:49:44 client-5 kernel: Lustre: 20585:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-OST0000-osc-ffff810329f77c00->192.168.4.129@o2ib netid 50000: select flavor null May 17 16:49:44 client-5 kernel: Lustre: Client lustre-client has started May 17 16:49:44 client-5 kernel: Lustre: 20602:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:49:44 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.129@o2ib. The ost_connect operation failed with -19 May 17 16:49:45 client-5 kernel: LustreError: 20670:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:49:45 client-5 kernel: LustreError: 20670:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:49:45 client-5 kernel: LustreError: 20670:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:49:45 client-5 kernel: LustreError: 20670:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:49:45 client-5 kernel: Lustre: client ffff810329f77c00 umount complete May 17 16:50:06 client-5 kernel: Lustre: Acceptor stopping May 17 16:50:07 client-5 kernel: Lustre: Removed LNI 10.10.4.5@tcp May 17 16:50:09 client-5 kernel: Lustre: Removed LNI 192.168.4.5@o2ib May 17 16:50:21 client-5 kernel: Lustre: 21177:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:50:21 client-5 kernel: Lustre: 21179:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:50:21 client-5 kernel: Lustre: OBD class driver, http://www.lustre.org/ May 17 16:50:21 client-5 kernel: Lustre: Lustre Version: 2.0.60 May 17 16:50:21 client-5 kernel: Lustre: Build Version: jenkins-g985ac5b-PRISTINE-2.6.18-194.17.1.el5 May 17 16:50:21 client-5 kernel: Lustre: Lustre LU module (ffffffff888eb1c0). May 17 16:50:21 client-5 kernel: Lustre: Register global MR array, MR size: 0xffffffffffffffff, array size: 1 May 17 16:50:21 client-5 kernel: Lustre: Added LNI 192.168.4.5@o2ib [8/64/0/180] May 17 16:50:21 client-5 kernel: Lustre: Added LNI 10.10.4.5@tcp [8/256/0/180] May 17 16:50:21 client-5 kernel: Lustre: Accept all, port 7988 May 17 16:50:22 client-5 kernel: Lustre: Lustre OSC module (ffffffff88b3c6c0). May 17 16:50:22 client-5 kernel: Lustre: Lustre LOV module (ffffffff88bab540). May 17 16:50:22 client-5 kernel: Lustre: Lustre client module (ffffffff88c643a0). May 17 16:50:45 client-5 kernel: Lustre: 21621:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 16:50:45 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 16:50:45 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 16:50:45 client-5 kernel: Lustre: 21621:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff81010f95cc00->192.168.4.128@o2ib netid 50000: select flavor null May 17 16:50:45 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 16:50:50 client-5 kernel: Lustre: 21621:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-OST0000-osc-ffff81010f95cc00->192.168.4.129@o2ib netid 50000: select flavor null May 17 16:50:50 client-5 kernel: Lustre: Client lustre-client has started May 17 16:50:50 client-5 kernel: Lustre: 21638:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:50:50 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.129@o2ib. The ost_connect operation failed with -19 May 17 16:50:51 client-5 kernel: LustreError: 21659:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:50:51 client-5 kernel: LustreError: 21659:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:50:51 client-5 kernel: LustreError: 21659:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:50:51 client-5 kernel: LustreError: 21659:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:50:51 client-5 kernel: Lustre: client ffff81010f95cc00 umount complete May 17 16:51:07 client-5 kernel: Lustre: Acceptor stopping May 17 16:51:08 client-5 kernel: Lustre: Removed LNI 10.10.4.5@tcp May 17 16:51:10 client-5 kernel: Lustre: Removed LNI 192.168.4.5@o2ib May 17 16:51:12 client-5 xinetd[3391]: START: shell pid=22117 from=192.168.4.5 May 17 16:51:12 client-5 rshd[22118]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 16:51:12 client-5 xinetd[3391]: EXIT: shell status=0 pid=22117 duration=0(sec) May 17 16:51:12 client-5 kernel: Lustre: 22254:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:51:12 client-5 kernel: Lustre: 22255:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:51:12 client-5 kernel: Lustre: OBD class driver, http://www.lustre.org/ May 17 16:51:12 client-5 kernel: Lustre: Lustre Version: 2.0.60 May 17 16:51:12 client-5 kernel: Lustre: Build Version: jenkins-g985ac5b-PRISTINE-2.6.18-194.17.1.el5 May 17 16:51:12 client-5 kernel: Lustre: Lustre LU module (ffffffff888eb1c0). May 17 16:51:12 client-5 kernel: Lustre: Register global MR array, MR size: 0xffffffffffffffff, array size: 1 May 17 16:51:12 client-5 kernel: Lustre: Added LNI 192.168.4.5@o2ib [8/64/0/180] May 17 16:51:12 client-5 kernel: Lustre: Added LNI 10.10.4.5@tcp [8/256/0/180] May 17 16:51:12 client-5 kernel: Lustre: Accept all, port 7988 May 17 16:51:12 client-5 kernel: Lustre: Lustre OSC module (ffffffff88b3c6c0). May 17 16:51:12 client-5 kernel: Lustre: Lustre LOV module (ffffffff88bab540). May 17 16:51:12 client-5 kernel: Lustre: Lustre client module (ffffffff88c643a0). May 17 16:51:12 client-5 xinetd[3391]: START: shell pid=22653 from=192.168.4.5 May 17 16:51:12 client-5 rshd[22654]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == conf-sanity test 54a: llverdev ==================================================================== 16:51:12 \(1305676272\)");echo XXRETCODE:$?' May 17 16:51:13 client-5 kernel: Lustre: DEBUG MARKER: == conf-sanity test 54a: llverdev ==================================================================== 16:51:12 (1305676272) May 17 16:51:13 client-5 xinetd[3391]: EXIT: shell status=0 pid=22653 duration=1(sec) May 17 16:51:17 client-5 xinetd[3391]: START: shell pid=22710 from=192.168.4.5 May 17 16:51:17 client-5 rshd[22711]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:; lsof -t /mnt/lustre || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre is still busy, wait one second && sleep 1; umount /mnt/lustre; fi fi");echo XXRETCODE:$?' May 17 16:51:17 client-5 xinetd[3391]: EXIT: shell status=0 pid=22710 duration=0(sec) May 17 16:51:17 client-5 xinetd[3391]: START: shell pid=22753 from=192.168.4.5 May 17 16:51:17 client-5 rshd[22755]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:; lsof -t /mnt/lustre2 || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre2 | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre2 is still busy, wait one second && sleep 1; umount /mnt/lustre2; fi fi");echo XXRETCODE:$?' May 17 16:51:18 client-5 xinetd[3391]: EXIT: shell status=0 pid=22753 duration=1(sec) May 17 16:51:19 client-5 kernel: Lustre: 22957:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:51:32 client-5 xinetd[3391]: START: shell pid=23564 from=192.168.4.5 May 17 16:51:32 client-5 rshd[23566]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark ost now in FULL state");echo XXRETCODE:$?' May 17 16:51:32 client-5 kernel: Lustre: DEBUG MARKER: ost now in FULL state May 17 16:51:32 client-5 xinetd[3391]: EXIT: shell status=0 pid=23564 duration=0(sec) May 17 16:51:43 client-5 xinetd[3391]: START: shell pid=23720 from=192.168.4.5 May 17 16:51:43 client-5 rshd[23723]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 16:51:43 client-5 xinetd[3391]: EXIT: shell status=0 pid=23720 duration=0(sec) May 17 16:51:44 client-5 xinetd[3391]: START: shell pid=23894 from=192.168.4.5 May 17 16:51:44 client-5 rshd[23895]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == conf-sanity test 54b: llverfs ===================================================================== 16:51:44 \(1305676304\)");echo XXRETCODE:$?' May 17 16:51:44 client-5 kernel: Lustre: DEBUG MARKER: == conf-sanity test 54b: llverfs ===================================================================== 16:51:44 (1305676304) May 17 16:51:44 client-5 xinetd[3391]: EXIT: shell status=0 pid=23894 duration=0(sec) May 17 16:51:45 client-5 kernel: Lustre: 24021:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 16:51:45 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 16:51:45 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 16:51:45 client-5 kernel: Lustre: 24021:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff81031ffc3000->192.168.4.128@o2ib netid 50000: select flavor null May 17 16:51:45 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 16:51:50 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 16:51:55 client-5 kernel: Lustre: 24021:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-OST0000-osc-ffff81031ffc3000->192.168.4.129@o2ib netid 50000: select flavor null May 17 16:51:55 client-5 kernel: Lustre: Client lustre-client has started May 17 16:51:55 client-5 kernel: Lustre: 24038:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:51:55 client-5 kernel: Lustre: 24038:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message May 17 16:52:06 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.129@o2ib. The ost_write operation failed with -28 May 17 16:52:06 client-5 kernel: LustreError: 24046:0:(vvp_io.c:987:vvp_io_commit_write()) Write page 40176 of inode ffff8102f1f56e90 failed -28 May 17 16:52:06 client-5 kernel: LustreError: 24046:0:(vvp_io.c:987:vvp_io_commit_write()) Write page 40176 of inode ffff8102f1f56e90 failed -28 May 17 16:52:09 client-5 kernel: LustreError: 24087:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:52:09 client-5 kernel: LustreError: 24087:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:52:09 client-5 kernel: LustreError: 24087:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:52:09 client-5 kernel: LustreError: 24087:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:52:09 client-5 kernel: Lustre: client ffff81031ffc3000 umount complete May 17 16:52:23 client-5 kernel: Lustre: Acceptor stopping May 17 16:52:24 client-5 kernel: Lustre: Removed LNI 10.10.4.5@tcp May 17 16:52:26 client-5 kernel: Lustre: Removed LNI 192.168.4.5@o2ib May 17 16:52:28 client-5 xinetd[3391]: START: shell pid=24542 from=192.168.4.5 May 17 16:52:28 client-5 rshd[24543]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 16:52:28 client-5 xinetd[3391]: EXIT: shell status=0 pid=24542 duration=0(sec) May 17 16:52:28 client-5 kernel: Lustre: 24679:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:52:28 client-5 kernel: Lustre: 24680:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:52:28 client-5 kernel: Lustre: OBD class driver, http://www.lustre.org/ May 17 16:52:28 client-5 kernel: Lustre: Lustre Version: 2.0.60 May 17 16:52:28 client-5 kernel: Lustre: Build Version: jenkins-g985ac5b-PRISTINE-2.6.18-194.17.1.el5 May 17 16:52:28 client-5 kernel: Lustre: Lustre LU module (ffffffff888eb1c0). May 17 16:52:28 client-5 kernel: Lustre: Register global MR array, MR size: 0xffffffffffffffff, array size: 1 May 17 16:52:28 client-5 kernel: Lustre: Added LNI 192.168.4.5@o2ib [8/64/0/180] May 17 16:52:28 client-5 kernel: Lustre: Added LNI 10.10.4.5@tcp [8/256/0/180] May 17 16:52:28 client-5 kernel: Lustre: Accept all, port 7988 May 17 16:52:28 client-5 kernel: Lustre: Lustre OSC module (ffffffff88b3c6c0). May 17 16:52:28 client-5 kernel: Lustre: Lustre LOV module (ffffffff88bab540). May 17 16:52:28 client-5 kernel: Lustre: Lustre client module (ffffffff88c643a0). May 17 16:52:28 client-5 xinetd[3391]: START: shell pid=25083 from=192.168.4.5 May 17 16:52:29 client-5 rshd[25084]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == conf-sanity test 55: check lov_objid size ========================================================= 16:52:28 \(1305676348\)");echo XXRETCODE:$?' May 17 16:52:29 client-5 kernel: Lustre: DEBUG MARKER: == conf-sanity test 55: check lov_objid size ========================================================= 16:52:28 (1305676348) May 17 16:52:29 client-5 xinetd[3391]: EXIT: shell status=0 pid=25083 duration=1(sec) May 17 16:52:29 client-5 xinetd[3391]: START: shell pid=25130 from=192.168.4.5 May 17 16:52:29 client-5 rshd[25131]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:; lsof -t /mnt/lustre || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre is still busy, wait one second && sleep 1; umount /mnt/lustre; fi fi");echo XXRETCODE:$?' May 17 16:52:29 client-5 xinetd[3391]: EXIT: shell status=0 pid=25130 duration=0(sec) May 17 16:52:29 client-5 xinetd[3391]: START: shell pid=25175 from=192.168.4.5 May 17 16:52:29 client-5 rshd[25176]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:; lsof -t /mnt/lustre2 || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre2 | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre2 is still busy, wait one second && sleep 1; umount /mnt/lustre2; fi fi");echo XXRETCODE:$?' May 17 16:52:29 client-5 xinetd[3391]: EXIT: shell status=0 pid=25175 duration=0(sec) May 17 16:52:30 client-5 kernel: Lustre: 25378:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:52:37 client-5 kernel: Lustre: 25821:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 16:52:37 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 16:52:37 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 16:52:37 client-5 kernel: Lustre: 25821:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff8102ff0b8800->192.168.4.128@o2ib netid 50000: select flavor null May 17 16:52:37 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 16:52:42 client-5 kernel: Lustre: 25821:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-OST0000-osc-ffff8102ff0b8800->192.168.4.129@o2ib netid 50000: select flavor null May 17 16:52:42 client-5 kernel: Lustre: Client lustre-client has started May 17 16:52:42 client-5 kernel: Lustre: 25838:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:52:42 client-5 kernel: Lustre: 25838:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message May 17 16:52:42 client-5 xinetd[3391]: START: shell pid=25849 from=192.168.4.5 May 17 16:52:42 client-5 rshd[25850]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:; lsof -t /mnt/lustre || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre is still busy, wait one second && sleep 1; umount /mnt/lustre; fi fi");echo XXRETCODE:$?' May 17 16:52:42 client-5 kernel: LustreError: 25891:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:52:42 client-5 kernel: LustreError: 25891:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:52:42 client-5 kernel: LustreError: 25891:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:52:42 client-5 kernel: LustreError: 25891:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:52:42 client-5 kernel: Lustre: client ffff8102ff0b8800 umount complete May 17 16:52:42 client-5 xinetd[3391]: EXIT: shell status=0 pid=25849 duration=0(sec) May 17 16:52:42 client-5 xinetd[3391]: START: shell pid=25900 from=192.168.4.5 May 17 16:52:42 client-5 rshd[25901]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:; lsof -t /mnt/lustre2 || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre2 | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre2 is still busy, wait one second && sleep 1; umount /mnt/lustre2; fi fi");echo XXRETCODE:$?' May 17 16:52:42 client-5 xinetd[3391]: EXIT: shell status=0 pid=25900 duration=0(sec) May 17 16:52:59 client-5 kernel: Lustre: 26209:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 16:52:59 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 16:52:59 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 16:52:59 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 16:53:04 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 16:53:09 client-5 kernel: Lustre: 26209:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-OST0000-osc-ffff81033f897000->192.168.4.129@o2ib netid 50000: select flavor null May 17 16:53:09 client-5 kernel: Lustre: 26209:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 1 previous similar message May 17 16:53:10 client-5 kernel: Lustre: Client lustre-client has started May 17 16:53:10 client-5 kernel: Lustre: 26226:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:53:10 client-5 kernel: Lustre: 26226:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message May 17 16:53:10 client-5 xinetd[3391]: START: shell pid=26251 from=192.168.4.5 May 17 16:53:10 client-5 rshd[26252]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:; lsof -t /mnt/lustre || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre is still busy, wait one second && sleep 1; umount /mnt/lustre; fi fi");echo XXRETCODE:$?' May 17 16:53:10 client-5 kernel: LustreError: 26293:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:53:10 client-5 kernel: LustreError: 26293:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:53:10 client-5 kernel: Lustre: client ffff81033f897000 umount complete May 17 16:53:10 client-5 xinetd[3391]: EXIT: shell status=0 pid=26251 duration=0(sec) May 17 16:53:10 client-5 xinetd[3391]: START: shell pid=26302 from=192.168.4.5 May 17 16:53:10 client-5 rshd[26303]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:; lsof -t /mnt/lustre2 || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre2 | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre2 is still busy, wait one second && sleep 1; umount /mnt/lustre2; fi fi");echo XXRETCODE:$?' May 17 16:53:10 client-5 xinetd[3391]: EXIT: shell status=0 pid=26302 duration=0(sec) May 17 16:53:26 client-5 xinetd[3391]: START: shell pid=26526 from=192.168.4.5 May 17 16:53:26 client-5 rshd[26527]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:; lsof -t /mnt/lustre || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre is still busy, wait one second && sleep 1; umount /mnt/lustre; fi fi");echo XXRETCODE:$?' May 17 16:53:27 client-5 xinetd[3391]: EXIT: shell status=0 pid=26526 duration=1(sec) May 17 16:53:27 client-5 xinetd[3391]: START: shell pid=26570 from=192.168.4.5 May 17 16:53:27 client-5 rshd[26571]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:; lsof -t /mnt/lustre2 || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre2 | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre2 is still busy, wait one second && sleep 1; umount /mnt/lustre2; fi fi");echo XXRETCODE:$?' May 17 16:53:27 client-5 xinetd[3391]: EXIT: shell status=0 pid=26570 duration=0(sec) May 17 16:53:28 client-5 kernel: Lustre: 26773:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:53:28 client-5 kernel: Lustre: 26773:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message May 17 16:53:35 client-5 kernel: Lustre: 27191:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 16:53:35 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 16:53:35 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 16:53:35 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 16:53:40 client-5 kernel: Lustre: Client lustre-client has started May 17 16:53:40 client-5 kernel: Lustre: 27212:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:53:40 client-5 kernel: Lustre: 27212:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message May 17 16:53:40 client-5 xinetd[3391]: START: shell pid=27223 from=192.168.4.5 May 17 16:53:40 client-5 rshd[27224]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:; lsof -t /mnt/lustre || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre is still busy, wait one second && sleep 1; umount /mnt/lustre; fi fi");echo XXRETCODE:$?' May 17 16:53:40 client-5 kernel: LustreError: 27265:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:53:40 client-5 kernel: LustreError: 27265:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Skipped 1 previous similar message May 17 16:53:40 client-5 kernel: LustreError: 27265:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:53:40 client-5 kernel: LustreError: 27265:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) Skipped 1 previous similar message May 17 16:53:40 client-5 kernel: Lustre: client ffff81031ecba400 umount complete May 17 16:53:40 client-5 xinetd[3391]: EXIT: shell status=0 pid=27223 duration=0(sec) May 17 16:53:40 client-5 xinetd[3391]: START: shell pid=27274 from=192.168.4.5 May 17 16:53:40 client-5 rshd[27275]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:; lsof -t /mnt/lustre2 || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre2 | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre2 is still busy, wait one second && sleep 1; umount /mnt/lustre2; fi fi");echo XXRETCODE:$?' May 17 16:53:40 client-5 xinetd[3391]: EXIT: shell status=0 pid=27274 duration=0(sec) May 17 16:53:57 client-5 kernel: Lustre: 27582:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 16:53:57 client-5 kernel: Lustre: 27582:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 2 previous similar messages May 17 16:53:57 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 16:53:57 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 16:53:57 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 16:54:02 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 16:54:07 client-5 kernel: Lustre: Client lustre-client has started May 17 16:54:07 client-5 kernel: Lustre: 27599:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:54:07 client-5 kernel: Lustre: 27599:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message May 17 16:54:08 client-5 xinetd[3391]: START: shell pid=27624 from=192.168.4.5 May 17 16:54:08 client-5 rshd[27625]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:; lsof -t /mnt/lustre || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre is still busy, wait one second && sleep 1; umount /mnt/lustre; fi fi");echo XXRETCODE:$?' May 17 16:54:08 client-5 kernel: LustreError: 27666:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:54:08 client-5 kernel: LustreError: 27666:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Skipped 1 previous similar message May 17 16:54:08 client-5 kernel: LustreError: 27666:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:54:08 client-5 kernel: LustreError: 27666:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) Skipped 1 previous similar message May 17 16:54:08 client-5 kernel: Lustre: client ffff81023d88b800 umount complete May 17 16:54:08 client-5 xinetd[3391]: EXIT: shell status=0 pid=27624 duration=0(sec) May 17 16:54:08 client-5 xinetd[3391]: START: shell pid=27675 from=192.168.4.5 May 17 16:54:08 client-5 rshd[27676]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:; lsof -t /mnt/lustre2 || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre2 | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre2 is still busy, wait one second && sleep 1; umount /mnt/lustre2; fi fi");echo XXRETCODE:$?' May 17 16:54:08 client-5 xinetd[3391]: EXIT: shell status=0 pid=27675 duration=0(sec) May 17 16:54:24 client-5 xinetd[3391]: START: shell pid=27899 from=192.168.4.5 May 17 16:54:24 client-5 rshd[27900]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:; lsof -t /mnt/lustre || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre is still busy, wait one second && sleep 1; umount /mnt/lustre; fi fi");echo XXRETCODE:$?' May 17 16:54:24 client-5 xinetd[3391]: EXIT: shell status=0 pid=27899 duration=0(sec) May 17 16:54:24 client-5 xinetd[3391]: START: shell pid=27943 from=192.168.4.5 May 17 16:54:24 client-5 rshd[27944]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:; lsof -t /mnt/lustre2 || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre2 | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre2 is still busy, wait one second && sleep 1; umount /mnt/lustre2; fi fi");echo XXRETCODE:$?' May 17 16:54:25 client-5 xinetd[3391]: EXIT: shell status=0 pid=27943 duration=1(sec) May 17 16:54:33 client-5 kernel: Lustre: 28564:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 16:54:33 client-5 kernel: Lustre: 28564:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 2 previous similar messages May 17 16:54:33 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 16:54:33 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 16:54:33 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 16:54:38 client-5 kernel: Lustre: Client lustre-client has started May 17 16:54:38 client-5 xinetd[3391]: START: shell pid=28592 from=192.168.4.5 May 17 16:54:38 client-5 rshd[28593]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:; lsof -t /mnt/lustre || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre is still busy, wait one second && sleep 1; umount /mnt/lustre; fi fi");echo XXRETCODE:$?' May 17 16:54:38 client-5 kernel: LustreError: 28634:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:54:38 client-5 kernel: LustreError: 28634:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Skipped 1 previous similar message May 17 16:54:38 client-5 kernel: LustreError: 28634:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:54:38 client-5 kernel: LustreError: 28634:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) Skipped 1 previous similar message May 17 16:54:38 client-5 kernel: Lustre: client ffff8102880ef800 umount complete May 17 16:54:38 client-5 xinetd[3391]: EXIT: shell status=0 pid=28592 duration=0(sec) May 17 16:54:38 client-5 xinetd[3391]: START: shell pid=28643 from=192.168.4.5 May 17 16:54:38 client-5 rshd[28644]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:; lsof -t /mnt/lustre2 || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre2 | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre2 is still busy, wait one second && sleep 1; umount /mnt/lustre2; fi fi");echo XXRETCODE:$?' May 17 16:54:38 client-5 xinetd[3391]: EXIT: shell status=0 pid=28643 duration=0(sec) May 17 16:54:56 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 16:54:56 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 16:54:56 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 16:55:01 client-5 kernel: Lustre: Client lustre-client has started May 17 16:55:01 client-5 kernel: Lustre: 28972:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:55:01 client-5 kernel: Lustre: 28972:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 5 previous similar messages May 17 16:55:01 client-5 xinetd[3391]: START: shell pid=28996 from=192.168.4.5 May 17 16:55:01 client-5 rshd[28998]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:; lsof -t /mnt/lustre || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre is still busy, wait one second && sleep 1; umount /mnt/lustre; fi fi");echo XXRETCODE:$?' May 17 16:55:02 client-5 kernel: LustreError: 29039:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:55:02 client-5 kernel: LustreError: 29039:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Skipped 1 previous similar message May 17 16:55:02 client-5 kernel: LustreError: 29039:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:55:02 client-5 kernel: LustreError: 29039:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) Skipped 1 previous similar message May 17 16:55:02 client-5 kernel: Lustre: client ffff8102f0a88000 umount complete May 17 16:55:02 client-5 xinetd[3391]: EXIT: shell status=0 pid=28996 duration=1(sec) May 17 16:55:02 client-5 xinetd[3391]: START: shell pid=29048 from=192.168.4.5 May 17 16:55:02 client-5 rshd[29049]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:; lsof -t /mnt/lustre2 || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre2 | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre2 is still busy, wait one second && sleep 1; umount /mnt/lustre2; fi fi");echo XXRETCODE:$?' May 17 16:55:02 client-5 xinetd[3391]: EXIT: shell status=0 pid=29048 duration=0(sec) May 17 16:55:18 client-5 xinetd[3391]: START: shell pid=29272 from=192.168.4.5 May 17 16:55:18 client-5 rshd[29273]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:; lsof -t /mnt/lustre || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre is still busy, wait one second && sleep 1; umount /mnt/lustre; fi fi");echo XXRETCODE:$?' May 17 16:55:18 client-5 xinetd[3391]: EXIT: shell status=0 pid=29272 duration=0(sec) May 17 16:55:18 client-5 xinetd[3391]: START: shell pid=29316 from=192.168.4.5 May 17 16:55:18 client-5 rshd[29317]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:; lsof -t /mnt/lustre2 || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre2 | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre2 is still busy, wait one second && sleep 1; umount /mnt/lustre2; fi fi");echo XXRETCODE:$?' May 17 16:55:18 client-5 xinetd[3391]: EXIT: shell status=0 pid=29316 duration=0(sec) May 17 16:55:25 client-5 xinetd[3391]: START: shell pid=29901 from=192.168.4.5 May 17 16:55:25 client-5 rshd[29902]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 16:55:25 client-5 xinetd[3391]: EXIT: shell status=0 pid=29901 duration=0(sec) May 17 16:55:25 client-5 xinetd[3391]: START: shell pid=30073 from=192.168.4.5 May 17 16:55:25 client-5 rshd[30074]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == conf-sanity test 56: check big indexes ============================================================ 16:55:25 \(1305676525\)");echo XXRETCODE:$?' May 17 16:55:26 client-5 kernel: Lustre: DEBUG MARKER: == conf-sanity test 56: check big indexes ============================================================ 16:55:25 (1305676525) May 17 16:55:26 client-5 xinetd[3391]: EXIT: shell status=0 pid=30073 duration=1(sec) May 17 16:55:29 client-5 kernel: Lustre: 30336:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 16:55:29 client-5 kernel: Lustre: 30336:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 5 previous similar messages May 17 16:55:29 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 16:55:29 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 16:55:30 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 16:55:35 client-5 kernel: Lustre: Client lustre-client has started May 17 16:55:35 client-5 xinetd[3391]: START: shell pid=30365 from=192.168.4.5 May 17 16:55:35 client-5 rshd[30366]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:; lsof -t /mnt/lustre || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre is still busy, wait one second && sleep 1; umount /mnt/lustre; fi fi");echo XXRETCODE:$?' May 17 16:55:35 client-5 kernel: LustreError: 30407:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:55:35 client-5 kernel: LustreError: 30407:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Skipped 1 previous similar message May 17 16:55:35 client-5 kernel: LustreError: 30407:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:55:35 client-5 kernel: LustreError: 30407:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) Skipped 1 previous similar message May 17 16:55:35 client-5 kernel: Lustre: client ffff810313c40c00 umount complete May 17 16:55:35 client-5 xinetd[3391]: EXIT: shell status=0 pid=30365 duration=0(sec) May 17 16:55:35 client-5 xinetd[3391]: START: shell pid=30416 from=192.168.4.5 May 17 16:55:35 client-5 rshd[30417]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:; lsof -t /mnt/lustre2 || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre2 | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre2 is still busy, wait one second && sleep 1; umount /mnt/lustre2; fi fi");echo XXRETCODE:$?' May 17 16:55:36 client-5 xinetd[3391]: EXIT: shell status=0 pid=30416 duration=1(sec) May 17 16:55:53 client-5 xinetd[3391]: START: shell pid=30648 from=192.168.4.5 May 17 16:55:53 client-5 rshd[30649]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:; lsof -t /mnt/lustre || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre is still busy, wait one second && sleep 1; umount /mnt/lustre; fi fi");echo XXRETCODE:$?' May 17 16:55:53 client-5 xinetd[3391]: EXIT: shell status=0 pid=30648 duration=0(sec) May 17 16:55:53 client-5 xinetd[3391]: START: shell pid=30692 from=192.168.4.5 May 17 16:55:53 client-5 rshd[30693]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:; lsof -t /mnt/lustre2 || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre2 | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre2 is still busy, wait one second && sleep 1; umount /mnt/lustre2; fi fi");echo XXRETCODE:$?' May 17 16:55:53 client-5 xinetd[3391]: EXIT: shell status=0 pid=30692 duration=0(sec) May 17 16:56:00 client-5 xinetd[3391]: START: shell pid=31277 from=192.168.4.5 May 17 16:56:00 client-5 rshd[31278]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 16:56:00 client-5 xinetd[3391]: EXIT: shell status=0 pid=31277 duration=0(sec) May 17 16:56:00 client-5 xinetd[3391]: START: shell pid=31449 from=192.168.4.5 May 17 16:56:00 client-5 rshd[31450]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == conf-sanity test 57: initial registration from failnode should fail \(should return errs\) ========== 16:56:00 \(1305676560\)");echo XXRETCODE:$?' May 17 16:56:00 client-5 kernel: Lustre: DEBUG MARKER: == conf-sanity test 57: initial registration from failnode should fail (should return errs) ========== 16:56:00 (1305676560) May 17 16:56:00 client-5 xinetd[3391]: EXIT: shell status=0 pid=31449 duration=0(sec) May 17 16:56:09 client-5 xinetd[3391]: START: shell pid=31773 from=192.168.4.5 May 17 16:56:09 client-5 rshd[31774]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:; lsof -t /mnt/lustre || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre is still busy, wait one second && sleep 1; umount /mnt/lustre; fi fi");echo XXRETCODE:$?' May 17 16:56:09 client-5 xinetd[3391]: EXIT: shell status=0 pid=31773 duration=0(sec) May 17 16:56:09 client-5 xinetd[3391]: START: shell pid=31817 from=192.168.4.5 May 17 16:56:09 client-5 rshd[31818]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:; lsof -t /mnt/lustre2 || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre2 | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre2 is still busy, wait one second && sleep 1; umount /mnt/lustre2; fi fi");echo XXRETCODE:$?' May 17 16:56:09 client-5 xinetd[3391]: EXIT: shell status=0 pid=31817 duration=0(sec) May 17 16:56:11 client-5 kernel: Lustre: 32020:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:56:11 client-5 kernel: Lustre: 32020:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 9 previous similar messages May 17 16:56:16 client-5 xinetd[3391]: START: shell pid=32400 from=192.168.4.5 May 17 16:56:16 client-5 rshd[32403]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 16:56:16 client-5 xinetd[3391]: EXIT: shell status=0 pid=32400 duration=0(sec) May 17 16:56:16 client-5 xinetd[3391]: START: shell pid=32574 from=192.168.4.5 May 17 16:56:16 client-5 rshd[32575]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == conf-sanity test 59: writeconf mount option ======================================================= 16:56:16 \(1305676576\)");echo XXRETCODE:$?' May 17 16:56:17 client-5 kernel: Lustre: DEBUG MARKER: == conf-sanity test 59: writeconf mount option ======================================================= 16:56:16 (1305676576) May 17 16:56:17 client-5 xinetd[3391]: EXIT: shell status=0 pid=32574 duration=1(sec) May 17 16:56:18 client-5 xinetd[3391]: START: shell pid=32724 from=192.168.4.5 May 17 16:56:18 client-5 rshd[32725]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:; lsof -t /mnt/lustre || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre is still busy, wait one second && sleep 1; umount /mnt/lustre; fi fi");echo XXRETCODE:$?' May 17 16:56:18 client-5 xinetd[3391]: EXIT: shell status=0 pid=32724 duration=0(sec) May 17 16:56:18 client-5 xinetd[3391]: START: shell pid=300 from=192.168.4.5 May 17 16:56:18 client-5 rshd[302]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:; lsof -t /mnt/lustre2 || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre2 | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre2 is still busy, wait one second && sleep 1; umount /mnt/lustre2; fi fi");echo XXRETCODE:$?' May 17 16:56:18 client-5 xinetd[3391]: EXIT: shell status=0 pid=300 duration=0(sec) May 17 16:56:54 client-5 kernel: Lustre: Acceptor stopping May 17 16:56:55 client-5 kernel: Lustre: Removed LNI 10.10.4.5@tcp May 17 16:56:57 client-5 kernel: Lustre: Removed LNI 192.168.4.5@o2ib May 17 16:56:58 client-5 xinetd[3391]: START: shell pid=1208 from=192.168.4.5 May 17 16:56:58 client-5 rshd[1210]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 16:56:58 client-5 xinetd[3391]: EXIT: shell status=0 pid=1208 duration=0(sec) May 17 16:56:58 client-5 kernel: Lustre: 1345:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:56:58 client-5 kernel: Lustre: 1347:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:56:59 client-5 kernel: Lustre: OBD class driver, http://www.lustre.org/ May 17 16:56:59 client-5 kernel: Lustre: Lustre Version: 2.0.60 May 17 16:56:59 client-5 kernel: Lustre: Build Version: jenkins-g985ac5b-PRISTINE-2.6.18-194.17.1.el5 May 17 16:56:59 client-5 kernel: Lustre: Lustre LU module (ffffffff888eb1c0). May 17 16:56:59 client-5 kernel: Lustre: Register global MR array, MR size: 0xffffffffffffffff, array size: 1 May 17 16:56:59 client-5 kernel: Lustre: Added LNI 192.168.4.5@o2ib [8/64/0/180] May 17 16:56:59 client-5 kernel: Lustre: Added LNI 10.10.4.5@tcp [8/256/0/180] May 17 16:56:59 client-5 kernel: Lustre: Accept all, port 7988 May 17 16:56:59 client-5 kernel: Lustre: Lustre OSC module (ffffffff88b3c6c0). May 17 16:56:59 client-5 kernel: Lustre: Lustre LOV module (ffffffff88bab540). May 17 16:56:59 client-5 kernel: Lustre: Lustre client module (ffffffff88c643a0). May 17 16:56:59 client-5 xinetd[3391]: START: shell pid=1745 from=192.168.4.5 May 17 16:56:59 client-5 rshd[1746]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == conf-sanity test 58: missing llog files must not prevent MDT from mounting ======================== 16:56:58 \(1305676618\)");echo XXRETCODE:$?' May 17 16:56:59 client-5 kernel: Lustre: DEBUG MARKER: == conf-sanity test 58: missing llog files must not prevent MDT from mounting ======================== 16:56:58 (1305676618) May 17 16:56:59 client-5 xinetd[3391]: EXIT: shell status=0 pid=1745 duration=0(sec) May 17 16:57:00 client-5 kernel: Lustre: 1884:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 16:57:00 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 16:57:00 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 16:57:01 client-5 kernel: Lustre: 1884:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff810327f42c00->192.168.4.128@o2ib netid 50000: select flavor null May 17 16:57:01 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 16:57:06 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 16:57:11 client-5 kernel: Lustre: 1884:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-OST0000-osc-ffff810327f42c00->192.168.4.129@o2ib netid 50000: select flavor null May 17 16:57:11 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.129@o2ib. The ost_connect operation failed with -19 May 17 16:57:11 client-5 kernel: Lustre: Client lustre-client has started May 17 16:57:11 client-5 kernel: Lustre: 1906:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:57:12 client-5 xinetd[3391]: START: shell pid=2009 from=192.168.4.5 May 17 16:57:12 client-5 rshd[2010]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:; lsof -t /mnt/lustre || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre is still busy, wait one second && sleep 1; umount /mnt/lustre; fi fi");echo XXRETCODE:$?' May 17 16:57:13 client-5 kernel: LustreError: 2052:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:57:13 client-5 kernel: LustreError: 2052:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:57:13 client-5 kernel: LustreError: 2052:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 16:57:13 client-5 kernel: LustreError: 2052:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 16:57:13 client-5 kernel: Lustre: client ffff810327f42c00 umount complete May 17 16:57:13 client-5 xinetd[3391]: EXIT: shell status=0 pid=2009 duration=1(sec) May 17 16:57:13 client-5 xinetd[3391]: START: shell pid=2061 from=192.168.4.5 May 17 16:57:13 client-5 rshd[2062]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:; lsof -t /mnt/lustre2 || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre2 | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre2 is still busy, wait one second && sleep 1; umount /mnt/lustre2; fi fi");echo XXRETCODE:$?' May 17 16:57:13 client-5 xinetd[3391]: EXIT: shell status=0 pid=2061 duration=0(sec) May 17 16:57:34 client-5 kernel: Lustre: 2298:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:57:34 client-5 kernel: Lustre: 2298:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message May 17 16:57:40 client-5 xinetd[3391]: START: shell pid=2698 from=192.168.4.5 May 17 16:57:40 client-5 rshd[2699]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 16:57:40 client-5 xinetd[3391]: EXIT: shell status=0 pid=2698 duration=0(sec) May 17 16:57:40 client-5 xinetd[3391]: START: shell pid=2862 from=192.168.4.5 May 17 16:57:40 client-5 rshd[2864]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == conf-sanity conf-sanity.sh test complete, duration 4292 sec ======================================= 16:57:40 \(1305676660\)");echo XXRETCODE:$?' May 17 16:57:40 client-5 kernel: Lustre: DEBUG MARKER: == conf-sanity conf-sanity.sh test complete, duration 4292 sec ======================================= 16:57:40 (1305676660) May 17 16:57:40 client-5 xinetd[3391]: EXIT: shell status=0 pid=2862 duration=0(sec) May 17 16:57:40 client-5 xinetd[3391]: START: shell pid=2968 from=192.168.4.5 May 17 16:57:40 client-5 rshd[2969]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark -----============= acceptance-small: recovery-small ============----- Tue May 17 16:57:40 PDT 2011");echo XXRETCODE:$?' May 17 16:57:40 client-5 kernel: Lustre: DEBUG MARKER: -----============= acceptance-small: recovery-small ============----- Tue May 17 16:57:40 PDT 2011 May 17 16:57:40 client-5 xinetd[3391]: EXIT: shell status=0 pid=2968 duration=0(sec) May 17 16:57:40 client-5 xinetd[3391]: START: shell pid=3123 from=192.168.4.5 May 17 16:57:40 client-5 rshd[3124]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark excepting tests: 24 52");echo XXRETCODE:$?' May 17 16:57:40 client-5 kernel: Lustre: DEBUG MARKER: excepting tests: 24 52 May 17 16:57:40 client-5 xinetd[3391]: EXIT: shell status=0 pid=3123 duration=0(sec) May 17 16:57:40 client-5 xinetd[3391]: START: shell pid=3183 from=192.168.4.5 May 17 16:57:41 client-5 rshd[3185]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:; lsof -t /mnt/lustre || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre is still busy, wait one second && sleep 1; umount /mnt/lustre; fi fi");echo XXRETCODE:$?' May 17 16:57:41 client-5 xinetd[3391]: EXIT: shell status=0 pid=3183 duration=1(sec) May 17 16:57:41 client-5 xinetd[3391]: START: shell pid=3230 from=192.168.4.5 May 17 16:57:41 client-5 rshd[3231]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:; lsof -t /mnt/lustre2 || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre2 | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre2 is still busy, wait one second && sleep 1; umount /mnt/lustre2; fi fi");echo XXRETCODE:$?' May 17 16:57:41 client-5 xinetd[3391]: EXIT: shell status=0 pid=3230 duration=0(sec) May 17 16:57:42 client-5 kernel: Lustre: 3459:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:57:42 client-5 kernel: Lustre: 3459:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message May 17 16:58:40 client-5 xinetd[3391]: START: shell pid=3912 from=192.168.4.5 May 17 16:58:40 client-5 rshd[3913]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); mpts=\$(mount | grep -w -c /mnt/lustre); if [ \$running -ne \$mpts ]; then echo \$(hostname) env are INSANE!; exit 1; fi");echo XXRETCODE:$?' May 17 16:58:40 client-5 xinetd[3391]: EXIT: shell status=0 pid=3912 duration=0(sec) May 17 16:58:40 client-5 xinetd[3391]: START: shell pid=3959 from=192.168.4.5 May 17 16:58:40 client-5 rshd[3960]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); mpts=\$(mount | grep -w -c /mnt/lustre2); if [ \$running -ne \$mpts ]; then echo \$(hostname) env are INSANE!; exit 1; fi");echo XXRETCODE:$?' May 17 16:58:41 client-5 xinetd[3391]: EXIT: shell status=0 pid=3959 duration=1(sec) May 17 16:58:41 client-5 kernel: Lustre: 4002:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:58:41 client-5 kernel: Lustre: 4002:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message May 17 16:58:45 client-5 kernel: Lustre: 4339:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 16:58:45 client-5 kernel: Lustre: 4339:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 1 previous similar message May 17 16:58:45 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 16:58:45 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 16:58:45 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 16:58:50 client-5 kernel: Lustre: 4339:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-OST0000-osc-ffff810318437400->192.168.4.129@o2ib netid 50000: select flavor null May 17 16:58:50 client-5 kernel: Lustre: 4339:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 1 previous similar message May 17 16:58:50 client-5 kernel: Lustre: Client lustre-client has started May 17 16:58:50 client-5 kernel: Lustre: 4356:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 16:58:50 client-5 kernel: Lustre: 4356:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message May 17 16:58:50 client-5 xinetd[3391]: START: shell pid=4366 from=192.168.4.5 May 17 16:58:50 client-5 rshd[4367]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c " running=\$(mount | grep -c /mnt/lustre' '); rc=0; if [ \$running -eq 0 ] ; then mkdir -p /mnt/lustre; mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre; rc=\$?; fi; exit \$rc");echo XXRETCODE:$?' May 17 16:58:50 client-5 xinetd[3391]: EXIT: shell status=0 pid=4366 duration=0(sec) May 17 16:58:50 client-5 xinetd[3391]: START: shell pid=4411 from=192.168.4.5 May 17 16:58:50 client-5 rshd[4412]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mount | grep -w /mnt/lustre");echo XXRETCODE:$?' May 17 16:58:51 client-5 xinetd[3391]: EXIT: shell status=0 pid=4411 duration=1(sec) May 17 16:58:51 client-5 xinetd[3391]: START: shell pid=4455 from=192.168.4.5 May 17 16:58:51 client-5 rshd[4456]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param debug=\"-1\"; lctl set_param subsystem_debug=\"0xffb7e3ff\"; lctl set_param debug_mb=32;");echo XXRETCODE:$?' May 17 16:58:51 client-5 xinetd[3391]: EXIT: shell status=0 pid=4455 duration=0(sec) May 17 16:58:51 client-5 xinetd[3391]: START: shell pid=4625 from=192.168.4.5 May 17 16:58:51 client-5 rshd[4626]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark Using TIMEOUT=20");echo XXRETCODE:$?' May 17 16:58:51 client-5 kernel: Lustre: DEBUG MARKER: Using TIMEOUT=20 May 17 16:58:51 client-5 xinetd[3391]: EXIT: shell status=0 pid=4625 duration=0(sec) May 17 16:58:51 client-5 kernel: LustreError: 4679:0:(quota_ctl.c:328:client_quota_ctl()) ptlrpc_queue_wait failed, rc: -114 May 17 16:58:56 client-5 xinetd[3391]: START: shell pid=4699 from=192.168.4.5 May 17 16:58:56 client-5 rshd[4700]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "cat /proc/mounts");echo XXRETCODE:$?' May 17 16:58:56 client-5 xinetd[3391]: EXIT: shell status=0 pid=4699 duration=0(sec) May 17 16:58:57 client-5 xinetd[3391]: START: shell pid=4970 from=192.168.4.5 May 17 16:58:57 client-5 rshd[4971]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == recovery-small test 1: mcreate: drop req, drop rep ================================================ 16:58:57 \(1305676737\)");echo XXRETCODE:$?' May 17 16:58:58 client-5 kernel: Lustre: DEBUG MARKER: == recovery-small test 1: mcreate: drop req, drop rep ================================================ 16:58:57 (1305676737) May 17 16:58:58 client-5 xinetd[3391]: EXIT: shell status=0 pid=4970 duration=1(sec) May 17 16:59:10 client-5 kernel: Lustre: 5018:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369101166445116 sent from lustre-MDT0000-mdc-ffff810318437400 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305676738] [real_sent 1305676738] [current 1305676750] [deadline 12s] [delay 0s] req@ffff81033f795000 x1369101166445116/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:30/10 lens 232/216 e 0 to 1 dl 1305676750 ref 2 fl Rpc:X/ffffffff/ffffffff rc 0/-1 May 17 16:59:10 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff810318437400: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 16:59:10 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff810318437400: Connection restored to service lustre-MDT0000 using nid 192.168.4.128@o2ib. May 17 16:59:17 client-5 kernel: Lustre: 5041:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369101166445134 sent from lustre-MDT0000-mdc-ffff810318437400 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305676750] [real_sent 1305676750] [current 1305676757] [deadline 7s] [delay 0s] req@ffff81030ffd2c00 x1369101166445134/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 456/536 e 0 to 1 dl 1305676757 ref 2 fl Rpc:X/ffffffff/ffffffff rc 0/-1 May 17 16:59:17 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff810318437400: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 16:59:17 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff810318437400: Connection restored to service lustre-MDT0000 using nid 192.168.4.128@o2ib. May 17 16:59:17 client-5 xinetd[3391]: START: shell pid=5106 from=192.168.4.5 May 17 16:59:17 client-5 rshd[5107]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 16:59:17 client-5 xinetd[3391]: EXIT: shell status=0 pid=5106 duration=0(sec) May 17 16:59:17 client-5 xinetd[3391]: START: shell pid=5278 from=192.168.4.5 May 17 16:59:18 client-5 rshd[5279]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == recovery-small test 2: chmod: drop req, drop rep ================================================== 16:59:17 \(1305676757\)");echo XXRETCODE:$?' May 17 16:59:18 client-5 kernel: Lustre: DEBUG MARKER: == recovery-small test 2: chmod: drop req, drop rep ================================================== 16:59:17 (1305676757) May 17 16:59:18 client-5 xinetd[3391]: EXIT: shell status=0 pid=5278 duration=1(sec) May 17 16:59:25 client-5 kernel: Lustre: 5326:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369101166445143 sent from lustre-MDT0000-mdc-ffff810318437400 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305676758] [real_sent 1305676758] [current 1305676765] [deadline 7s] [delay 0s] req@ffff81030b21b000 x1369101166445143/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 544/1112 e 0 to 1 dl 1305676765 ref 2 fl Rpc:X/ffffffff/ffffffff rc 0/-1 May 17 16:59:25 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff810318437400: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 16:59:25 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff810318437400: Connection restored to service lustre-MDT0000 using nid 192.168.4.128@o2ib. May 17 16:59:32 client-5 kernel: Lustre: 5348:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369101166445155 sent from lustre-MDT0000-mdc-ffff810318437400 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305676765] [real_sent 1305676765] [current 1305676772] [deadline 7s] [delay 0s] req@ffff8102c6373000 x1369101166445155/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 456/984 e 0 to 1 dl 1305676772 ref 2 fl Rpc:X/ffffffff/ffffffff rc 0/-1 May 17 16:59:32 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff810318437400: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 16:59:32 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff810318437400: Connection restored to service lustre-MDT0000 using nid 192.168.4.128@o2ib. May 17 16:59:32 client-5 xinetd[3391]: START: shell pid=5413 from=192.168.4.5 May 17 16:59:32 client-5 rshd[5414]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 16:59:32 client-5 xinetd[3391]: EXIT: shell status=0 pid=5413 duration=0(sec) May 17 16:59:33 client-5 xinetd[3391]: START: shell pid=5585 from=192.168.4.5 May 17 16:59:33 client-5 rshd[5586]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == recovery-small test 3: stat: drop req, drop rep =================================================== 16:59:33 \(1305676773\)");echo XXRETCODE:$?' May 17 16:59:33 client-5 kernel: Lustre: DEBUG MARKER: == recovery-small test 3: stat: drop req, drop rep =================================================== 16:59:33 (1305676773) May 17 16:59:33 client-5 xinetd[3391]: EXIT: shell status=0 pid=5585 duration=0(sec) May 17 16:59:40 client-5 kernel: Lustre: 5633:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369101166445164 sent from lustre-MDT0000-mdc-ffff810318437400 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305676773] [real_sent 1305676773] [current 1305676780] [deadline 7s] [delay 0s] req@ffff8102f52d9000 x1369101166445164/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 424/928 e 0 to 1 dl 1305676780 ref 2 fl Rpc:X/ffffffff/ffffffff rc 0/-1 May 17 16:59:40 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff810318437400: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 16:59:40 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff810318437400: Connection restored to service lustre-MDT0000 using nid 192.168.4.128@o2ib. May 17 16:59:47 client-5 kernel: Lustre: 5654:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369101166445173 sent from lustre-MDT0000-mdc-ffff810318437400 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305676780] [real_sent 1305676780] [current 1305676787] [deadline 7s] [delay 0s] req@ffff81033f795400 x1369101166445173/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 424/928 e 0 to 1 dl 1305676787 ref 2 fl Rpc:X/ffffffff/ffffffff rc 0/-1 May 17 16:59:47 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff810318437400: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 16:59:47 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff810318437400: Connection restored to service lustre-MDT0000 using nid 192.168.4.128@o2ib. May 17 16:59:47 client-5 xinetd[3391]: START: shell pid=5720 from=192.168.4.5 May 17 16:59:47 client-5 rshd[5721]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 16:59:47 client-5 xinetd[3391]: EXIT: shell status=0 pid=5720 duration=0(sec) May 17 16:59:48 client-5 xinetd[3391]: START: shell pid=5893 from=192.168.4.5 May 17 16:59:48 client-5 rshd[5894]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == recovery-small test 4: open: drop req, drop rep =================================================== 16:59:48 \(1305676788\)");echo XXRETCODE:$?' May 17 16:59:48 client-5 kernel: Lustre: DEBUG MARKER: == recovery-small test 4: open: drop req, drop rep =================================================== 16:59:48 (1305676788) May 17 16:59:48 client-5 xinetd[3391]: EXIT: shell status=0 pid=5893 duration=0(sec) May 17 16:59:56 client-5 kernel: Lustre: 5948:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369101166445193 sent from lustre-MDT0000-mdc-ffff810318437400 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305676789] [real_sent 1305676789] [current 1305676796] [deadline 7s] [delay 0s] req@ffff8102fcfac400 x1369101166445193/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 560/1104 e 0 to 1 dl 1305676796 ref 2 fl Rpc:XP/ffffffff/ffffffff rc 0/-1 May 17 16:59:56 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff810318437400: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 16:59:56 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff810318437400: Connection restored to service lustre-MDT0000 using nid 192.168.4.128@o2ib. May 17 17:00:03 client-5 kernel: LustreError: 5975:0:(file.c:155:ll_close_inode_openhandle()) inode 144115205255725059 mdc close failed: rc = -116 May 17 17:00:03 client-5 xinetd[3391]: START: shell pid=6040 from=192.168.4.5 May 17 17:00:03 client-5 rshd[6041]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 17:00:03 client-5 xinetd[3391]: EXIT: shell status=0 pid=6040 duration=0(sec) May 17 17:00:03 client-5 xinetd[3391]: START: shell pid=6211 from=192.168.4.5 May 17 17:00:04 client-5 rshd[6213]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == recovery-small test 5: rename: drop req, drop rep ================================================= 17:00:03 \(1305676803\)");echo XXRETCODE:$?' May 17 17:00:04 client-5 kernel: Lustre: DEBUG MARKER: == recovery-small test 5: rename: drop req, drop rep ================================================= 17:00:03 (1305676803) May 17 17:00:04 client-5 xinetd[3391]: EXIT: shell status=0 pid=6211 duration=1(sec) May 17 17:00:18 client-5 kernel: Lustre: 6281:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369101166445231 sent from lustre-MDT0000-mdc-ffff810318437400 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305676811] [real_sent 1305676811] [current 1305676818] [deadline 7s] [delay 0s] req@ffff8102f7bd1800 x1369101166445231/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 496/1048 e 0 to 1 dl 1305676818 ref 2 fl Rpc:X/ffffffff/ffffffff rc 0/-1 May 17 17:00:18 client-5 kernel: Lustre: 6281:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 2 previous similar messages May 17 17:00:18 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff810318437400: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 17:00:18 client-5 kernel: Lustre: Skipped 2 previous similar messages May 17 17:00:18 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff810318437400: Connection restored to service lustre-MDT0000 using nid 192.168.4.128@o2ib. May 17 17:00:18 client-5 kernel: Lustre: Skipped 2 previous similar messages May 17 17:00:18 client-5 xinetd[3391]: START: shell pid=6351 from=192.168.4.5 May 17 17:00:18 client-5 rshd[6354]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 17:00:18 client-5 xinetd[3391]: EXIT: shell status=0 pid=6351 duration=0(sec) May 17 17:00:18 client-5 xinetd[3391]: START: shell pid=6525 from=192.168.4.5 May 17 17:00:19 client-5 rshd[6526]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == recovery-small test 6: link: drop req, drop rep =================================================== 17:00:18 \(1305676818\)");echo XXRETCODE:$?' May 17 17:00:19 client-5 kernel: Lustre: DEBUG MARKER: == recovery-small test 6: link: drop req, drop rep =================================================== 17:00:18 (1305676818) May 17 17:00:19 client-5 xinetd[3391]: EXIT: shell status=0 pid=6525 duration=1(sec) May 17 17:00:33 client-5 xinetd[3391]: START: shell pid=6660 from=192.168.4.5 May 17 17:00:33 client-5 rshd[6661]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 17:00:33 client-5 xinetd[3391]: EXIT: shell status=0 pid=6660 duration=0(sec) May 17 17:00:33 client-5 xinetd[3391]: START: shell pid=6831 from=192.168.4.5 May 17 17:00:33 client-5 rshd[6833]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == recovery-small test 7: unlink: drop req, drop rep ================================================= 17:00:33 \(1305676833\)");echo XXRETCODE:$?' May 17 17:00:34 client-5 kernel: Lustre: DEBUG MARKER: == recovery-small test 7: unlink: drop req, drop rep ================================================= 17:00:33 (1305676833) May 17 17:00:34 client-5 xinetd[3391]: EXIT: shell status=0 pid=6831 duration=1(sec) May 17 17:00:48 client-5 xinetd[3391]: START: shell pid=6967 from=192.168.4.5 May 17 17:00:48 client-5 rshd[6968]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 17:00:48 client-5 xinetd[3391]: EXIT: shell status=0 pid=6967 duration=0(sec) May 17 17:00:48 client-5 xinetd[3391]: START: shell pid=7139 from=192.168.4.5 May 17 17:00:48 client-5 rshd[7140]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == recovery-small test 8: touch: drop rep \(bug 1423\) ================================================= 17:00:48 \(1305676848\)");echo XXRETCODE:$?' May 17 17:00:48 client-5 kernel: Lustre: DEBUG MARKER: == recovery-small test 8: touch: drop rep (bug 1423) ================================================= 17:00:48 (1305676848) May 17 17:00:48 client-5 xinetd[3391]: EXIT: shell status=0 pid=7139 duration=0(sec) May 17 17:00:56 client-5 kernel: Lustre: 7187:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369101166445284 sent from lustre-MDT0000-mdc-ffff810318437400 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305676849] [real_sent 1305676849] [current 1305676856] [deadline 7s] [delay 0s] req@ffff81031dd81400 x1369101166445284/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 456/984 e 0 to 1 dl 1305676856 ref 2 fl Rpc:X/ffffffff/ffffffff rc 0/-1 May 17 17:00:56 client-5 kernel: Lustre: 7187:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 4 previous similar messages May 17 17:00:56 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff810318437400: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 17:00:56 client-5 kernel: Lustre: Skipped 4 previous similar messages May 17 17:00:56 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff810318437400: Connection restored to service lustre-MDT0000 using nid 192.168.4.128@o2ib. May 17 17:00:56 client-5 kernel: Lustre: Skipped 4 previous similar messages May 17 17:00:56 client-5 xinetd[3391]: START: shell pid=7250 from=192.168.4.5 May 17 17:00:56 client-5 rshd[7253]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 17:00:56 client-5 xinetd[3391]: EXIT: shell status=0 pid=7250 duration=0(sec) May 17 17:00:56 client-5 xinetd[3391]: START: shell pid=7423 from=192.168.4.5 May 17 17:00:56 client-5 rshd[7425]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == recovery-small test 9: pause bulk on OST \(bug 1420\) =============================================== 17:00:56 \(1305676856\)");echo XXRETCODE:$?' May 17 17:00:56 client-5 kernel: Lustre: DEBUG MARKER: == recovery-small test 9: pause bulk on OST (bug 1420) =============================================== 17:00:56 (1305676856) May 17 17:00:56 client-5 xinetd[3391]: EXIT: shell status=0 pid=7423 duration=0(sec) May 17 17:01:02 client-5 xinetd[3391]: START: shell pid=7592 from=192.168.4.5 May 17 17:01:02 client-5 rshd[7593]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 17:01:02 client-5 xinetd[3391]: EXIT: shell status=0 pid=7592 duration=0(sec) May 17 17:01:02 client-5 xinetd[3391]: START: shell pid=7764 from=192.168.4.5 May 17 17:01:02 client-5 rshd[7765]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == recovery-small test 10: finish request on server after client eviction \(bug 1521\) ================= 17:01:02 \(1305676862\)");echo XXRETCODE:$?' May 17 17:01:02 client-5 kernel: Lustre: DEBUG MARKER: == recovery-small test 10: finish request on server after client eviction (bug 1521) ================= 17:01:02 (1305676862) May 17 17:01:02 client-5 xinetd[3391]: EXIT: shell status=0 pid=7764 duration=0(sec) May 17 17:01:02 client-5 kernel: LustreError: 7816:0:(obd_support.h:456:obd_fail_check_set()) *** obd_fail_loc=305 *** May 17 17:01:02 client-5 kernel: LustreError: 4346:0:(obd_support.h:456:obd_fail_check_set()) *** obd_fail_loc=305 *** May 17 17:01:13 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The ldlm_enqueue operation failed with -107 May 17 17:01:13 client-5 kernel: LustreError: 167-0: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. May 17 17:01:13 client-5 kernel: LustreError: 7832:0:(mdc_locks.c:722:mdc_enqueue()) ldlm_cli_enqueue: -4 May 17 17:01:13 client-5 xinetd[3391]: START: shell pid=7902 from=192.168.4.5 May 17 17:01:13 client-5 rshd[7903]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 17:01:13 client-5 xinetd[3391]: EXIT: shell status=0 pid=7902 duration=0(sec) May 17 17:01:14 client-5 xinetd[3391]: START: shell pid=8074 from=192.168.4.5 May 17 17:01:14 client-5 rshd[8075]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == recovery-small test 11: wake up a thread waiting for completion after eviction \(b=2460\) =========== 17:01:14 \(1305676874\)");echo XXRETCODE:$?' May 17 17:01:14 client-5 kernel: Lustre: DEBUG MARKER: == recovery-small test 11: wake up a thread waiting for completion after eviction (b=2460) =========== 17:01:14 (1305676874) May 17 17:01:14 client-5 xinetd[3391]: EXIT: shell status=0 pid=8074 duration=0(sec) May 17 17:01:14 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 17:01:14 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 17:01:14 client-5 xinetd[3391]: START: shell pid=8221 from=192.168.4.5 May 17 17:01:14 client-5 rshd[8222]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 17:01:14 client-5 xinetd[3391]: EXIT: shell status=0 pid=8221 duration=0(sec) May 17 17:01:14 client-5 xinetd[3391]: START: shell pid=8392 from=192.168.4.5 May 17 17:01:14 client-5 rshd[8394]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == recovery-small test 12: recover from timed out resend in ptlrpcd \(b=2494\) ========================= 17:01:14 \(1305676874\)");echo XXRETCODE:$?' May 17 17:01:14 client-5 kernel: Lustre: DEBUG MARKER: == recovery-small test 12: recover from timed out resend in ptlrpcd (b=2494) ========================= 17:01:14 (1305676874) May 17 17:01:14 client-5 kernel: Lustre: DEBUG MARKER: multiop /mnt/lustre/f12 OS_c May 17 17:01:14 client-5 xinetd[3391]: EXIT: shell status=0 pid=8392 duration=0(sec) May 17 17:01:22 client-5 xinetd[3391]: START: shell pid=8504 from=192.168.4.5 May 17 17:01:22 client-5 rshd[8505]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 17:01:22 client-5 xinetd[3391]: EXIT: shell status=0 pid=8504 duration=0(sec) May 17 17:01:55 client-5 xinetd[3391]: START: shell pid=8684 from=192.168.4.5 May 17 17:01:55 client-5 rshd[8685]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == recovery-small test 13: mdc_readpage restart test \(bug 1138\) ====================================== 17:01:55 \(1305676915\)");echo XXRETCODE:$?' May 17 17:01:55 client-5 kernel: Lustre: DEBUG MARKER: == recovery-small test 13: mdc_readpage restart test (bug 1138) ====================================== 17:01:55 (1305676915) May 17 17:01:55 client-5 xinetd[3391]: EXIT: shell status=0 pid=8684 duration=0(sec) May 17 17:02:02 client-5 kernel: Lustre: 8729:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369101166445457 sent from lustre-MDT0000-mdc-ffff810318437400 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305676915] [real_sent 1305676915] [current 1305676922] [deadline 7s] [delay 0s] req@ffff81033214dc00 x1369101166445457/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:23/10 lens 416/408 e 0 to 1 dl 1305676922 ref 2 fl Rpc:X/ffffffff/ffffffff rc 0/-1 May 17 17:02:02 client-5 kernel: Lustre: 8729:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 1 previous similar message May 17 17:02:02 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff810318437400: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 17:02:02 client-5 kernel: Lustre: Skipped 2 previous similar messages May 17 17:02:02 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff810318437400: Connection restored to service lustre-MDT0000 using nid 192.168.4.128@o2ib. May 17 17:02:02 client-5 kernel: Lustre: Skipped 2 previous similar messages May 17 17:02:02 client-5 xinetd[3391]: START: shell pid=8796 from=192.168.4.5 May 17 17:02:02 client-5 rshd[8797]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 17:02:02 client-5 xinetd[3391]: EXIT: shell status=0 pid=8796 duration=0(sec) May 17 17:02:02 client-5 xinetd[3391]: START: shell pid=8968 from=192.168.4.5 May 17 17:02:03 client-5 rshd[8969]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == recovery-small test 14: mdc_readpage resend test \(bug 1138\) ======================================= 17:02:02 \(1305676922\)");echo XXRETCODE:$?' May 17 17:02:03 client-5 kernel: Lustre: DEBUG MARKER: == recovery-small test 14: mdc_readpage resend test (bug 1138) ======================================= 17:02:02 (1305676922) May 17 17:02:03 client-5 xinetd[3391]: EXIT: shell status=0 pid=8968 duration=1(sec) May 17 17:02:03 client-5 xinetd[3391]: START: shell pid=9078 from=192.168.4.5 May 17 17:02:03 client-5 rshd[9079]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 17:02:03 client-5 xinetd[3391]: EXIT: shell status=0 pid=9078 duration=0(sec) May 17 17:02:03 client-5 xinetd[3391]: START: shell pid=9250 from=192.168.4.5 May 17 17:02:03 client-5 rshd[9251]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == recovery-small test 15: failed open \(-ENOMEM\) ===================================================== 17:02:03 \(1305676923\)");echo XXRETCODE:$?' May 17 17:02:03 client-5 kernel: Lustre: DEBUG MARKER: == recovery-small test 15: failed open (-ENOMEM) ===================================================== 17:02:03 (1305676923) May 17 17:02:03 client-5 xinetd[3391]: EXIT: shell status=0 pid=9250 duration=0(sec) May 17 17:02:03 client-5 xinetd[3391]: START: shell pid=9350 from=192.168.4.5 May 17 17:02:03 client-5 rshd[9351]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 17:02:04 client-5 xinetd[3391]: EXIT: shell status=0 pid=9350 duration=1(sec) May 17 17:02:04 client-5 xinetd[3391]: START: shell pid=9525 from=192.168.4.5 May 17 17:02:04 client-5 rshd[9526]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == recovery-small test 16: timeout bulk put, don\'t evict client \(2732\) =============================== 17:02:04 \(1305676924\)");echo XXRETCODE:$?' May 17 17:02:04 client-5 kernel: Lustre: DEBUG MARKER: == recovery-small test 16: timeout bulk put, don't evict client (2732) =============================== 17:02:04 (1305676924) May 17 17:02:04 client-5 xinetd[3391]: EXIT: shell status=0 pid=9525 duration=0(sec) May 17 17:02:04 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 17:02:05 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 17:02:38 client-5 xinetd[3391]: START: shell pid=9697 from=192.168.4.5 May 17 17:02:38 client-5 rshd[9698]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 17:02:38 client-5 xinetd[3391]: EXIT: shell status=0 pid=9697 duration=0(sec) May 17 17:02:38 client-5 xinetd[3391]: START: shell pid=9869 from=192.168.4.5 May 17 17:02:38 client-5 rshd[9870]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == recovery-small test 17: timeout bulk get, don\'t evict client \(2732\) =============================== 17:02:38 \(1305676958\)");echo XXRETCODE:$?' May 17 17:02:38 client-5 kernel: Lustre: DEBUG MARKER: == recovery-small test 17: timeout bulk get, don't evict client (2732) =============================== 17:02:38 (1305676958) May 17 17:02:38 client-5 xinetd[3391]: EXIT: shell status=0 pid=9869 duration=0(sec) May 17 17:03:19 client-5 xinetd[3391]: START: shell pid=10061 from=192.168.4.5 May 17 17:03:19 client-5 rshd[10062]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 17:03:19 client-5 xinetd[3391]: EXIT: shell status=0 pid=10061 duration=0(sec) May 17 17:03:20 client-5 xinetd[3391]: START: shell pid=10233 from=192.168.4.5 May 17 17:03:20 client-5 rshd[10234]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == recovery-small test 18a: manual ost invalidate clears page cache immediately ====================== 17:03:20 \(1305677000\)");echo XXRETCODE:$?' May 17 17:03:20 client-5 kernel: Lustre: DEBUG MARKER: == recovery-small test 18a: manual ost invalidate clears page cache immediately ====================== 17:03:20 (1305677000) May 17 17:03:20 client-5 xinetd[3391]: EXIT: shell status=0 pid=10233 duration=0(sec) May 17 17:03:20 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 17:03:20 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 17:03:21 client-5 kernel: Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request May 17 17:03:21 client-5 kernel: LustreError: 167-0: This client was evicted by lustre-OST0001; in progress operations using this service will fail. May 17 17:03:22 client-5 xinetd[3391]: START: shell pid=10380 from=192.168.4.5 May 17 17:03:22 client-5 rshd[10381]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 17:03:22 client-5 xinetd[3391]: EXIT: shell status=0 pid=10380 duration=0(sec) May 17 17:03:22 client-5 xinetd[3391]: START: shell pid=10552 from=192.168.4.5 May 17 17:03:22 client-5 rshd[10553]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == recovery-small test 18b: eviction and reconnect clears page cache \(2766\) ========================== 17:03:22 \(1305677002\)");echo XXRETCODE:$?' May 17 17:03:22 client-5 kernel: Lustre: DEBUG MARKER: == recovery-small test 18b: eviction and reconnect clears page cache (2766) ========================== 17:03:22 (1305677002) May 17 17:03:22 client-5 xinetd[3391]: EXIT: shell status=0 pid=10552 duration=0(sec) May 17 17:03:22 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 17:03:22 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 17:03:32 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.129@o2ib. The obd_ping operation failed with -107 May 17 17:03:32 client-5 kernel: LustreError: 167-0: This client was evicted by lustre-OST0000; in progress operations using this service will fail. May 17 17:03:45 client-5 xinetd[3391]: START: shell pid=10735 from=192.168.4.5 May 17 17:03:45 client-5 rshd[10736]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 17:03:46 client-5 xinetd[3391]: EXIT: shell status=0 pid=10735 duration=1(sec) May 17 17:03:46 client-5 xinetd[3391]: START: shell pid=10907 from=192.168.4.5 May 17 17:03:46 client-5 rshd[10908]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == recovery-small test 18c: Dropped connect reply after eviction handing \(14755\) ===================== 17:03:46 \(1305677026\)");echo XXRETCODE:$?' May 17 17:03:46 client-5 kernel: Lustre: DEBUG MARKER: == recovery-small test 18c: Dropped connect reply after eviction handing (14755) ===================== 17:03:46 (1305677026) May 17 17:03:46 client-5 xinetd[3391]: EXIT: shell status=0 pid=10907 duration=0(sec) May 17 17:03:46 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 17:03:46 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 17:03:47 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.129@o2ib. The ost_statfs operation failed with -107 May 17 17:03:57 client-5 kernel: Lustre: 1454:0:(import.c:529:import_select_connection()) lustre-OST0000-osc-ffff810318437400: tried all connections, increasing latency to 6s May 17 17:03:57 client-5 kernel: Lustre: 1453:0:(import.c:885:ptlrpc_connect_interpret()) lustre-OST0000_UUID@192.168.4.129@o2ib changed server handle from 0xd9b2a6be77faa4f2 to 0xd9b2a6be77faa515 May 17 17:03:57 client-5 kernel: LustreError: 167-0: This client was evicted by lustre-OST0000; in progress operations using this service will fail. May 17 17:03:59 client-5 xinetd[3391]: START: shell pid=11095 from=192.168.4.5 May 17 17:03:59 client-5 rshd[11096]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 17:03:59 client-5 xinetd[3391]: EXIT: shell status=0 pid=11095 duration=0(sec) May 17 17:03:59 client-5 xinetd[3391]: START: shell pid=11267 from=192.168.4.5 May 17 17:03:59 client-5 rshd[11268]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == recovery-small test 19a: test expired_lock_main on mds \(2867\) ===================================== 17:03:59 \(1305677039\)");echo XXRETCODE:$?' May 17 17:04:00 client-5 kernel: Lustre: DEBUG MARKER: == recovery-small test 19a: test expired_lock_main on mds (2867) ===================================== 17:03:59 (1305677039) May 17 17:04:00 client-5 xinetd[3391]: EXIT: shell status=0 pid=11267 duration=1(sec) May 17 17:04:05 client-5 xinetd[3391]: START: shell pid=11394 from=192.168.4.5 May 17 17:04:05 client-5 rshd[11396]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 17:04:05 client-5 xinetd[3391]: EXIT: shell status=0 pid=11394 duration=0(sec) May 17 17:04:05 client-5 xinetd[3391]: START: shell pid=11567 from=192.168.4.5 May 17 17:04:05 client-5 rshd[11568]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == recovery-small test 19b: test expired_lock_main on ost \(2867\) ===================================== 17:04:05 \(1305677045\)");echo XXRETCODE:$?' May 17 17:04:05 client-5 kernel: Lustre: DEBUG MARKER: == recovery-small test 19b: test expired_lock_main on ost (2867) ===================================== 17:04:05 (1305677045) May 17 17:04:05 client-5 xinetd[3391]: EXIT: shell status=0 pid=11567 duration=0(sec) May 17 17:04:05 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 17:04:05 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 17:04:05 client-5 xinetd[3391]: START: shell pid=11714 from=192.168.4.5 May 17 17:04:05 client-5 rshd[11715]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 17:04:05 client-5 xinetd[3391]: EXIT: shell status=0 pid=11714 duration=0(sec) May 17 17:04:06 client-5 xinetd[3391]: START: shell pid=11886 from=192.168.4.5 May 17 17:04:06 client-5 rshd[11887]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == recovery-small test 20a: ldlm_handle_enqueue error \(should return error\) ========================== 17:04:06 \(1305677046\)");echo XXRETCODE:$?' May 17 17:04:06 client-5 kernel: Lustre: DEBUG MARKER: == recovery-small test 20a: ldlm_handle_enqueue error (should return error) ========================== 17:04:06 (1305677046) May 17 17:04:06 client-5 xinetd[3391]: EXIT: shell status=0 pid=11886 duration=0(sec) May 17 17:04:06 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 17:04:06 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 17:04:06 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.129@o2ib. The ldlm_enqueue operation failed with -12 May 17 17:04:06 client-5 xinetd[3391]: START: shell pid=12029 from=192.168.4.5 May 17 17:04:06 client-5 rshd[12030]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 17:04:06 client-5 xinetd[3391]: EXIT: shell status=0 pid=12029 duration=0(sec) May 17 17:04:06 client-5 xinetd[3391]: START: shell pid=12201 from=192.168.4.5 May 17 17:04:06 client-5 rshd[12202]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == recovery-small test 20b: ldlm_handle_enqueue error \(should return error\) ========================== 17:04:06 \(1305677046\)");echo XXRETCODE:$?' May 17 17:04:06 client-5 kernel: Lustre: DEBUG MARKER: == recovery-small test 20b: ldlm_handle_enqueue error (should return error) ========================== 17:04:06 (1305677046) May 17 17:04:06 client-5 xinetd[3391]: EXIT: shell status=0 pid=12201 duration=0(sec) May 17 17:04:06 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 17:04:06 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 17:04:07 client-5 xinetd[3391]: START: shell pid=12342 from=192.168.4.5 May 17 17:04:07 client-5 rshd[12343]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 17:04:07 client-5 xinetd[3391]: EXIT: shell status=0 pid=12342 duration=0(sec) May 17 17:04:07 client-5 xinetd[3391]: START: shell pid=12514 from=192.168.4.5 May 17 17:04:07 client-5 rshd[12515]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == recovery-small test 21a: drop close request while close and open are both in flight =============== 17:04:07 \(1305677047\)");echo XXRETCODE:$?' May 17 17:04:07 client-5 kernel: Lustre: DEBUG MARKER: == recovery-small test 21a: drop close request while close and open are both in flight =============== 17:04:07 (1305677047) May 17 17:04:07 client-5 xinetd[3391]: EXIT: shell status=0 pid=12514 duration=0(sec) May 17 17:04:08 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc start May 17 17:04:08 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc stop May 17 17:04:15 client-5 kernel: Lustre: 12552:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369101166446868 sent from lustre-MDT0000-mdc-ffff810318437400 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305677048] [real_sent 1305677048] [current 1305677055] [deadline 7s] [delay 0s] req@ffff81031eaaf400 x1369101166446868/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:23/10 lens 360/1048 e 0 to 1 dl 1305677055 ref 2 fl Rpc:X/ffffffff/ffffffff rc 0/-1 May 17 17:04:15 client-5 kernel: Lustre: 12552:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 2 previous similar messages May 17 17:04:15 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff810318437400: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 17:04:15 client-5 kernel: Lustre: Skipped 4 previous similar messages May 17 17:04:15 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff810318437400: Connection restored to service lustre-MDT0000 using nid 192.168.4.128@o2ib. May 17 17:04:15 client-5 kernel: Lustre: Skipped 4 previous similar messages May 17 17:04:16 client-5 xinetd[3391]: START: shell pid=12654 from=192.168.4.5 May 17 17:04:16 client-5 rshd[12657]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 17:04:16 client-5 xinetd[3391]: EXIT: shell status=0 pid=12654 duration=0(sec) May 17 17:04:16 client-5 xinetd[3391]: START: shell pid=12828 from=192.168.4.5 May 17 17:04:16 client-5 rshd[12829]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == recovery-small test 21b: drop open request while close and open are both in flight ================ 17:04:16 \(1305677056\)");echo XXRETCODE:$?' May 17 17:04:16 client-5 kernel: Lustre: DEBUG MARKER: == recovery-small test 21b: drop open request while close and open are both in flight ================ 17:04:16 (1305677056) May 17 17:04:16 client-5 xinetd[3391]: EXIT: shell status=0 pid=12828 duration=0(sec) May 17 17:04:17 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc start May 17 17:04:17 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc stop May 17 17:05:00 client-5 xinetd[3391]: START: shell pid=12957 from=192.168.4.5 May 17 17:05:00 client-5 rshd[12959]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 17:05:00 client-5 xinetd[3391]: EXIT: shell status=0 pid=12957 duration=0(sec) May 17 17:05:01 client-5 xinetd[3391]: START: shell pid=13130 from=192.168.4.5 May 17 17:05:01 client-5 rshd[13131]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == recovery-small test 21c: drop both request while close and open are both in flight ================ 17:05:01 \(1305677101\)");echo XXRETCODE:$?' May 17 17:05:01 client-5 kernel: Lustre: DEBUG MARKER: == recovery-small test 21c: drop both request while close and open are both in flight ================ 17:05:01 (1305677101) May 17 17:05:01 client-5 xinetd[3391]: EXIT: shell status=0 pid=13130 duration=0(sec) May 17 17:05:04 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc start May 17 17:05:04 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc stop May 17 17:05:11 client-5 xinetd[3391]: START: shell pid=13270 from=192.168.4.5 May 17 17:05:11 client-5 rshd[13273]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 17:05:11 client-5 xinetd[3391]: EXIT: shell status=0 pid=13270 duration=0(sec) May 17 17:05:12 client-5 xinetd[3391]: START: shell pid=13444 from=192.168.4.5 May 17 17:05:12 client-5 rshd[13445]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == recovery-small test 21d: drop close reply while close and open are both in flight ================= 17:05:12 \(1305677112\)");echo XXRETCODE:$?' May 17 17:05:12 client-5 kernel: Lustre: DEBUG MARKER: == recovery-small test 21d: drop close reply while close and open are both in flight ================= 17:05:12 (1305677112) May 17 17:05:12 client-5 xinetd[3391]: EXIT: shell status=0 pid=13444 duration=0(sec) May 17 17:05:13 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc start May 17 17:05:13 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc stop May 17 17:05:20 client-5 xinetd[3391]: START: shell pid=13585 from=192.168.4.5 May 17 17:05:20 client-5 rshd[13587]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 17:05:20 client-5 xinetd[3391]: EXIT: shell status=0 pid=13585 duration=0(sec) May 17 17:05:21 client-5 xinetd[3391]: START: shell pid=13758 from=192.168.4.5 May 17 17:05:21 client-5 rshd[13759]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == recovery-small test 21e: drop open reply while close and open are both in flight ================== 17:05:20 \(1305677120\)");echo XXRETCODE:$?' May 17 17:05:21 client-5 kernel: Lustre: DEBUG MARKER: == recovery-small test 21e: drop open reply while close and open are both in flight ================== 17:05:20 (1305677120) May 17 17:05:21 client-5 xinetd[3391]: EXIT: shell status=0 pid=13758 duration=0(sec) May 17 17:05:22 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc start May 17 17:05:22 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc stop May 17 17:06:05 client-5 xinetd[3391]: START: shell pid=13884 from=192.168.4.5 May 17 17:06:05 client-5 rshd[13886]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 17:06:05 client-5 xinetd[3391]: EXIT: shell status=0 pid=13884 duration=0(sec) May 17 17:06:05 client-5 xinetd[3391]: START: shell pid=14057 from=192.168.4.5 May 17 17:06:05 client-5 rshd[14058]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == recovery-small test 21f: drop both reply while close and open are both in flight ================== 17:06:05 \(1305677165\)");echo XXRETCODE:$?' May 17 17:06:05 client-5 kernel: Lustre: DEBUG MARKER: == recovery-small test 21f: drop both reply while close and open are both in flight ================== 17:06:05 (1305677165) May 17 17:06:05 client-5 xinetd[3391]: EXIT: shell status=0 pid=14057 duration=0(sec) May 17 17:06:07 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc start May 17 17:06:07 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc stop May 17 17:06:14 client-5 xinetd[3391]: START: shell pid=14203 from=192.168.4.5 May 17 17:06:14 client-5 rshd[14204]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 17:06:14 client-5 xinetd[3391]: EXIT: shell status=0 pid=14203 duration=0(sec) May 17 17:06:14 client-5 xinetd[3391]: START: shell pid=14375 from=192.168.4.5 May 17 17:06:14 client-5 rshd[14376]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == recovery-small test 21g: drop open reply and close request while close and open are both in flight ====================================================================================================== 17:06:14 \(1305677174\)");echo XXRETCODE:$?' May 17 17:06:14 client-5 kernel: Lustre: DEBUG MARKER: == recovery-small test 21g: drop open reply and close request while close and open are both in flight ====================================================================================================== 17:06:14 (1305677174) May 17 17:06:14 client-5 xinetd[3391]: EXIT: shell status=0 pid=14375 duration=0(sec) May 17 17:06:16 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc start May 17 17:06:16 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc stop May 17 17:06:23 client-5 xinetd[3391]: START: shell pid=14515 from=192.168.4.5 May 17 17:06:23 client-5 rshd[14518]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 17:06:23 client-5 xinetd[3391]: EXIT: shell status=0 pid=14515 duration=0(sec) May 17 17:06:23 client-5 xinetd[3391]: START: shell pid=14689 from=192.168.4.5 May 17 17:06:23 client-5 rshd[14690]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == recovery-small test 21h: drop open request and close reply while close and open are both in flight ====================================================================================================== 17:06:23 \(1305677183\)");echo XXRETCODE:$?' May 17 17:06:23 client-5 kernel: Lustre: DEBUG MARKER: == recovery-small test 21h: drop open request and close reply while close and open are both in flight ====================================================================================================== 17:06:23 (1305677183) May 17 17:06:23 client-5 xinetd[3391]: EXIT: shell status=0 pid=14689 duration=0(sec) May 17 17:06:25 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc start May 17 17:06:25 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc stop May 17 17:06:32 client-5 xinetd[3391]: START: shell pid=14829 from=192.168.4.5 May 17 17:06:32 client-5 rshd[14832]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 17:06:32 client-5 xinetd[3391]: EXIT: shell status=0 pid=14829 duration=0(sec) May 17 17:06:32 client-5 xinetd[3391]: START: shell pid=15002 from=192.168.4.5 May 17 17:06:32 client-5 rshd[15004]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == recovery-small test 22: drop close request and do mknod =========================================== 17:06:32 \(1305677192\)");echo XXRETCODE:$?' May 17 17:06:32 client-5 kernel: Lustre: DEBUG MARKER: == recovery-small test 22: drop close request and do mknod =========================================== 17:06:32 (1305677192) May 17 17:06:32 client-5 xinetd[3391]: EXIT: shell status=0 pid=15002 duration=0(sec) May 17 17:06:33 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc start May 17 17:06:33 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc stop May 17 17:06:39 client-5 xinetd[3391]: START: shell pid=15123 from=192.168.4.5 May 17 17:06:39 client-5 rshd[15124]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 17:06:39 client-5 xinetd[3391]: EXIT: shell status=0 pid=15123 duration=0(sec) May 17 17:06:40 client-5 xinetd[3391]: START: shell pid=15295 from=192.168.4.5 May 17 17:06:40 client-5 rshd[15296]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == recovery-small test 23: client hang when close a file after mds crash ============================= 17:06:40 \(1305677200\)");echo XXRETCODE:$?' May 17 17:06:40 client-5 kernel: Lustre: DEBUG MARKER: == recovery-small test 23: client hang when close a file after mds crash ============================= 17:06:40 (1305677200) May 17 17:06:40 client-5 xinetd[3391]: EXIT: shell status=0 pid=15295 duration=0(sec) May 17 17:06:56 client-5 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail. May 17 17:07:01 client-5 kernel: Lustre: 1454:0:(import.c:529:import_select_connection()) lustre-MDT0000-mdc-ffff810318437400: tried all connections, increasing latency to 6s May 17 17:07:04 client-5 xinetd[3391]: START: shell pid=15474 from=192.168.4.5 May 17 17:07:04 client-5 rshd[15475]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 17:07:06 client-5 kernel: Lustre: 1454:0:(import.c:529:import_select_connection()) MGC192.168.4.128@o2ib: tried all connections, increasing latency to 6s May 17 17:07:06 client-5 kernel: Lustre: 1453:0:(import.c:885:ptlrpc_connect_interpret()) MGS@MGC192.168.4.128@o2ib_0 changed server handle from 0xfff98862bf2aa5c5 to 0xfff98862bf2acc30 May 17 17:07:06 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 17:07:16 client-5 kernel: Lustre: 1454:0:(import.c:529:import_select_connection()) lustre-MDT0000-mdc-ffff810318437400: tried all connections, increasing latency to 11s May 17 17:07:16 client-5 xinetd[3391]: EXIT: shell status=0 pid=15474 duration=12(sec) May 17 17:07:16 client-5 xinetd[3391]: START: shell pid=15568 from=192.168.4.5 May 17 17:07:16 client-5 rshd[15569]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 17:07:16 client-5 xinetd[3391]: EXIT: shell status=0 pid=15568 duration=0(sec) May 17 17:07:17 client-5 xinetd[3391]: START: shell pid=15731 from=192.168.4.5 May 17 17:07:17 client-5 rshd[15732]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark SKIP: recovery-small test_24 skipping ALWAYS excluded test 24");echo XXRETCODE:$?' May 17 17:07:17 client-5 kernel: Lustre: DEBUG MARKER: SKIP: recovery-small test_24 skipping ALWAYS excluded test 24 May 17 17:07:17 client-5 xinetd[3391]: EXIT: shell status=0 pid=15731 duration=0(sec) May 17 17:07:17 client-5 xinetd[3391]: START: shell pid=15836 from=192.168.4.5 May 17 17:07:17 client-5 rshd[15837]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == recovery-small test 26a: evict dead exports ======================================================= 17:07:17 \(1305677237\)");echo XXRETCODE:$?' May 17 17:07:17 client-5 kernel: Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports ======================================================= 17:07:17 (1305677237) May 17 17:07:17 client-5 xinetd[3391]: EXIT: shell status=0 pid=15836 duration=0(sec) May 17 17:07:21 client-5 kernel: LustreError: 1452:0:(obd_support.h:456:obd_fail_check_set()) *** obd_fail_loc=505 *** May 17 17:07:22 client-5 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail. May 17 17:07:22 client-5 kernel: LustreError: 1453:0:(obd_support.h:456:obd_fail_check_set()) *** obd_fail_loc=505 *** May 17 17:07:22 client-5 kernel: LustreError: 1453:0:(obd_support.h:456:obd_fail_check_set()) Skipped 7 previous similar messages May 17 17:07:23 client-5 kernel: LustreError: 1453:0:(obd_support.h:456:obd_fail_check_set()) *** obd_fail_loc=505 *** May 17 17:07:28 client-5 kernel: Lustre: 1454:0:(import.c:529:import_select_connection()) MGC192.168.4.128@o2ib: tried all connections, increasing latency to 11s May 17 17:07:28 client-5 kernel: LustreError: 1453:0:(obd_support.h:456:obd_fail_check_set()) *** obd_fail_loc=505 *** May 17 17:07:28 client-5 kernel: LustreError: 1453:0:(obd_support.h:456:obd_fail_check_set()) Skipped 6 previous similar messages May 17 17:07:33 client-5 kernel: LustreError: 1453:0:(obd_support.h:456:obd_fail_check_set()) *** obd_fail_loc=505 *** May 17 17:07:33 client-5 kernel: LustreError: 1453:0:(obd_support.h:456:obd_fail_check_set()) Skipped 7 previous similar messages May 17 17:07:38 client-5 kernel: Lustre: 1454:0:(import.c:529:import_select_connection()) MGC192.168.4.128@o2ib: tried all connections, increasing latency to 16s May 17 17:07:38 client-5 kernel: Lustre: 1454:0:(import.c:529:import_select_connection()) Skipped 7 previous similar messages May 17 17:07:38 client-5 kernel: LustreError: 1453:0:(obd_support.h:456:obd_fail_check_set()) *** obd_fail_loc=505 *** May 17 17:07:38 client-5 kernel: LustreError: 1453:0:(obd_support.h:456:obd_fail_check_set()) Skipped 7 previous similar messages May 17 17:07:48 client-5 kernel: Lustre: 1454:0:(import.c:529:import_select_connection()) MGC192.168.4.128@o2ib: tried all connections, increasing latency to 21s May 17 17:07:48 client-5 kernel: Lustre: 1454:0:(import.c:529:import_select_connection()) Skipped 7 previous similar messages May 17 17:07:48 client-5 kernel: LustreError: 1453:0:(obd_support.h:456:obd_fail_check_set()) *** obd_fail_loc=505 *** May 17 17:07:48 client-5 kernel: LustreError: 1453:0:(obd_support.h:456:obd_fail_check_set()) Skipped 15 previous similar messages May 17 17:08:03 client-5 xinetd[3391]: START: shell pid=16094 from=192.168.4.5 May 17 17:08:03 client-5 rshd[16095]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 17:08:03 client-5 xinetd[3391]: EXIT: shell status=0 pid=16094 duration=0(sec) May 17 17:08:04 client-5 xinetd[3391]: START: shell pid=16266 from=192.168.4.5 May 17 17:08:04 client-5 rshd[16267]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == recovery-small test 26b: evict dead exports ======================================================= 17:08:04 \(1305677284\)");echo XXRETCODE:$?' May 17 17:08:04 client-5 kernel: Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports ======================================================= 17:08:04 (1305677284) May 17 17:08:04 client-5 xinetd[3391]: EXIT: shell status=0 pid=16266 duration=0(sec) May 17 17:08:05 client-5 xinetd[3391]: START: shell pid=16344 from=192.168.4.5 May 17 17:08:05 client-5 rshd[16345]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 17:08:08 client-5 kernel: Lustre: 1454:0:(import.c:529:import_select_connection()) MGC192.168.4.128@o2ib: tried all connections, increasing latency to 21s May 17 17:08:08 client-5 kernel: Lustre: 1454:0:(import.c:529:import_select_connection()) Skipped 23 previous similar messages May 17 17:08:08 client-5 kernel: Lustre: 1453:0:(import.c:885:ptlrpc_connect_interpret()) MGS@MGC192.168.4.128@o2ib_0 changed server handle from 0xfff98862bf2acc30 to 0xfff98862bf2acca0 May 17 17:08:08 client-5 kernel: LustreError: 167-0: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. May 17 17:08:08 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 17:08:08 client-5 kernel: LustreError: 16377:0:(lmv_obd.c:1198:lmv_statfs()) can't stat MDS #0 (lustre-MDT0000-mdc-ffff810318437400), error -4 May 17 17:08:08 client-5 kernel: LustreError: 16377:0:(llite_lib.c:1370:ll_statfs_internal()) md_statfs fails: rc = -4 May 17 17:08:08 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 17:08:08 client-5 xinetd[3391]: EXIT: shell status=0 pid=16344 duration=3(sec) May 17 17:08:08 client-5 kernel: Lustre: 16394:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff8102f4217000->192.168.4.128@o2ib netid 50000: select flavor null May 17 17:08:08 client-5 kernel: Lustre: 16394:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 5 previous similar messages May 17 17:08:08 client-5 kernel: Lustre: Client lustre-client has started May 17 17:08:08 client-5 kernel: Lustre: 16402:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 17:08:08 client-5 kernel: Lustre: 16402:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 3 previous similar messages May 17 17:08:10 client-5 kernel: Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request May 17 17:08:10 client-5 kernel: Lustre: client ffff8102f4217000 umount complete May 17 17:09:01 client-5 xinetd[3391]: START: shell pid=16593 from=192.168.4.5 May 17 17:09:01 client-5 rshd[16594]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 17:09:01 client-5 xinetd[3391]: EXIT: shell status=0 pid=16593 duration=0(sec) May 17 17:09:02 client-5 xinetd[3391]: START: shell pid=16763 from=192.168.4.5 May 17 17:09:02 client-5 rshd[16766]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == recovery-small test 27: fail LOV while using OSC\'s ================================================ 17:09:02 \(1305677342\)");echo XXRETCODE:$?' May 17 17:09:02 client-5 kernel: Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ================================================ 17:09:02 (1305677342) May 17 17:09:02 client-5 xinetd[3391]: EXIT: shell status=0 pid=16763 duration=0(sec) May 17 17:09:03 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_reint operation failed with -19 May 17 17:09:03 client-5 kernel: LustreError: Skipped 1 previous similar message May 17 17:09:03 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff810318437400: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 17:09:03 client-5 kernel: Lustre: Skipped 16 previous similar messages May 17 17:09:23 client-5 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail. May 17 17:09:23 client-5 kernel: Lustre: 1453:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf2acca0 to 0xfff98862bf2b0d60 May 17 17:09:23 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 17:09:23 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Connection restored to service MGS using nid 192.168.4.128@o2ib. May 17 17:09:23 client-5 kernel: Lustre: Skipped 18 previous similar messages May 17 17:09:29 client-5 kernel: Lustre: 1453:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369101166448788 sent from lustre-MDT0000-mdc-ffff810318437400 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305677343] [real_sent 1305677343] [current 1305677369] [deadline 26s] [delay 0s] req@ffff8102efe90800 x1369101166448788/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 368/392 e 0 to 1 dl 1305677369 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 17:09:29 client-5 kernel: Lustre: 1453:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 93 previous similar messages May 17 17:09:33 client-5 kernel: Lustre: 1454:0:(import.c:529:import_select_connection()) lustre-MDT0000-mdc-ffff810318437400: tried all connections, increasing latency to 21s May 17 17:09:33 client-5 kernel: Lustre: 1454:0:(import.c:529:import_select_connection()) Skipped 7 previous similar messages May 17 17:09:48 client-5 kernel: LustreError: 16805:0:(client.c:1896:ptlrpc_interrupted_set()) INTERRUPTED SET ffff81031eaaf740 May 17 17:09:48 client-5 kernel: LustreError: 16805:0:(file.c:155:ll_close_inode_openhandle()) inode 144115238810157266 mdc close failed: rc = -4 May 17 17:09:48 client-5 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail. May 17 17:09:48 client-5 kernel: Lustre: 1453:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf2b0d60 to 0xfff98862bf2b0e78 May 17 17:09:48 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 17:10:04 client-5 xinetd[3391]: START: shell pid=17116 from=192.168.4.5 May 17 17:10:04 client-5 rshd[17117]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 17:10:04 client-5 xinetd[3391]: EXIT: shell status=0 pid=17116 duration=0(sec) May 17 17:10:04 client-5 xinetd[3391]: START: shell pid=17288 from=192.168.4.5 May 17 17:10:04 client-5 rshd[17289]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == recovery-small test 28: handle error adding new clients \(bug 6086\) ================================ 17:10:04 \(1305677404\)");echo XXRETCODE:$?' May 17 17:10:04 client-5 kernel: Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) ================================ 17:10:04 (1305677404) May 17 17:10:04 client-5 xinetd[3391]: EXIT: shell status=0 pid=17288 duration=0(sec) May 17 17:10:04 client-5 kernel: LustreError: 17340:0:(obd_support.h:456:obd_fail_check_set()) *** obd_fail_loc=305 *** May 17 17:10:04 client-5 kernel: LustreError: 4345:0:(obd_support.h:456:obd_fail_check_set()) *** obd_fail_loc=305 *** May 17 17:10:16 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_statfs operation failed with -107 May 17 17:10:16 client-5 kernel: LustreError: Skipped 2 previous similar messages May 17 17:10:23 client-5 kernel: LustreError: 167-0: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. May 17 17:10:23 client-5 kernel: LustreError: Skipped 6 previous similar messages May 17 17:10:23 client-5 kernel: LustreError: 17356:0:(lmv_obd.c:1198:lmv_statfs()) can't stat MDS #0 (lustre-MDT0000-mdc-ffff810318437400), error -4 May 17 17:10:23 client-5 kernel: LustreError: 17356:0:(llite_lib.c:1370:ll_statfs_internal()) md_statfs fails: rc = -4 May 17 17:10:53 client-5 xinetd[3391]: START: shell pid=17481 from=192.168.4.5 May 17 17:10:53 client-5 rshd[17482]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 17:10:53 client-5 xinetd[3391]: EXIT: shell status=0 pid=17481 duration=0(sec) May 17 17:10:53 client-5 xinetd[3391]: START: shell pid=17569 from=192.168.4.5 May 17 17:10:53 client-5 rshd[17572]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 17:10:53 client-5 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail. May 17 17:10:53 client-5 kernel: Lustre: 1453:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf2b0e78 to 0xfff98862bf2b10d9 May 17 17:10:53 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 17:10:53 client-5 xinetd[3391]: EXIT: shell status=0 pid=17569 duration=0(sec) May 17 17:10:54 client-5 xinetd[3391]: START: shell pid=17745 from=192.168.4.5 May 17 17:10:54 client-5 rshd[17746]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == recovery-small test 29a: error adding new clients doesn\'t cause LBUG \(bug 22273\) ================== 17:10:54 \(1305677454\)");echo XXRETCODE:$?' May 17 17:10:54 client-5 kernel: Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) ================== 17:10:54 (1305677454) May 17 17:10:54 client-5 xinetd[3391]: EXIT: shell status=0 pid=17745 duration=0(sec) May 17 17:11:02 client-5 xinetd[3391]: START: shell pid=17866 from=192.168.4.5 May 17 17:11:02 client-5 rshd[17867]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 17:11:03 client-5 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail. May 17 17:11:03 client-5 kernel: LustreError: 17608:0:(client.c:1057:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@ffff81015bc69800 x1369101166448998/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:/ffffffff/ffffffff rc 0/-1 May 17 17:11:03 client-5 kernel: LustreError: 17608:0:(client.c:1057:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@ffff81015bc69800 x1369101166448999/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:/ffffffff/ffffffff rc 0/-1 May 17 17:11:03 client-5 kernel: Lustre: 1453:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf2b10d9 to 0xfff98862bf2b11ea May 17 17:11:08 client-5 kernel: Lustre: 1454:0:(import.c:529:import_select_connection()) lustre-MDT0000-mdc-ffff810318437400: tried all connections, increasing latency to 21s May 17 17:11:08 client-5 kernel: Lustre: 1454:0:(import.c:529:import_select_connection()) Skipped 2 previous similar messages May 17 17:11:08 client-5 kernel: LustreError: 167-0: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. May 17 17:11:08 client-5 kernel: LustreError: 17899:0:(lmv_obd.c:1198:lmv_statfs()) can't stat MDS #0 (lustre-MDT0000-mdc-ffff810318437400), error -4 May 17 17:11:08 client-5 kernel: LustreError: 17899:0:(llite_lib.c:1370:ll_statfs_internal()) md_statfs fails: rc = -4 May 17 17:11:08 client-5 xinetd[3391]: EXIT: shell status=0 pid=17866 duration=6(sec) May 17 17:11:09 client-5 xinetd[3391]: START: shell pid=17914 from=192.168.4.5 May 17 17:11:09 client-5 rshd[17915]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 17:11:10 client-5 xinetd[3391]: EXIT: shell status=0 pid=17914 duration=1(sec) May 17 17:11:11 client-5 xinetd[3391]: START: shell pid=18005 from=192.168.4.5 May 17 17:11:11 client-5 rshd[18007]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 17:11:11 client-5 xinetd[3391]: EXIT: shell status=0 pid=18005 duration=0(sec) May 17 17:11:11 client-5 xinetd[3391]: START: shell pid=18177 from=192.168.4.5 May 17 17:11:11 client-5 rshd[18179]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == recovery-small test 29b: error adding new clients doesn\'t cause LBUG \(bug 22273\) ================== 17:11:11 \(1305677471\)");echo XXRETCODE:$?' May 17 17:11:11 client-5 kernel: Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ================== 17:11:11 (1305677471) May 17 17:11:11 client-5 xinetd[3391]: EXIT: shell status=0 pid=18177 duration=0(sec) May 17 17:11:15 client-5 xinetd[3391]: START: shell pid=18298 from=192.168.4.5 May 17 17:11:15 client-5 rshd[18300]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 17:11:23 client-5 xinetd[3391]: EXIT: shell status=0 pid=18298 duration=8(sec) May 17 17:11:24 client-5 xinetd[3391]: START: shell pid=18342 from=192.168.4.5 May 17 17:11:24 client-5 rshd[18343]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 17:11:25 client-5 xinetd[3391]: EXIT: shell status=0 pid=18342 duration=1(sec) May 17 17:11:26 client-5 xinetd[3391]: START: shell pid=18434 from=192.168.4.5 May 17 17:11:26 client-5 rshd[18436]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 17:11:26 client-5 xinetd[3391]: EXIT: shell status=0 pid=18434 duration=0(sec) May 17 17:11:26 client-5 xinetd[3391]: START: shell pid=18606 from=192.168.4.5 May 17 17:11:26 client-5 rshd[18608]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == recovery-small test 50: failover MDS under load =================================================== 17:11:26 \(1305677486\)");echo XXRETCODE:$?' May 17 17:11:26 client-5 kernel: Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load =================================================== 17:11:26 (1305677486) May 17 17:11:26 client-5 xinetd[3391]: EXIT: shell status=0 pid=18606 duration=0(sec) May 17 17:11:53 client-5 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail. May 17 17:11:53 client-5 kernel: Lustre: 1453:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf2b11ea to 0xfff98862bf2d235f May 17 17:11:53 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 17:11:53 client-5 kernel: Lustre: Skipped 1 previous similar message May 17 17:11:55 client-5 xinetd[3391]: START: shell pid=18769 from=192.168.4.5 May 17 17:11:55 client-5 rshd[18770]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 17:12:13 client-5 xinetd[3391]: EXIT: shell status=0 pid=18769 duration=18(sec) May 17 17:13:14 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_close operation failed with -19 May 17 17:13:14 client-5 kernel: LustreError: Skipped 11 previous similar messages May 17 17:13:32 client-5 xinetd[3391]: START: shell pid=18928 from=192.168.4.5 May 17 17:13:32 client-5 rshd[18929]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 17:13:33 client-5 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail. May 17 17:13:33 client-5 kernel: Lustre: 1453:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf2d235f to 0xfff98862bf391379 May 17 17:13:33 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 17:13:43 client-5 kernel: Lustre: 1454:0:(import.c:529:import_select_connection()) lustre-MDT0000-mdc-ffff810318437400: tried all connections, increasing latency to 21s May 17 17:13:43 client-5 kernel: Lustre: 1454:0:(import.c:529:import_select_connection()) Skipped 3 previous similar messages May 17 17:13:43 client-5 xinetd[3391]: EXIT: shell status=0 pid=18928 duration=11(sec) May 17 17:15:02 client-5 xinetd[3391]: START: shell pid=19094 from=192.168.4.5 May 17 17:15:02 client-5 rshd[19095]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 17:15:03 client-5 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail. May 17 17:15:03 client-5 kernel: Lustre: 1453:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf391379 to 0xfff98862bf44ae9e May 17 17:15:03 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 17:15:23 client-5 xinetd[3391]: EXIT: shell status=0 pid=19094 duration=21(sec) May 17 17:15:43 client-5 xinetd[3391]: START: shell pid=19189 from=192.168.4.5 May 17 17:15:43 client-5 rshd[19190]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 17:15:44 client-5 xinetd[3391]: EXIT: shell status=0 pid=19189 duration=1(sec) May 17 17:15:44 client-5 xinetd[3391]: START: shell pid=19361 from=192.168.4.5 May 17 17:15:44 client-5 rshd[19362]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == recovery-small test 51: failover MDS during recovery ============================================== 17:15:44 \(1305677744\)");echo XXRETCODE:$?' May 17 17:15:44 client-5 kernel: Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery ============================================== 17:15:44 (1305677744) May 17 17:15:44 client-5 xinetd[3391]: EXIT: shell status=0 pid=19361 duration=0(sec) May 17 17:16:03 client-5 kernel: Lustre: 1453:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf44ae9e to 0xfff98862bf48cb7d May 17 17:16:23 client-5 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail. May 17 17:16:23 client-5 kernel: LustreError: Skipped 1 previous similar message May 17 17:16:23 client-5 kernel: LustreError: 19129:0:(client.c:1057:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@ffff8103192f3c00 x1369101166588333/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:/ffffffff/ffffffff rc 0/-1 May 17 17:16:23 client-5 kernel: Lustre: 1453:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf48cb7d to 0xfff98862bf48cc8e May 17 17:16:24 client-5 kernel: LustreError: 1453:0:(client.c:2570:ptlrpc_replay_interpret()) @@@ status -116, old was 0 req@ffff8103072cdc00 x1369101166588272/t34359748853(34359748853) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:23/10 lens 360/424 e 0 to 0 dl 1305677790 ref 2 fl Interpret:R/ffffffff/ffffffff rc -116/-1 May 17 17:16:50 client-5 kernel: Lustre: 1453:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf48cc8e to 0xfff98862bf497f23 May 17 17:17:12 client-5 kernel: Lustre: 1453:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf497f23 to 0xfff98862bf49c729 May 17 17:17:12 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 17:17:12 client-5 kernel: Lustre: Skipped 3 previous similar messages May 17 17:17:33 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The ldlm_enqueue operation failed with -19 May 17 17:17:33 client-5 kernel: LustreError: Skipped 12 previous similar messages May 17 17:18:04 client-5 kernel: Lustre: 1454:0:(import.c:529:import_select_connection()) MGC192.168.4.128@o2ib: tried all connections, increasing latency to 11s May 17 17:18:04 client-5 kernel: Lustre: 1454:0:(import.c:529:import_select_connection()) Skipped 9 previous similar messages May 17 17:18:04 client-5 kernel: Lustre: 1453:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf49c729 to 0xfff98862bf4bc684 May 17 17:18:04 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Connection restored to service MGS using nid 192.168.4.128@o2ib. May 17 17:18:04 client-5 kernel: Lustre: Skipped 21 previous similar messages May 17 17:18:05 client-5 kernel: LustreError: 1453:0:(client.c:2570:ptlrpc_replay_interpret()) @@@ status -116, old was 0 req@ffff810326225400 x1369101166602265/t47244645332(47244645332) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:23/10 lens 360/424 e 0 to 0 dl 1305677911 ref 2 fl Interpret:R/ffffffff/ffffffff rc -116/-1 May 17 17:18:16 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff810318437400: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 17:18:16 client-5 kernel: Lustre: Skipped 11 previous similar messages May 17 17:18:34 client-5 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail. May 17 17:18:34 client-5 kernel: LustreError: Skipped 3 previous similar messages May 17 17:18:34 client-5 kernel: Lustre: 1453:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf4bc684 to 0xfff98862bf4df74f May 17 17:18:42 client-5 kernel: Lustre: 1453:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369101166612551 sent from lustre-MDT0000-mdc-ffff810318437400 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305677896] [real_sent 1305677896] [current 1305677922] [deadline 26s] [delay 0s] req@ffff81018a17b400 x1369101166612551/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 368/392 e 0 to 1 dl 1305677922 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 17:18:42 client-5 kernel: Lustre: 1453:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 13 previous similar messages May 17 17:18:44 client-5 kernel: LustreError: 1453:0:(client.c:2570:ptlrpc_replay_interpret()) @@@ status -116, old was 0 req@ffff8102f4021800 x1369101166612548/t51539613126(51539613126) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:23/10 lens 360/424 e 0 to 0 dl 1305677951 ref 2 fl Interpret:R/ffffffff/ffffffff rc -116/-1 May 17 17:19:21 client-5 kernel: Lustre: 1453:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf4df74f to 0xfff98862bf51aed1 May 17 17:19:42 client-5 xinetd[3391]: START: shell pid=20291 from=192.168.4.5 May 17 17:19:42 client-5 rshd[20292]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 17:19:42 client-5 xinetd[3391]: EXIT: shell status=0 pid=20291 duration=0(sec) May 17 17:19:42 client-5 xinetd[3391]: START: shell pid=20454 from=192.168.4.5 May 17 17:19:42 client-5 rshd[20455]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark SKIP: recovery-small test_52 skipping ALWAYS excluded test 52");echo XXRETCODE:$?' May 17 17:19:42 client-5 kernel: Lustre: DEBUG MARKER: SKIP: recovery-small test_52 skipping ALWAYS excluded test 52 May 17 17:19:42 client-5 xinetd[3391]: EXIT: shell status=0 pid=20454 duration=0(sec) May 17 17:19:42 client-5 xinetd[3391]: START: shell pid=20557 from=192.168.4.5 May 17 17:19:42 client-5 rshd[20560]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == recovery-small test 53: touch: drop rep =========================================================== 17:19:42 \(1305677982\)");echo XXRETCODE:$?' May 17 17:19:42 client-5 kernel: Lustre: DEBUG MARKER: == recovery-small test 53: touch: drop rep =========================================================== 17:19:42 (1305677982) May 17 17:19:42 client-5 xinetd[3391]: EXIT: shell status=0 pid=20557 duration=0(sec) May 17 17:20:34 client-5 xinetd[3391]: START: shell pid=20677 from=192.168.4.5 May 17 17:20:34 client-5 rshd[20678]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 17:20:34 client-5 xinetd[3391]: EXIT: shell status=0 pid=20677 duration=0(sec) May 17 17:20:34 client-5 xinetd[3391]: START: shell pid=20849 from=192.168.4.5 May 17 17:20:34 client-5 rshd[20850]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == recovery-small test 54: back in time ============================================================== 17:20:34 \(1305678034\)");echo XXRETCODE:$?' May 17 17:20:34 client-5 kernel: Lustre: DEBUG MARKER: == recovery-small test 54: back in time ============================================================== 17:20:34 (1305678034) May 17 17:20:34 client-5 xinetd[3391]: EXIT: shell status=0 pid=20849 duration=0(sec) May 17 17:20:34 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 17:20:34 client-5 kernel: Lustre: 20893:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff810311946000->192.168.4.128@o2ib netid 50000: select flavor null May 17 17:20:34 client-5 kernel: Lustre: 20893:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 6 previous similar messages May 17 17:20:34 client-5 kernel: Lustre: Client lustre-client has started May 17 17:20:34 client-5 kernel: Lustre: 20900:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 17:20:34 client-5 kernel: Lustre: 20900:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message May 17 17:21:03 client-5 xinetd[3391]: START: shell pid=21025 from=192.168.4.5 May 17 17:21:04 client-5 rshd[21026]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 17:21:04 client-5 kernel: LustreError: 1453:0:(client.c:2570:ptlrpc_replay_interpret()) @@@ status -116, old was 0 req@ffff8103178ab000 x1369101166634810/t60129544703(60129544703) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:23/10 lens 360/424 e 0 to 0 dl 1305678091 ref 2 fl Interpret:R/ffffffff/ffffffff rc -116/-1 May 17 17:21:04 client-5 kernel: Lustre: 1453:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf51aed1 to 0xfff98862bf52b1b5 May 17 17:21:04 client-5 xinetd[3391]: EXIT: shell status=0 pid=21025 duration=1(sec) May 17 17:21:04 client-5 kernel: Lustre: client ffff810311946000 umount complete May 17 17:21:05 client-5 xinetd[3391]: START: shell pid=21123 from=192.168.4.5 May 17 17:21:05 client-5 rshd[21124]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 17:21:05 client-5 xinetd[3391]: EXIT: shell status=0 pid=21123 duration=0(sec) May 17 17:21:05 client-5 xinetd[3391]: START: shell pid=21295 from=192.168.4.5 May 17 17:21:05 client-5 rshd[21296]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == recovery-small test 55: ost_brw_read\/write drops timed-out read\/write request ===================== 17:21:05 \(1305678065\)");echo XXRETCODE:$?' May 17 17:21:05 client-5 kernel: Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request ===================== 17:21:05 (1305678065) May 17 17:21:05 client-5 xinetd[3391]: EXIT: shell status=0 pid=21295 duration=0(sec) May 17 17:22:37 client-5 xinetd[3391]: START: shell pid=21785 from=192.168.4.5 May 17 17:22:37 client-5 rshd[21786]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 17:22:37 client-5 xinetd[3391]: EXIT: shell status=0 pid=21785 duration=0(sec) May 17 17:22:37 client-5 xinetd[3391]: START: shell pid=21957 from=192.168.4.5 May 17 17:22:37 client-5 rshd[21958]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == recovery-small test 56: do not allow reconnect to busy exports ==================================== 17:22:37 \(1305678157\)");echo XXRETCODE:$?' May 17 17:22:38 client-5 kernel: Lustre: DEBUG MARKER: == recovery-small test 56: do not allow reconnect to busy exports ==================================== 17:22:37 (1305678157) May 17 17:22:38 client-5 xinetd[3391]: EXIT: shell status=0 pid=21957 duration=1(sec) May 17 17:23:18 client-5 xinetd[3391]: START: shell pid=22067 from=192.168.4.5 May 17 17:23:18 client-5 rshd[22068]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 17:23:18 client-5 xinetd[3391]: EXIT: shell status=0 pid=22067 duration=0(sec) May 17 17:23:18 client-5 xinetd[3391]: START: shell pid=22239 from=192.168.4.5 May 17 17:23:18 client-5 rshd[22240]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == recovery-small test 57: read procfs entries causes kernel crash =================================== 17:23:18 \(1305678198\)");echo XXRETCODE:$?' May 17 17:23:18 client-5 kernel: Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash =================================== 17:23:18 (1305678198) May 17 17:23:18 client-5 xinetd[3391]: EXIT: shell status=0 pid=22239 duration=0(sec) May 17 17:23:19 client-5 kernel: LustreError: 22421:0:(lvfs_lib.c:150:__obd_fail_timeout_set()) obd_fail_timeout id b00 sleeping for 10000ms May 17 17:23:29 client-5 kernel: LustreError: 22421:0:(lvfs_lib.c:154:__obd_fail_timeout_set()) obd_fail_timeout id b00 awake May 17 17:23:29 client-5 kernel: LustreError: 22440:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 17:23:29 client-5 kernel: LustreError: 22440:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Skipped 2 previous similar messages May 17 17:23:29 client-5 kernel: LustreError: 22440:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 17:23:29 client-5 kernel: LustreError: 22440:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) Skipped 2 previous similar messages May 17 17:23:29 client-5 kernel: Lustre: client ffff810318437400 umount complete May 17 17:23:39 client-5 xinetd[3391]: START: shell pid=22525 from=192.168.4.5 May 17 17:23:39 client-5 rshd[22526]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 17:23:39 client-5 xinetd[3391]: EXIT: shell status=0 pid=22525 duration=0(sec) May 17 17:23:49 client-5 xinetd[3391]: START: shell pid=22565 from=192.168.4.5 May 17 17:23:49 client-5 rshd[22566]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 17:23:49 client-5 xinetd[3391]: EXIT: shell status=0 pid=22565 duration=0(sec) May 17 17:23:49 client-5 kernel: Lustre: 22609:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 17:23:49 client-5 kernel: Lustre: 22609:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 6 previous similar messages May 17 17:23:49 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 17:23:49 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 17:23:49 client-5 kernel: Lustre: Skipped 4 previous similar messages May 17 17:23:49 client-5 kernel: Lustre: Client lustre-client has started May 17 17:23:49 client-5 kernel: Lustre: 22626:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 17:23:49 client-5 kernel: Lustre: 22626:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message May 17 17:23:49 client-5 xinetd[3391]: START: shell pid=22691 from=192.168.4.5 May 17 17:23:49 client-5 rshd[22692]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 17:23:49 client-5 xinetd[3391]: EXIT: shell status=0 pid=22691 duration=0(sec) May 17 17:23:49 client-5 xinetd[3391]: START: shell pid=22863 from=192.168.4.5 May 17 17:23:49 client-5 rshd[22864]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == recovery-small test 58: Eviction in the middle of open RPC reply processing ======================= 17:23:49 \(1305678229\)");echo XXRETCODE:$?' May 17 17:23:49 client-5 kernel: Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing ======================= 17:23:49 (1305678229) May 17 17:23:49 client-5 xinetd[3391]: EXIT: shell status=0 pid=22863 duration=0(sec) May 17 17:23:49 client-5 kernel: LustreError: 22901:0:(lvfs_lib.c:150:__obd_fail_timeout_set()) obd_fail_timeout id 801 sleeping for 20000ms May 17 17:23:50 client-5 kernel: LustreError: 22915:0:(obd_support.h:456:obd_fail_check_set()) *** obd_fail_loc=305 *** May 17 17:23:50 client-5 kernel: LustreError: 22616:0:(obd_support.h:456:obd_fail_check_set()) *** obd_fail_loc=305 *** May 17 17:24:01 client-5 kernel: LustreError: 22616:0:(obd_support.h:456:obd_fail_check_set()) *** obd_fail_loc=305 *** May 17 17:24:09 client-5 kernel: LustreError: 167-0: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. May 17 17:24:09 client-5 kernel: LustreError: Skipped 1 previous similar message May 17 17:24:09 client-5 kernel: LustreError: 22901:0:(lvfs_lib.c:154:__obd_fail_timeout_set()) obd_fail_timeout id 801 awake May 17 17:24:09 client-5 kernel: LustreError: 22901:0:(file.c:2161:ll_inode_revalidate_fini()) failure -2 inode 144115339473453057 May 17 17:24:09 client-5 kernel: LustreError: 22901:0:(file.c:155:ll_close_inode_openhandle()) inode 144115339473453057 mdc close failed: rc = -116 May 17 17:24:10 client-5 xinetd[3391]: START: shell pid=22985 from=192.168.4.5 May 17 17:24:10 client-5 rshd[22986]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 17:24:10 client-5 xinetd[3391]: EXIT: shell status=0 pid=22985 duration=0(sec) May 17 17:24:10 client-5 xinetd[3391]: START: shell pid=23157 from=192.168.4.5 May 17 17:24:10 client-5 rshd[23158]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == recovery-small test 59: Read cancel race on client eviction ======================================= 17:24:10 \(1305678250\)");echo XXRETCODE:$?' May 17 17:24:10 client-5 kernel: Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction ======================================= 17:24:10 (1305678250) May 17 17:24:10 client-5 xinetd[3391]: EXIT: shell status=0 pid=23157 duration=0(sec) May 17 17:24:10 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 17:24:10 client-5 kernel: Lustre: 23201:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff810314f2f000->192.168.4.128@o2ib netid 50000: select flavor null May 17 17:24:10 client-5 kernel: Lustre: 23201:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 7 previous similar messages May 17 17:24:10 client-5 kernel: Lustre: Client lustre-client has started May 17 17:24:10 client-5 kernel: Lustre: 23208:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 17:24:10 client-5 kernel: Lustre: 23208:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message May 17 17:24:10 client-5 kernel: Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request May 17 17:24:10 client-5 kernel: Lustre: Skipped 6 previous similar messages May 17 17:24:10 client-5 kernel: Lustre: client ffff810314f2f000 umount complete May 17 17:24:11 client-5 xinetd[3391]: START: shell pid=23301 from=192.168.4.5 May 17 17:24:11 client-5 rshd[23302]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 17:24:11 client-5 xinetd[3391]: EXIT: shell status=0 pid=23301 duration=0(sec) May 17 17:24:11 client-5 xinetd[3391]: START: shell pid=23473 from=192.168.4.5 May 17 17:24:11 client-5 rshd[23474]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == recovery-small test 60: Add Changelog entries during MDS failover ================================= 17:24:11 \(1305678251\)");echo XXRETCODE:$?' May 17 17:24:11 client-5 kernel: Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ================================= 17:24:11 (1305678251) May 17 17:24:11 client-5 xinetd[3391]: EXIT: shell status=0 pid=23473 duration=0(sec) May 17 17:24:15 client-5 kernel: LustreError: 167-0: This client was evicted by lustre-OST0005; in progress operations using this service will fail. May 17 17:25:12 client-5 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail. May 17 17:25:12 client-5 kernel: LustreError: Skipped 2 previous similar messages May 17 17:25:22 client-5 kernel: Lustre: 1453:0:(import.c:885:ptlrpc_connect_interpret()) MGS@MGC192.168.4.128@o2ib_0 changed server handle from 0xfff98862bf52b5d6 to 0xfff98862bf595947 May 17 17:26:40 client-5 xinetd[3391]: START: shell pid=23742 from=192.168.4.5 May 17 17:26:40 client-5 rshd[23743]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 17:26:40 client-5 xinetd[3391]: EXIT: shell status=0 pid=23742 duration=0(sec) May 17 17:26:40 client-5 xinetd[3391]: START: shell pid=23914 from=192.168.4.5 May 17 17:26:40 client-5 rshd[23915]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ============================ 17:26:40 \(1305678400\)");echo XXRETCODE:$?' May 17 17:26:41 client-5 kernel: Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ============================ 17:26:40 (1305678400) May 17 17:26:41 client-5 xinetd[3391]: EXIT: shell status=0 pid=23914 duration=1(sec) May 17 17:26:41 client-5 xinetd[3391]: START: shell pid=24046 from=192.168.4.5 May 17 17:26:41 client-5 rshd[24047]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 17:26:41 client-5 xinetd[3391]: EXIT: shell status=0 pid=24046 duration=0(sec) May 17 17:26:42 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 17:26:52 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The obd_ping operation failed with -107 May 17 17:26:52 client-5 kernel: LustreError: Skipped 11 previous similar messages May 17 17:26:52 client-5 kernel: Lustre: 1453:0:(import.c:885:ptlrpc_connect_interpret()) MGS@MGC192.168.4.128@o2ib_0 changed server handle from 0xfff98862bf595947 to 0xfff98862bf645cbf May 17 17:26:54 client-5 xinetd[3391]: START: shell pid=24200 from=192.168.4.5 May 17 17:26:54 client-5 rshd[24201]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 17:27:02 client-5 kernel: LustreError: 167-0: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. May 17 17:27:02 client-5 kernel: LustreError: 24233:0:(lmv_obd.c:1198:lmv_statfs()) can't stat MDS #0 (lustre-MDT0000-mdc-ffff81031397f800), error -4 May 17 17:27:02 client-5 kernel: LustreError: 24233:0:(llite_lib.c:1370:ll_statfs_internal()) md_statfs fails: rc = -4 May 17 17:27:02 client-5 xinetd[3391]: EXIT: shell status=0 pid=24200 duration=8(sec) May 17 17:27:03 client-5 xinetd[3391]: START: shell pid=24242 from=192.168.4.5 May 17 17:27:03 client-5 rshd[24244]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 17:27:03 client-5 xinetd[3391]: EXIT: shell status=0 pid=24242 duration=0(sec) May 17 17:27:03 client-5 xinetd[3391]: START: shell pid=24339 from=192.168.4.5 May 17 17:27:03 client-5 rshd[24340]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 17:27:03 client-5 xinetd[3391]: EXIT: shell status=0 pid=24339 duration=0(sec) May 17 17:27:04 client-5 xinetd[3391]: START: shell pid=24504 from=192.168.4.5 May 17 17:27:04 client-5 rshd[24505]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == recovery-small recovery-small.sh test complete, duration 1764 sec ================================= 17:27:04 \(1305678424\)");echo XXRETCODE:$?' May 17 17:27:04 client-5 kernel: Lustre: DEBUG MARKER: == recovery-small recovery-small.sh test complete, duration 1764 sec ================================= 17:27:04 (1305678424) May 17 17:27:04 client-5 xinetd[3391]: EXIT: shell status=0 pid=24504 duration=0(sec) May 17 17:27:23 client-5 xinetd[3391]: START: shell pid=24676 from=192.168.4.5 May 17 17:27:23 client-5 rshd[24677]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 17:27:30 client-5 xinetd[3391]: EXIT: shell status=0 pid=24676 duration=7(sec) May 17 17:27:30 client-5 xinetd[3391]: START: shell pid=24717 from=192.168.4.5 May 17 17:27:30 client-5 rshd[24718]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:-f; lsof -t /mnt/lustre || need_kill=no; if [ x-f != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount -f /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre is still busy, wait one second && sleep 1; umount -f /mnt/lustre; fi fi");echo XXRETCODE:$?' May 17 17:27:30 client-5 kernel: Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request May 17 17:27:30 client-5 kernel: Lustre: Skipped 6 previous similar messages May 17 17:27:30 client-5 kernel: Lustre: client ffff81031397f800 umount complete May 17 17:27:30 client-5 xinetd[3391]: EXIT: shell status=0 pid=24717 duration=0(sec) May 17 17:27:30 client-5 xinetd[3391]: START: shell pid=24768 from=192.168.4.5 May 17 17:27:30 client-5 rshd[24769]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:-f; lsof -t /mnt/lustre2 || need_kill=no; if [ x-f != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre2 | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount -f /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre2 is still busy, wait one second && sleep 1; umount -f /mnt/lustre2; fi fi");echo XXRETCODE:$?' May 17 17:27:30 client-5 xinetd[3391]: EXIT: shell status=0 pid=24768 duration=0(sec) May 17 17:28:11 client-5 kernel: Lustre: Acceptor stopping May 17 17:28:12 client-5 kernel: Lustre: Removed LNI 10.10.4.5@tcp May 17 17:28:14 client-5 kernel: Lustre: Removed LNI 192.168.4.5@o2ib May 17 17:28:16 client-5 kernel: Lustre: 25389:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 17:28:16 client-5 kernel: Lustre: 25390:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 17:28:16 client-5 kernel: Lustre: OBD class driver, http://www.lustre.org/ May 17 17:28:16 client-5 kernel: Lustre: Lustre Version: 2.0.60 May 17 17:28:16 client-5 kernel: Lustre: Build Version: jenkins-g985ac5b-PRISTINE-2.6.18-194.17.1.el5 May 17 17:28:16 client-5 kernel: Lustre: Lustre LU module (ffffffff888eb1c0). May 17 17:28:16 client-5 kernel: Lustre: Register global MR array, MR size: 0xffffffffffffffff, array size: 1 May 17 17:28:16 client-5 kernel: Lustre: Added LNI 192.168.4.5@o2ib [8/64/0/180] May 17 17:28:16 client-5 kernel: Lustre: Added LNI 10.10.4.5@tcp [8/256/0/180] May 17 17:28:16 client-5 kernel: Lustre: Accept all, port 7988 May 17 17:28:16 client-5 kernel: Lustre: Lustre OSC module (ffffffff88b3c6c0). May 17 17:28:16 client-5 kernel: Lustre: Lustre LOV module (ffffffff88bab540). May 17 17:28:16 client-5 kernel: Lustre: Lustre client module (ffffffff88c643a0). May 17 17:28:16 client-5 xinetd[3391]: START: shell pid=25796 from=192.168.4.5 May 17 17:28:16 client-5 rshd[25798]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark -----============= acceptance-small: replay-ost-single ============----- Tue May 17 17:28:16 PDT 2011");echo XXRETCODE:$?' May 17 17:28:16 client-5 kernel: Lustre: DEBUG MARKER: -----============= acceptance-small: replay-ost-single ============----- Tue May 17 17:28:16 PDT 2011 May 17 17:28:16 client-5 xinetd[3391]: EXIT: shell status=0 pid=25796 duration=0(sec) May 17 17:28:16 client-5 xinetd[3391]: START: shell pid=25962 from=192.168.4.5 May 17 17:28:16 client-5 rshd[25963]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark excepting tests: ");echo XXRETCODE:$?' May 17 17:28:17 client-5 kernel: Lustre: DEBUG MARKER: excepting tests: May 17 17:28:17 client-5 xinetd[3391]: EXIT: shell status=0 pid=25962 duration=1(sec) May 17 17:28:17 client-5 xinetd[3391]: START: shell pid=26015 from=192.168.4.5 May 17 17:28:17 client-5 rshd[26016]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:; lsof -t /mnt/lustre || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre is still busy, wait one second && sleep 1; umount /mnt/lustre; fi fi");echo XXRETCODE:$?' May 17 17:28:17 client-5 xinetd[3391]: EXIT: shell status=0 pid=26015 duration=0(sec) May 17 17:28:17 client-5 xinetd[3391]: START: shell pid=26059 from=192.168.4.5 May 17 17:28:17 client-5 rshd[26060]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:; lsof -t /mnt/lustre2 || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre2 | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre2 is still busy, wait one second && sleep 1; umount /mnt/lustre2; fi fi");echo XXRETCODE:$?' May 17 17:28:17 client-5 xinetd[3391]: EXIT: shell status=0 pid=26059 duration=0(sec) May 17 17:28:18 client-5 kernel: Lustre: 26262:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 17:29:16 client-5 xinetd[3391]: START: shell pid=26701 from=192.168.4.5 May 17 17:29:16 client-5 rshd[26702]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); mpts=\$(mount | grep -w -c /mnt/lustre); if [ \$running -ne \$mpts ]; then echo \$(hostname) env are INSANE!; exit 1; fi");echo XXRETCODE:$?' May 17 17:29:16 client-5 xinetd[3391]: EXIT: shell status=0 pid=26701 duration=0(sec) May 17 17:29:16 client-5 xinetd[3391]: START: shell pid=26747 from=192.168.4.5 May 17 17:29:16 client-5 rshd[26749]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); mpts=\$(mount | grep -w -c /mnt/lustre2); if [ \$running -ne \$mpts ]; then echo \$(hostname) env are INSANE!; exit 1; fi");echo XXRETCODE:$?' May 17 17:29:17 client-5 xinetd[3391]: EXIT: shell status=0 pid=26747 duration=1(sec) May 17 17:29:17 client-5 kernel: Lustre: 26791:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 17:29:17 client-5 kernel: Lustre: 26791:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message May 17 17:29:21 client-5 kernel: Lustre: 27128:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 17:29:21 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 17:29:21 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 17:29:21 client-5 kernel: Lustre: 27128:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff810324360800->192.168.4.128@o2ib netid 50000: select flavor null May 17 17:29:21 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 17:29:26 client-5 kernel: Lustre: 27128:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-OST0000-osc-ffff810324360800->192.168.4.129@o2ib netid 50000: select flavor null May 17 17:29:26 client-5 kernel: Lustre: Client lustre-client has started May 17 17:29:26 client-5 kernel: Lustre: 27145:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 17:29:26 client-5 kernel: Lustre: 27145:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message May 17 17:29:26 client-5 xinetd[3391]: START: shell pid=27155 from=192.168.4.5 May 17 17:29:26 client-5 rshd[27156]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c " running=\$(mount | grep -c /mnt/lustre' '); rc=0; if [ \$running -eq 0 ] ; then mkdir -p /mnt/lustre; mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre; rc=\$?; fi; exit \$rc");echo XXRETCODE:$?' May 17 17:29:26 client-5 xinetd[3391]: EXIT: shell status=0 pid=27155 duration=0(sec) May 17 17:29:26 client-5 xinetd[3391]: START: shell pid=27200 from=192.168.4.5 May 17 17:29:26 client-5 rshd[27201]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mount | grep -w /mnt/lustre");echo XXRETCODE:$?' May 17 17:29:26 client-5 xinetd[3391]: EXIT: shell status=0 pid=27200 duration=0(sec) May 17 17:29:26 client-5 xinetd[3391]: START: shell pid=27244 from=192.168.4.5 May 17 17:29:26 client-5 rshd[27245]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param debug=\"0x33f0406\"; lctl set_param subsystem_debug=\"0xffb7e3ff\"; lctl set_param debug_mb=32;");echo XXRETCODE:$?' May 17 17:29:26 client-5 xinetd[3391]: EXIT: shell status=0 pid=27244 duration=0(sec) May 17 17:29:27 client-5 xinetd[3391]: START: shell pid=27414 from=192.168.4.5 May 17 17:29:27 client-5 rshd[27415]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark Using TIMEOUT=20");echo XXRETCODE:$?' May 17 17:29:27 client-5 kernel: Lustre: DEBUG MARKER: Using TIMEOUT=20 May 17 17:29:27 client-5 xinetd[3391]: EXIT: shell status=0 pid=27414 duration=0(sec) May 17 17:29:27 client-5 kernel: LustreError: 27468:0:(quota_ctl.c:328:client_quota_ctl()) ptlrpc_queue_wait failed, rc: -114 May 17 17:29:32 client-5 xinetd[3391]: START: shell pid=27488 from=192.168.4.5 May 17 17:29:32 client-5 rshd[27489]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "cat /proc/mounts");echo XXRETCODE:$?' May 17 17:29:32 client-5 xinetd[3391]: EXIT: shell status=0 pid=27488 duration=0(sec) May 17 17:29:33 client-5 xinetd[3391]: START: shell pid=27763 from=192.168.4.5 May 17 17:29:33 client-5 rshd[27764]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-ost-single test 0a: target handle mismatch \(bug 5317\) 17:29:33 ============================= 17:29:33 \(1305678573\)");echo XXRETCODE:$?' May 17 17:29:33 client-5 kernel: Lustre: DEBUG MARKER: == replay-ost-single test 0a: target handle mismatch (bug 5317) 17:29:33 ============================= 17:29:33 (1305678573) May 17 17:29:33 client-5 xinetd[3391]: EXIT: shell status=0 pid=27763 duration=0(sec) May 17 17:29:33 client-5 kernel: Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request May 17 17:29:33 client-5 kernel: Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request May 17 17:29:33 client-5 kernel: LustreError: 27816:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 17:29:33 client-5 kernel: LustreError: 27816:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 17:29:33 client-5 kernel: Lustre: client ffff810324360800 umount complete May 17 17:29:34 client-5 kernel: Lustre: 27834:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 17:29:34 client-5 kernel: Lustre: 27834:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 5 previous similar messages May 17 17:29:34 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 17:29:34 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 17:29:34 client-5 kernel: Lustre: Client lustre-client has started May 17 17:29:34 client-5 kernel: Lustre: 27850:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 17:29:34 client-5 kernel: Lustre: 27850:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 3 previous similar messages May 17 17:29:39 client-5 kernel: Lustre: 25499:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369103134621758 sent from lustre-OST0000-osc-ffff8102fddef400 to NID 192.168.4.129@o2ib has timed out for slow reply: [sent 1305678574] [real_sent 1305678574] [current 1305678579] [deadline 5s] [delay 0s] req@ffff8102f45c7000 x1369103134621758/t0(0) o-1->lustre-OST0000_UUID@192.168.4.129@o2ib:28/4 lens 368/392 e 0 to 1 dl 1305678579 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 17:29:44 client-5 kernel: Lustre: 25500:0:(import.c:529:import_select_connection()) lustre-OST0000-osc-ffff8102fddef400: tried all connections, increasing latency to 5s May 17 17:29:44 client-5 xinetd[3391]: START: shell pid=27911 from=192.168.4.5 May 17 17:29:44 client-5 rshd[27912]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 17:29:44 client-5 xinetd[3391]: EXIT: shell status=0 pid=27911 duration=0(sec) May 17 17:29:44 client-5 xinetd[3391]: START: shell pid=28083 from=192.168.4.5 May 17 17:29:44 client-5 rshd[28084]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-ost-single test 0b: empty replay =========================================================== 17:29:44 \(1305678584\)");echo XXRETCODE:$?' May 17 17:29:44 client-5 kernel: Lustre: DEBUG MARKER: == replay-ost-single test 0b: empty replay =========================================================== 17:29:44 (1305678584) May 17 17:29:44 client-5 xinetd[3391]: EXIT: shell status=0 pid=28083 duration=0(sec) May 17 17:29:54 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.129@o2ib. The obd_ping operation failed with -107 May 17 17:29:54 client-5 kernel: Lustre: lustre-OST0000-osc-ffff8102fddef400: Connection to service lustre-OST0000 via nid 192.168.4.129@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 17:29:58 client-5 xinetd[3391]: START: shell pid=28239 from=192.168.4.5 May 17 17:29:58 client-5 rshd[28241]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 17:30:04 client-5 kernel: Lustre: 25500:0:(import.c:529:import_select_connection()) lustre-OST0000-osc-ffff8102fddef400: tried all connections, increasing latency to 10s May 17 17:31:04 client-5 kernel: Lustre: lustre-OST0000-osc-ffff8102fddef400: Connection restored to service lustre-OST0000 using nid 192.168.4.129@o2ib. May 17 17:31:04 client-5 xinetd[3391]: EXIT: shell status=0 pid=28239 duration=66(sec) May 17 17:31:04 client-5 xinetd[3391]: START: shell pid=28334 from=192.168.4.5 May 17 17:31:04 client-5 rshd[28335]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 17:31:04 client-5 xinetd[3391]: EXIT: shell status=0 pid=28334 duration=0(sec) May 17 17:31:04 client-5 xinetd[3391]: START: shell pid=28505 from=192.168.4.5 May 17 17:31:04 client-5 rshd[28507]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-ost-single test 1: touch =================================================================== 17:31:04 \(1305678664\)");echo XXRETCODE:$?' May 17 17:31:04 client-5 kernel: Lustre: DEBUG MARKER: == replay-ost-single test 1: touch =================================================================== 17:31:04 (1305678664) May 17 17:31:04 client-5 xinetd[3391]: EXIT: shell status=0 pid=28505 duration=0(sec) May 17 17:31:14 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.129@o2ib. The obd_ping operation failed with -107 May 17 17:31:14 client-5 kernel: LustreError: Skipped 1 previous similar message May 17 17:31:14 client-5 kernel: Lustre: lustre-OST0000-osc-ffff8102fddef400: Connection to service lustre-OST0000 via nid 192.168.4.129@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 17:31:18 client-5 xinetd[3391]: START: shell pid=28664 from=192.168.4.5 May 17 17:31:18 client-5 rshd[28665]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 17:31:24 client-5 kernel: Lustre: 25500:0:(import.c:529:import_select_connection()) lustre-OST0000-osc-ffff8102fddef400: tried all connections, increasing latency to 15s May 17 17:31:24 client-5 kernel: Lustre: lustre-OST0000-osc-ffff8102fddef400: Connection restored to service lustre-OST0000 using nid 192.168.4.129@o2ib. May 17 17:31:24 client-5 xinetd[3391]: EXIT: shell status=0 pid=28664 duration=6(sec) May 17 17:31:24 client-5 xinetd[3391]: START: shell pid=28756 from=192.168.4.5 May 17 17:31:24 client-5 rshd[28757]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 17:31:24 client-5 xinetd[3391]: EXIT: shell status=0 pid=28756 duration=0(sec) May 17 17:31:24 client-5 xinetd[3391]: START: shell pid=28928 from=192.168.4.5 May 17 17:31:24 client-5 rshd[28929]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-ost-single test 2: \|x\| 10 open\(O_CREAT\)s =================================================== 17:31:24 \(1305678684\)");echo XXRETCODE:$?' May 17 17:31:24 client-5 kernel: Lustre: DEBUG MARKER: == replay-ost-single test 2: |x| 10 open(O_CREAT)s =================================================== 17:31:24 (1305678684) May 17 17:31:24 client-5 xinetd[3391]: EXIT: shell status=0 pid=28928 duration=0(sec) May 17 17:31:34 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.129@o2ib. The obd_ping operation failed with -107 May 17 17:31:34 client-5 kernel: LustreError: Skipped 1 previous similar message May 17 17:31:34 client-5 kernel: Lustre: lustre-OST0000-osc-ffff8102fddef400: Connection to service lustre-OST0000 via nid 192.168.4.129@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 17:31:38 client-5 xinetd[3391]: START: shell pid=29082 from=192.168.4.5 May 17 17:31:38 client-5 rshd[29083]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 17:31:44 client-5 kernel: Lustre: 25500:0:(import.c:529:import_select_connection()) lustre-OST0000-osc-ffff8102fddef400: tried all connections, increasing latency to 20s May 17 17:31:44 client-5 kernel: Lustre: lustre-OST0000-osc-ffff8102fddef400: Connection restored to service lustre-OST0000 using nid 192.168.4.129@o2ib. May 17 17:31:44 client-5 xinetd[3391]: EXIT: shell status=0 pid=29082 duration=6(sec) May 17 17:31:44 client-5 xinetd[3391]: START: shell pid=29184 from=192.168.4.5 May 17 17:31:44 client-5 rshd[29185]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 17:31:44 client-5 xinetd[3391]: EXIT: shell status=0 pid=29184 duration=0(sec) May 17 17:31:44 client-5 xinetd[3391]: START: shell pid=29356 from=192.168.4.5 May 17 17:31:44 client-5 rshd[29357]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-ost-single test 3: Fail OST during write, with verification ================================ 17:31:44 \(1305678704\)");echo XXRETCODE:$?' May 17 17:31:44 client-5 kernel: Lustre: DEBUG MARKER: == replay-ost-single test 3: Fail OST during write, with verification ================================ 17:31:44 (1305678704) May 17 17:31:44 client-5 xinetd[3391]: EXIT: shell status=0 pid=29356 duration=0(sec) May 17 17:31:45 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.129@o2ib. The ost_write operation failed with -107 May 17 17:31:45 client-5 kernel: LustreError: Skipped 1 previous similar message May 17 17:31:45 client-5 kernel: Lustre: lustre-OST0000-osc-ffff8102fddef400: Connection to service lustre-OST0000 via nid 192.168.4.129@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 17:31:54 client-5 kernel: Lustre: 25500:0:(import.c:529:import_select_connection()) lustre-OST0000-osc-ffff8102fddef400: tried all connections, increasing latency to 20s May 17 17:31:54 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.129@o2ib. The ost_connect operation failed with -19 May 17 17:31:54 client-5 kernel: LustreError: Skipped 1 previous similar message May 17 17:31:57 client-5 xinetd[3391]: START: shell pid=29512 from=192.168.4.5 May 17 17:31:58 client-5 rshd[29513]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 17:32:04 client-5 kernel: Lustre: 25500:0:(import.c:529:import_select_connection()) lustre-OST0000-osc-ffff8102fddef400: tried all connections, increasing latency to 20s May 17 17:32:04 client-5 kernel: Lustre: lustre-OST0000-osc-ffff8102fddef400: Connection restored to service lustre-OST0000 using nid 192.168.4.129@o2ib. May 17 17:32:04 client-5 xinetd[3391]: EXIT: shell status=0 pid=29512 duration=7(sec) May 17 17:32:04 client-5 xinetd[3391]: START: shell pid=29602 from=192.168.4.5 May 17 17:32:04 client-5 rshd[29605]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 17:32:04 client-5 xinetd[3391]: EXIT: shell status=0 pid=29602 duration=0(sec) May 17 17:32:04 client-5 xinetd[3391]: START: shell pid=29776 from=192.168.4.5 May 17 17:32:05 client-5 rshd[29777]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-ost-single test 4: Fail OST during read, with verification ================================= 17:32:04 \(1305678724\)");echo XXRETCODE:$?' May 17 17:32:05 client-5 kernel: Lustre: DEBUG MARKER: == replay-ost-single test 4: Fail OST during read, with verification ================================= 17:32:04 (1305678724) May 17 17:32:05 client-5 xinetd[3391]: EXIT: shell status=0 pid=29776 duration=1(sec) May 17 17:32:05 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 17:32:06 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 17:32:14 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.129@o2ib. The obd_ping operation failed with -107 May 17 17:32:14 client-5 kernel: Lustre: lustre-OST0000-osc-ffff8102fddef400: Connection to service lustre-OST0000 via nid 192.168.4.129@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 17:32:19 client-5 xinetd[3391]: START: shell pid=29946 from=192.168.4.5 May 17 17:32:19 client-5 rshd[29947]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 17:32:24 client-5 kernel: Lustre: 25500:0:(import.c:529:import_select_connection()) lustre-OST0000-osc-ffff8102fddef400: tried all connections, increasing latency to 20s May 17 17:32:24 client-5 kernel: Lustre: lustre-OST0000-osc-ffff8102fddef400: Connection restored to service lustre-OST0000 using nid 192.168.4.129@o2ib. May 17 17:32:24 client-5 xinetd[3391]: EXIT: shell status=0 pid=29946 duration=5(sec) May 17 17:32:24 client-5 xinetd[3391]: START: shell pid=30041 from=192.168.4.5 May 17 17:32:24 client-5 rshd[30042]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 17:32:24 client-5 xinetd[3391]: EXIT: shell status=0 pid=30041 duration=0(sec) May 17 17:32:24 client-5 xinetd[3391]: START: shell pid=30213 from=192.168.4.5 May 17 17:32:24 client-5 rshd[30214]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-ost-single test 5: Fail OST during iozone ================================================== 17:32:24 \(1305678744\)");echo XXRETCODE:$?' May 17 17:32:24 client-5 kernel: Lustre: DEBUG MARKER: == replay-ost-single test 5: Fail OST during iozone ================================================== 17:32:24 (1305678744) May 17 17:32:24 client-5 xinetd[3391]: EXIT: shell status=0 pid=30213 duration=0(sec) May 17 17:32:32 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.129@o2ib. The ost_write operation failed with -107 May 17 17:32:32 client-5 kernel: LustreError: Skipped 1 previous similar message May 17 17:32:32 client-5 kernel: Lustre: lustre-OST0000-osc-ffff8102fddef400: Connection to service lustre-OST0000 via nid 192.168.4.129@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 17:32:44 client-5 kernel: Lustre: 25500:0:(import.c:529:import_select_connection()) lustre-OST0000-osc-ffff8102fddef400: tried all connections, increasing latency to 20s May 17 17:32:44 client-5 kernel: Lustre: 25500:0:(import.c:529:import_select_connection()) Skipped 1 previous similar message May 17 17:32:46 client-5 xinetd[3391]: START: shell pid=30378 from=192.168.4.5 May 17 17:32:46 client-5 rshd[30380]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 17:32:54 client-5 kernel: Lustre: lustre-OST0000-osc-ffff8102fddef400: Connection restored to service lustre-OST0000 using nid 192.168.4.129@o2ib. May 17 17:32:54 client-5 xinetd[3391]: EXIT: shell status=0 pid=30378 duration=8(sec) May 17 17:42:42 client-5 xinetd[3391]: START: shell pid=30508 from=192.168.4.5 May 17 17:42:42 client-5 rshd[30509]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark iozone rc=0");echo XXRETCODE:$?' May 17 17:42:42 client-5 kernel: Lustre: DEBUG MARKER: iozone rc=0 May 17 17:42:42 client-5 xinetd[3391]: EXIT: shell status=0 pid=30508 duration=0(sec) May 17 17:42:42 client-5 xinetd[3391]: START: shell pid=30600 from=192.168.4.5 May 17 17:42:42 client-5 rshd[30601]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 17:42:42 client-5 xinetd[3391]: EXIT: shell status=0 pid=30600 duration=0(sec) May 17 17:42:42 client-5 xinetd[3391]: START: shell pid=30772 from=192.168.4.5 May 17 17:42:42 client-5 rshd[30773]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-ost-single test 6: Fail OST before obd_destroy ============================================= 17:42:42 \(1305679362\)");echo XXRETCODE:$?' May 17 17:42:42 client-5 kernel: Lustre: DEBUG MARKER: == replay-ost-single test 6: Fail OST before obd_destroy ============================================= 17:42:42 (1305679362) May 17 17:42:42 client-5 xinetd[3391]: EXIT: shell status=0 pid=30772 duration=0(sec) May 17 17:42:48 client-5 xinetd[3391]: START: shell pid=30911 from=192.168.4.5 May 17 17:42:48 client-5 rshd[30912]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark before: 56540232 after_dd: 56535112");echo XXRETCODE:$?' May 17 17:42:48 client-5 kernel: Lustre: DEBUG MARKER: before: 56540232 after_dd: 56535112 May 17 17:42:48 client-5 xinetd[3391]: EXIT: shell status=0 pid=30911 duration=0(sec) May 17 17:42:55 client-5 kernel: Lustre: 30946:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369103134633366 sent from lustre-MDT0000-mdc-ffff8102fddef400 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305679368] [real_sent 1305679368] [current 1305679375] [deadline 7s] [delay 0s] req@ffff81032260cc00 x1369103134633366/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 456/1048 e 0 to 1 dl 1305679375 ref 2 fl Rpc:X/ffffffff/ffffffff rc 0/-1 May 17 17:42:55 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff8102fddef400: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 17:42:55 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff8102fddef400: Connection restored to service lustre-MDT0000 using nid 192.168.4.128@o2ib. May 17 17:43:00 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.129@o2ib. The obd_ping operation failed with -107 May 17 17:43:00 client-5 kernel: LustreError: Skipped 3 previous similar messages May 17 17:43:00 client-5 kernel: Lustre: lustre-OST0000-osc-ffff8102fddef400: Connection to service lustre-OST0000 via nid 192.168.4.129@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 17:43:09 client-5 xinetd[3391]: START: shell pid=31065 from=192.168.4.5 May 17 17:43:09 client-5 rshd[31066]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 17:43:10 client-5 kernel: Lustre: 25500:0:(import.c:529:import_select_connection()) lustre-OST0000-osc-ffff8102fddef400: tried all connections, increasing latency to 7s May 17 17:43:10 client-5 kernel: Lustre: 25500:0:(import.c:529:import_select_connection()) Skipped 1 previous similar message May 17 17:43:16 client-5 kernel: Lustre: lustre-OST0000-osc-ffff8102fddef400: Connection restored to service lustre-OST0000 using nid 192.168.4.129@o2ib. May 17 17:43:16 client-5 xinetd[3391]: EXIT: shell status=0 pid=31065 duration=7(sec) May 17 17:43:19 client-5 xinetd[3391]: START: shell pid=31220 from=192.168.4.5 May 17 17:43:19 client-5 rshd[31221]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark before: 56540232 after: 56540232");echo XXRETCODE:$?' May 17 17:43:20 client-5 kernel: Lustre: DEBUG MARKER: before: 56540232 after: 56540232 May 17 17:43:20 client-5 xinetd[3391]: EXIT: shell status=0 pid=31220 duration=1(sec) May 17 17:43:20 client-5 xinetd[3391]: START: shell pid=31310 from=192.168.4.5 May 17 17:43:20 client-5 rshd[31312]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 17:43:20 client-5 xinetd[3391]: EXIT: shell status=0 pid=31310 duration=0(sec) May 17 17:43:20 client-5 xinetd[3391]: START: shell pid=31483 from=192.168.4.5 May 17 17:43:20 client-5 rshd[31484]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-ost-single test 7: Fail OST before obd_destroy ============================================= 17:43:20 \(1305679400\)");echo XXRETCODE:$?' May 17 17:43:20 client-5 kernel: Lustre: DEBUG MARKER: == replay-ost-single test 7: Fail OST before obd_destroy ============================================= 17:43:20 (1305679400) May 17 17:43:20 client-5 xinetd[3391]: EXIT: shell status=0 pid=31483 duration=0(sec) May 17 17:43:28 client-5 xinetd[3391]: START: shell pid=31594 from=192.168.4.5 May 17 17:43:28 client-5 rshd[31595]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark before: 56540232 after_dd: 56535112");echo XXRETCODE:$?' May 17 17:43:28 client-5 kernel: Lustre: DEBUG MARKER: before: 56540232 after_dd: 56535112 May 17 17:43:28 client-5 xinetd[3391]: EXIT: shell status=0 pid=31594 duration=0(sec) May 17 17:43:28 client-5 xinetd[3391]: START: shell pid=31645 from=192.168.4.5 May 17 17:43:28 client-5 rshd[31646]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 17:43:28 client-5 xinetd[3391]: EXIT: shell status=0 pid=31645 duration=0(sec) May 17 17:43:29 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-OST0000 May 17 17:43:35 client-5 kernel: Lustre: lustre-OST0000-osc-ffff8102fddef400: Connection to service lustre-OST0000 via nid 192.168.4.129@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 17:43:42 client-5 xinetd[3391]: START: shell pid=31832 from=192.168.4.5 May 17 17:43:42 client-5 rshd[31833]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 17:43:45 client-5 xinetd[3391]: EXIT: shell status=0 pid=31832 duration=3(sec) May 17 17:43:48 client-5 xinetd[3391]: START: shell pid=31987 from=192.168.4.5 May 17 17:43:48 client-5 rshd[31988]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark before: 56540232 after: 56540232");echo XXRETCODE:$?' May 17 17:43:48 client-5 kernel: Lustre: DEBUG MARKER: before: 56540232 after: 56540232 May 17 17:43:48 client-5 xinetd[3391]: EXIT: shell status=0 pid=31987 duration=0(sec) May 17 17:43:48 client-5 xinetd[3391]: START: shell pid=32077 from=192.168.4.5 May 17 17:43:48 client-5 rshd[32079]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 17:43:49 client-5 xinetd[3391]: EXIT: shell status=0 pid=32077 duration=1(sec) May 17 17:43:49 client-5 xinetd[3391]: START: shell pid=32242 from=192.168.4.5 May 17 17:43:49 client-5 rshd[32244]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-ost-single replay-ost-single.sh test complete, duration 933 sec ============================ 17:43:49 \(1305679429\)");echo XXRETCODE:$?' May 17 17:43:49 client-5 kernel: Lustre: DEBUG MARKER: == replay-ost-single replay-ost-single.sh test complete, duration 933 sec ============================ 17:43:49 (1305679429) May 17 17:43:49 client-5 xinetd[3391]: EXIT: shell status=0 pid=32242 duration=0(sec) May 17 17:43:49 client-5 xinetd[3391]: START: shell pid=32293 from=192.168.4.5 May 17 17:43:49 client-5 rshd[32294]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:-f; lsof -t /mnt/lustre || need_kill=no; if [ x-f != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount -f /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre is still busy, wait one second && sleep 1; umount -f /mnt/lustre; fi fi");echo XXRETCODE:$?' May 17 17:43:49 client-5 kernel: Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request May 17 17:43:49 client-5 kernel: Lustre: Skipped 5 previous similar messages May 17 17:43:49 client-5 kernel: LustreError: 32335:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 17:43:49 client-5 kernel: LustreError: 32335:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 17:43:49 client-5 kernel: Lustre: client ffff8102fddef400 umount complete May 17 17:43:49 client-5 xinetd[3391]: EXIT: shell status=0 pid=32293 duration=0(sec) May 17 17:43:49 client-5 xinetd[3391]: START: shell pid=32344 from=192.168.4.5 May 17 17:43:49 client-5 rshd[32345]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:-f; lsof -t /mnt/lustre2 || need_kill=no; if [ x-f != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre2 | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount -f /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre2 is still busy, wait one second && sleep 1; umount -f /mnt/lustre2; fi fi");echo XXRETCODE:$?' May 17 17:43:49 client-5 xinetd[3391]: EXIT: shell status=0 pid=32344 duration=0(sec) May 17 17:44:14 client-5 kernel: Lustre: Acceptor stopping May 17 17:44:15 client-5 kernel: Lustre: Removed LNI 10.10.4.5@tcp May 17 17:44:17 client-5 kernel: Lustre: Removed LNI 192.168.4.5@o2ib May 17 17:44:18 client-5 kernel: Lustre: 507:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 17:44:18 client-5 kernel: Lustre: 508:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 17:44:18 client-5 kernel: Lustre: OBD class driver, http://www.lustre.org/ May 17 17:44:18 client-5 kernel: Lustre: Lustre Version: 2.0.60 May 17 17:44:18 client-5 kernel: Lustre: Build Version: jenkins-g985ac5b-PRISTINE-2.6.18-194.17.1.el5 May 17 17:44:18 client-5 kernel: Lustre: Lustre LU module (ffffffff888eb1c0). May 17 17:44:18 client-5 kernel: Lustre: Register global MR array, MR size: 0xffffffffffffffff, array size: 1 May 17 17:44:18 client-5 kernel: Lustre: Added LNI 192.168.4.5@o2ib [8/64/0/180] May 17 17:44:18 client-5 kernel: Lustre: Added LNI 10.10.4.5@tcp [8/256/0/180] May 17 17:44:18 client-5 kernel: Lustre: Accept all, port 7988 May 17 17:44:19 client-5 kernel: Lustre: Lustre OSC module (ffffffff88b3c6c0). May 17 17:44:19 client-5 kernel: Lustre: Lustre LOV module (ffffffff88bab540). May 17 17:44:19 client-5 kernel: Lustre: Lustre client module (ffffffff88c643a0). May 17 17:44:19 client-5 xinetd[3391]: START: shell pid=925 from=192.168.4.5 May 17 17:44:19 client-5 rshd[928]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark -----============= acceptance-small: replay-dual ============----- Tue May 17 17:44:18 PDT 2011");echo XXRETCODE:$?' May 17 17:44:19 client-5 kernel: Lustre: DEBUG MARKER: -----============= acceptance-small: replay-dual ============----- Tue May 17 17:44:18 PDT 2011 May 17 17:44:19 client-5 xinetd[3391]: EXIT: shell status=0 pid=925 duration=0(sec) May 17 17:44:19 client-5 xinetd[3391]: START: shell pid=1077 from=192.168.4.5 May 17 17:44:19 client-5 rshd[1078]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark excepting tests: 15c");echo XXRETCODE:$?' May 17 17:44:19 client-5 kernel: Lustre: DEBUG MARKER: excepting tests: 15c May 17 17:44:19 client-5 xinetd[3391]: EXIT: shell status=0 pid=1077 duration=0(sec) May 17 17:44:19 client-5 xinetd[3391]: START: shell pid=1130 from=192.168.4.5 May 17 17:44:19 client-5 rshd[1131]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:; lsof -t /mnt/lustre || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre is still busy, wait one second && sleep 1; umount /mnt/lustre; fi fi");echo XXRETCODE:$?' May 17 17:44:19 client-5 xinetd[3391]: EXIT: shell status=0 pid=1130 duration=0(sec) May 17 17:44:19 client-5 xinetd[3391]: START: shell pid=1174 from=192.168.4.5 May 17 17:44:19 client-5 rshd[1175]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:; lsof -t /mnt/lustre2 || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre2 | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre2 is still busy, wait one second && sleep 1; umount /mnt/lustre2; fi fi");echo XXRETCODE:$?' May 17 17:44:19 client-5 xinetd[3391]: EXIT: shell status=0 pid=1174 duration=0(sec) May 17 17:44:21 client-5 kernel: Lustre: 1377:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 17:45:19 client-5 xinetd[3391]: START: shell pid=1817 from=192.168.4.5 May 17 17:45:19 client-5 rshd[1818]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); mpts=\$(mount | grep -w -c /mnt/lustre); if [ \$running -ne \$mpts ]; then echo \$(hostname) env are INSANE!; exit 1; fi");echo XXRETCODE:$?' May 17 17:45:19 client-5 xinetd[3391]: EXIT: shell status=0 pid=1817 duration=0(sec) May 17 17:45:19 client-5 xinetd[3391]: START: shell pid=1864 from=192.168.4.5 May 17 17:45:19 client-5 rshd[1865]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); mpts=\$(mount | grep -w -c /mnt/lustre2); if [ \$running -ne \$mpts ]; then echo \$(hostname) env are INSANE!; exit 1; fi");echo XXRETCODE:$?' May 17 17:45:19 client-5 xinetd[3391]: EXIT: shell status=0 pid=1864 duration=0(sec) May 17 17:45:19 client-5 kernel: Lustre: 1912:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 17:45:19 client-5 kernel: Lustre: 1912:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message May 17 17:45:23 client-5 kernel: Lustre: 2263:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 17:45:23 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 17:45:23 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 17:45:23 client-5 kernel: Lustre: 2263:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff810333bd1400->192.168.4.128@o2ib netid 50000: select flavor null May 17 17:45:23 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 17:45:28 client-5 kernel: Lustre: 2263:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-OST0000-osc-ffff810333bd1400->192.168.4.129@o2ib netid 50000: select flavor null May 17 17:45:28 client-5 kernel: Lustre: Client lustre-client has started May 17 17:45:28 client-5 kernel: Lustre: 2281:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 17:45:28 client-5 kernel: Lustre: 2281:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message May 17 17:45:28 client-5 xinetd[3391]: START: shell pid=2291 from=192.168.4.5 May 17 17:45:28 client-5 rshd[2292]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c " running=\$(mount | grep -c /mnt/lustre' '); rc=0; if [ \$running -eq 0 ] ; then mkdir -p /mnt/lustre; mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre; rc=\$?; fi; exit \$rc");echo XXRETCODE:$?' May 17 17:45:29 client-5 xinetd[3391]: EXIT: shell status=0 pid=2291 duration=1(sec) May 17 17:45:29 client-5 xinetd[3391]: START: shell pid=2336 from=192.168.4.5 May 17 17:45:29 client-5 rshd[2337]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mount | grep -w /mnt/lustre");echo XXRETCODE:$?' May 17 17:45:29 client-5 xinetd[3391]: EXIT: shell status=0 pid=2336 duration=0(sec) May 17 17:45:29 client-5 xinetd[3391]: START: shell pid=2380 from=192.168.4.5 May 17 17:45:29 client-5 rshd[2381]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param debug=\"-1\"; lctl set_param subsystem_debug=\"0xffb7e3ff\"; lctl set_param debug_mb=32;");echo XXRETCODE:$?' May 17 17:45:29 client-5 xinetd[3391]: EXIT: shell status=0 pid=2380 duration=0(sec) May 17 17:45:29 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 17:45:29 client-5 kernel: Lustre: Client lustre-client has started May 17 17:45:29 client-5 xinetd[3391]: START: shell pid=2454 from=192.168.4.5 May 17 17:45:29 client-5 rshd[2455]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c " running=\$(mount | grep -c /mnt/lustre2' '); rc=0; if [ \$running -eq 0 ] ; then mkdir -p /mnt/lustre2; mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre2; rc=\$?; fi; exit \$rc");echo XXRETCODE:$?' May 17 17:45:29 client-5 xinetd[3391]: EXIT: shell status=0 pid=2454 duration=0(sec) May 17 17:45:29 client-5 xinetd[3391]: START: shell pid=2500 from=192.168.4.5 May 17 17:45:29 client-5 rshd[2501]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mount | grep -w /mnt/lustre2");echo XXRETCODE:$?' May 17 17:45:29 client-5 xinetd[3391]: EXIT: shell status=0 pid=2500 duration=0(sec) May 17 17:45:29 client-5 xinetd[3391]: START: shell pid=2549 from=192.168.4.5 May 17 17:45:29 client-5 rshd[2550]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param debug=\"-1\"; lctl set_param subsystem_debug=\"0xffb7e3ff\"; lctl set_param debug_mb=32;");echo XXRETCODE:$?' May 17 17:45:29 client-5 xinetd[3391]: EXIT: shell status=0 pid=2549 duration=0(sec) May 17 17:45:30 client-5 xinetd[3391]: START: shell pid=2719 from=192.168.4.5 May 17 17:45:30 client-5 rshd[2720]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark Using TIMEOUT=20");echo XXRETCODE:$?' May 17 17:45:30 client-5 kernel: Lustre: DEBUG MARKER: Using TIMEOUT=20 May 17 17:45:30 client-5 xinetd[3391]: EXIT: shell status=0 pid=2719 duration=0(sec) May 17 17:45:30 client-5 kernel: LustreError: 2773:0:(quota_ctl.c:328:client_quota_ctl()) ptlrpc_queue_wait failed, rc: -114 May 17 17:45:35 client-5 xinetd[3391]: START: shell pid=2793 from=192.168.4.5 May 17 17:45:35 client-5 rshd[2794]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "cat /proc/mounts");echo XXRETCODE:$?' May 17 17:45:35 client-5 xinetd[3391]: EXIT: shell status=0 pid=2793 duration=0(sec) May 17 17:45:36 client-5 xinetd[3391]: START: shell pid=3077 from=192.168.4.5 May 17 17:45:36 client-5 rshd[3080]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-dual test 0a: expired recovery with lost client ============================================ 17:45:36 \(1305679536\)");echo XXRETCODE:$?' May 17 17:45:36 client-5 kernel: Lustre: DEBUG MARKER: == replay-dual test 0a: expired recovery with lost client ============================================ 17:45:36 (1305679536) May 17 17:45:36 client-5 xinetd[3391]: EXIT: shell status=0 pid=3077 duration=0(sec) May 17 17:45:36 client-5 xinetd[3391]: START: shell pid=3137 from=192.168.4.5 May 17 17:45:36 client-5 rshd[3138]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 17:45:37 client-5 xinetd[3391]: EXIT: shell status=0 pid=3137 duration=1(sec) May 17 17:45:37 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 17:45:46 client-5 kernel: Lustre: 628:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369104143351992 sent from MGC192.168.4.128@o2ib to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305679539] [real_sent 1305679539] [current 1305679546] [deadline 7s] [delay 0s] req@ffff81030f55cc00 x1369104143351992/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 192/192 e 0 to 1 dl 1305679546 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 17:45:46 client-5 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail. May 17 17:45:51 client-5 kernel: Lustre: 628:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369104143351998 sent from lustre-MDT0000-mdc-ffff810333bd1400 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305679544] [real_sent 1305679544] [current 1305679551] [deadline 7s] [delay 0s] req@ffff810307997c00 x1369104143351998/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 192/192 e 0 to 1 dl 1305679551 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 17:45:51 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff810333bd1400: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 17:45:51 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff8102f95fc800: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 17:45:52 client-5 kernel: Lustre: 629:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369104143352012 sent from MGC192.168.4.128@o2ib to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305679546] [real_sent 1305679546] [current 1305679552] [deadline 6s] [delay 0s] req@ffff8102c8572800 x1369104143352012/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 368/392 e 0 to 1 dl 1305679552 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 17:45:52 client-5 kernel: Lustre: 629:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 1 previous similar message May 17 17:45:56 client-5 kernel: Lustre: 630:0:(import.c:529:import_select_connection()) MGC192.168.4.128@o2ib: tried all connections, increasing latency to 6s May 17 17:45:56 client-5 kernel: Lustre: 629:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf64768f to 0xfff98862bf647f6b May 17 17:45:56 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 17:45:56 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Connection restored to service MGS using nid 192.168.4.128@o2ib. May 17 17:45:57 client-5 kernel: Lustre: 629:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369104143352027 sent from lustre-MDT0000-mdc-ffff810333bd1400 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305679551] [real_sent 1305679551] [current 1305679557] [deadline 6s] [delay 0s] req@ffff81030fa93400 x1369104143352027/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 368/392 e 0 to 1 dl 1305679557 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 17:46:01 client-5 kernel: Lustre: 630:0:(import.c:529:import_select_connection()) lustre-MDT0000-mdc-ffff810333bd1400: tried all connections, increasing latency to 6s May 17 17:46:07 client-5 kernel: LustreError: 629:0:(client.c:2570:ptlrpc_replay_interpret()) @@@ status -116, old was 0 req@ffff81031788a000 x1369104143351874/t4294967300(4294967300) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:23/10 lens 360/424 e 1 to 0 dl 1305679593 ref 2 fl Interpret:R/ffffffff/ffffffff rc -116/-1 May 17 17:46:07 client-5 kernel: LustreError: 629:0:(obd_support.h:456:obd_fail_check_set()) *** obd_fail_loc=514 *** May 17 17:46:07 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff8102f95fc800: Connection restored to service lustre-MDT0000 using nid 192.168.4.128@o2ib. May 17 17:47:07 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff810333bd1400: Connection restored to service lustre-MDT0000 using nid 192.168.4.128@o2ib. May 17 17:47:07 client-5 kernel: Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request May 17 17:47:07 client-5 kernel: Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request May 17 17:47:08 client-5 kernel: Lustre: client ffff8102f95fc800 umount complete May 17 17:47:09 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 17:47:09 client-5 kernel: Lustre: 3360:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff8102ef675c00->192.168.4.128@o2ib netid 50000: select flavor null May 17 17:47:09 client-5 kernel: Lustre: 3360:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 12 previous similar messages May 17 17:47:09 client-5 kernel: Lustre: Client lustre-client has started May 17 17:47:09 client-5 kernel: Lustre: 3368:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 17:47:09 client-5 kernel: Lustre: 3368:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 7 previous similar messages May 17 17:47:09 client-5 xinetd[3391]: START: shell pid=3439 from=192.168.4.5 May 17 17:47:09 client-5 rshd[3440]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 17:47:09 client-5 xinetd[3391]: EXIT: shell status=0 pid=3439 duration=0(sec) May 17 17:47:09 client-5 xinetd[3391]: START: shell pid=3625 from=192.168.4.5 May 17 17:47:09 client-5 rshd[3626]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-dual test 0b: lost client during waiting for next transno ================================== 17:47:09 \(1305679629\)");echo XXRETCODE:$?' May 17 17:47:10 client-5 kernel: Lustre: DEBUG MARKER: == replay-dual test 0b: lost client during waiting for next transno ================================== 17:47:09 (1305679629) May 17 17:47:10 client-5 xinetd[3391]: EXIT: shell status=0 pid=3625 duration=1(sec) May 17 17:47:10 client-5 xinetd[3391]: START: shell pid=3679 from=192.168.4.5 May 17 17:47:10 client-5 rshd[3680]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 17:47:10 client-5 xinetd[3391]: EXIT: shell status=0 pid=3679 duration=0(sec) May 17 17:47:10 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 17:47:10 client-5 kernel: LustreError: 3745:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 17:47:10 client-5 kernel: LustreError: 3745:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 17:47:10 client-5 kernel: Lustre: client ffff8102ef675c00 umount complete May 17 17:47:26 client-5 kernel: Lustre: 628:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369104143352467 sent from MGC192.168.4.128@o2ib to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305679634] [real_sent 1305679634] [current 1305679646] [deadline 12s] [delay 0s] req@ffff8102eed54c00 x1369104143352467/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 192/192 e 0 to 1 dl 1305679646 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 17:47:26 client-5 kernel: Lustre: 628:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 1 previous similar message May 17 17:47:26 client-5 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail. May 17 17:47:28 client-5 kernel: Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request May 17 17:47:28 client-5 kernel: Lustre: Skipped 5 previous similar messages May 17 17:47:37 client-5 kernel: Lustre: 629:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369104143352484 sent from MGC192.168.4.128@o2ib to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305679646] [real_sent 1305679646] [current 1305679657] [deadline 11s] [delay 0s] req@ffff810317854800 x1369104143352484/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 368/392 e 0 to 1 dl 1305679657 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 17:47:37 client-5 kernel: Lustre: client ffff810333bd1400 umount complete May 17 17:47:37 client-5 kernel: Lustre: 3872:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 17:47:37 client-5 kernel: Lustre: 3872:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 6 previous similar messages May 17 17:47:37 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 17:47:37 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 17:47:37 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -16 May 17 17:48:07 client-5 last message repeated 5 times May 17 17:48:07 client-5 kernel: LustreError: Skipped 1 previous similar message May 17 17:48:27 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -16 May 17 17:48:27 client-5 kernel: LustreError: Skipped 3 previous similar messages May 17 17:49:02 client-5 kernel: Lustre: 3872:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-OST0000-osc-ffff810319af2800->192.168.4.129@o2ib netid 50000: select flavor null May 17 17:49:02 client-5 kernel: Lustre: 3872:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 1 previous similar message May 17 17:49:02 client-5 kernel: Lustre: Client lustre-client has started May 17 17:49:02 client-5 kernel: Lustre: 3893:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 17:49:02 client-5 kernel: Lustre: 3893:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message May 17 17:49:02 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 17:49:02 client-5 xinetd[3391]: START: shell pid=3973 from=192.168.4.5 May 17 17:49:02 client-5 rshd[3974]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 17:49:02 client-5 xinetd[3391]: EXIT: shell status=0 pid=3973 duration=0(sec) May 17 17:49:03 client-5 xinetd[3391]: START: shell pid=4145 from=192.168.4.5 May 17 17:49:03 client-5 rshd[4146]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-dual test 1: \|X\| simple create ============================================================= 17:49:02 \(1305679742\)");echo XXRETCODE:$?' May 17 17:49:03 client-5 kernel: Lustre: DEBUG MARKER: == replay-dual test 1: |X| simple create ============================================================= 17:49:02 (1305679742) May 17 17:49:03 client-5 xinetd[3391]: EXIT: shell status=0 pid=4145 duration=0(sec) May 17 17:49:03 client-5 xinetd[3391]: START: shell pid=4197 from=192.168.4.5 May 17 17:49:03 client-5 rshd[4198]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 17:49:03 client-5 xinetd[3391]: EXIT: shell status=0 pid=4197 duration=0(sec) May 17 17:49:03 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 17:49:14 client-5 kernel: Lustre: 628:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369104143352584 sent from MGC192.168.4.128@o2ib to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305679747] [real_sent 1305679747] [current 1305679754] [deadline 7s] [delay 0s] req@ffff8102e53d3000 x1369104143352584/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 192/192 e 0 to 1 dl 1305679754 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 17:49:14 client-5 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail. May 17 17:49:22 client-5 xinetd[3391]: START: shell pid=4380 from=192.168.4.5 May 17 17:49:22 client-5 rshd[4381]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 17:49:22 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_statfs operation failed with -107 May 17 17:49:22 client-5 kernel: LustreError: Skipped 6 previous similar messages May 17 17:49:22 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff810319af2800: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 17:49:24 client-5 kernel: Lustre: 630:0:(import.c:529:import_select_connection()) MGC192.168.4.128@o2ib: tried all connections, increasing latency to 6s May 17 17:49:24 client-5 kernel: Lustre: 630:0:(import.c:529:import_select_connection()) Skipped 1 previous similar message May 17 17:49:24 client-5 kernel: Lustre: 629:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf64922d to 0xfff98862bf649536 May 17 17:49:24 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff8102c8d33400: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 17:49:24 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 17:49:24 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Connection restored to service MGS using nid 192.168.4.128@o2ib. May 17 17:49:26 client-5 kernel: LustreError: 629:0:(client.c:2570:ptlrpc_replay_interpret()) @@@ status -116, old was 0 req@ffff8102e894d400 x1369104143352566/t12884901892(12884901892) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:23/10 lens 360/424 e 1 to 0 dl 1305679796 ref 2 fl Interpret:R/ffffffff/ffffffff rc -116/-1 May 17 17:49:26 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff810319af2800: Connection restored to service lustre-MDT0000 using nid 192.168.4.128@o2ib. May 17 17:49:26 client-5 xinetd[3391]: EXIT: shell status=0 pid=4380 duration=4(sec) May 17 17:49:26 client-5 xinetd[3391]: START: shell pid=4476 from=192.168.4.5 May 17 17:49:26 client-5 rshd[4478]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 17:49:26 client-5 xinetd[3391]: EXIT: shell status=0 pid=4476 duration=0(sec) May 17 17:49:27 client-5 xinetd[3391]: START: shell pid=4649 from=192.168.4.5 May 17 17:49:27 client-5 rshd[4650]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-dual test 2: \|X\| mkdir adir ================================================================ 17:49:27 \(1305679767\)");echo XXRETCODE:$?' May 17 17:49:27 client-5 kernel: Lustre: DEBUG MARKER: == replay-dual test 2: |X| mkdir adir ================================================================ 17:49:27 (1305679767) May 17 17:49:27 client-5 xinetd[3391]: EXIT: shell status=0 pid=4649 duration=0(sec) May 17 17:49:27 client-5 xinetd[3391]: START: shell pid=4700 from=192.168.4.5 May 17 17:49:27 client-5 rshd[4701]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 17:49:27 client-5 xinetd[3391]: EXIT: shell status=0 pid=4700 duration=0(sec) May 17 17:49:27 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 17:49:40 client-5 kernel: Lustre: 4415:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369104143352666 sent from MGC192.168.4.128@o2ib to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305679768] [real_sent 1305679768] [current 1305679780] [deadline 12s] [delay 0s] req@ffff810315cae800 x1369104143352666/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 296/352 e 0 to 1 dl 1305679780 ref 2 fl Rpc:X/ffffffff/ffffffff rc 0/-1 May 17 17:49:40 client-5 kernel: Lustre: 4415:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 1 previous similar message May 17 17:49:40 client-5 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail. May 17 17:49:40 client-5 kernel: LustreError: 4415:0:(client.c:1057:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@ffff810315cae800 x1369104143352698/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:/ffffffff/ffffffff rc 0/-1 May 17 17:49:40 client-5 kernel: LustreError: 4415:0:(client.c:1057:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@ffff810315cae800 x1369104143352699/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:/ffffffff/ffffffff rc 0/-1 May 17 17:49:44 client-5 kernel: LustreError: 4415:0:(client.c:1057:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@ffff810315cae800 x1369104143352702/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:/ffffffff/ffffffff rc 0/-1 May 17 17:49:44 client-5 kernel: LustreError: 4415:0:(client.c:1057:ptlrpc_import_delay_req()) Skipped 2 previous similar messages May 17 17:49:45 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff810319af2800: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 17:49:46 client-5 xinetd[3391]: START: shell pid=4883 from=192.168.4.5 May 17 17:49:46 client-5 rshd[4884]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 17:49:46 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff8102c8d33400: Connection restored to service lustre-MDT0000 using nid 192.168.4.128@o2ib. May 17 17:49:46 client-5 kernel: Lustre: Skipped 1 previous similar message May 17 17:49:46 client-5 xinetd[3391]: EXIT: shell status=0 pid=4883 duration=0(sec) May 17 17:49:46 client-5 xinetd[3391]: START: shell pid=4976 from=192.168.4.5 May 17 17:49:46 client-5 rshd[4977]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 17:49:46 client-5 xinetd[3391]: EXIT: shell status=0 pid=4976 duration=0(sec) May 17 17:49:47 client-5 xinetd[3391]: START: shell pid=5146 from=192.168.4.5 May 17 17:49:47 client-5 rshd[5149]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-dual test 3: \|X\| mkdir adir, mkdir adir\/bdir =============================================== 17:49:47 \(1305679787\)");echo XXRETCODE:$?' May 17 17:49:47 client-5 kernel: Lustre: DEBUG MARKER: == replay-dual test 3: |X| mkdir adir, mkdir adir/bdir =============================================== 17:49:47 (1305679787) May 17 17:49:47 client-5 xinetd[3391]: EXIT: shell status=0 pid=5146 duration=0(sec) May 17 17:49:47 client-5 xinetd[3391]: START: shell pid=5199 from=192.168.4.5 May 17 17:49:47 client-5 rshd[5200]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 17:49:47 client-5 xinetd[3391]: EXIT: shell status=0 pid=5199 duration=0(sec) May 17 17:49:47 client-5 kernel: LustreError: 4415:0:(client.c:1057:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@ffff8102d09e2800 x1369104143352744/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:/ffffffff/ffffffff rc 0/-1 May 17 17:49:47 client-5 kernel: LustreError: 4415:0:(client.c:1057:ptlrpc_import_delay_req()) Skipped 4 previous similar messages May 17 17:49:47 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 17:49:52 client-5 kernel: LustreError: 4415:0:(client.c:1057:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@ffff81022ff1c800 x1369104143352758/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:/ffffffff/ffffffff rc 0/-1 May 17 17:49:52 client-5 kernel: LustreError: 4415:0:(client.c:1057:ptlrpc_import_delay_req()) Skipped 4 previous similar messages May 17 17:49:55 client-5 kernel: Lustre: 630:0:(import.c:529:import_select_connection()) MGC192.168.4.128@o2ib: tried all connections, increasing latency to 11s May 17 17:49:56 client-5 kernel: LustreError: 4415:0:(client.c:1057:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@ffff81030758a400 x1369104143352778/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:/ffffffff/ffffffff rc 0/-1 May 17 17:49:56 client-5 kernel: LustreError: 4415:0:(client.c:1057:ptlrpc_import_delay_req()) Skipped 4 previous similar messages May 17 17:50:05 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff810319af2800: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 17:50:05 client-5 kernel: Lustre: Skipped 1 previous similar message May 17 17:50:06 client-5 xinetd[3391]: START: shell pid=5387 from=192.168.4.5 May 17 17:50:06 client-5 rshd[5388]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 17:50:06 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff810319af2800: Connection restored to service lustre-MDT0000 using nid 192.168.4.128@o2ib. May 17 17:50:06 client-5 kernel: Lustre: Skipped 1 previous similar message May 17 17:50:06 client-5 xinetd[3391]: EXIT: shell status=0 pid=5387 duration=0(sec) May 17 17:50:06 client-5 xinetd[3391]: START: shell pid=5482 from=192.168.4.5 May 17 17:50:06 client-5 rshd[5484]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 17:50:06 client-5 xinetd[3391]: EXIT: shell status=0 pid=5482 duration=0(sec) May 17 17:50:07 client-5 xinetd[3391]: START: shell pid=5655 from=192.168.4.5 May 17 17:50:07 client-5 rshd[5656]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-dual test 4: \|X\| mkdir adir \(-EEXIST\), mkdir adir\/bdir ===================================== 17:50:07 \(1305679807\)");echo XXRETCODE:$?' May 17 17:50:07 client-5 kernel: Lustre: DEBUG MARKER: == replay-dual test 4: |X| mkdir adir (-EEXIST), mkdir adir/bdir ===================================== 17:50:07 (1305679807) May 17 17:50:07 client-5 xinetd[3391]: EXIT: shell status=0 pid=5655 duration=0(sec) May 17 17:50:07 client-5 xinetd[3391]: START: shell pid=5707 from=192.168.4.5 May 17 17:50:07 client-5 rshd[5708]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 17:50:07 client-5 xinetd[3391]: EXIT: shell status=0 pid=5707 duration=0(sec) May 17 17:50:07 client-5 kernel: LustreError: 4415:0:(client.c:1057:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@ffff8102fb3b9400 x1369104143352840/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:/ffffffff/ffffffff rc 0/-1 May 17 17:50:07 client-5 kernel: LustreError: 4415:0:(client.c:1057:ptlrpc_import_delay_req()) Skipped 14 previous similar messages May 17 17:50:07 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 17:50:15 client-5 kernel: Lustre: 630:0:(import.c:529:import_select_connection()) MGC192.168.4.128@o2ib: tried all connections, increasing latency to 16s May 17 17:50:25 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff810319af2800: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 17:50:25 client-5 kernel: Lustre: Skipped 1 previous similar message May 17 17:50:26 client-5 xinetd[3391]: START: shell pid=5891 from=192.168.4.5 May 17 17:50:26 client-5 rshd[5892]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 17:50:26 client-5 kernel: Lustre: 628:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369104143352764 sent from lustre-MDT0000-mdc-ffff810319af2800 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305679795] [real_sent 1305679795] [current 1305679826] [deadline 31s] [delay 0s] req@ffff81027a3dec00 x1369104143352764/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 192/192 e 0 to 1 dl 1305679826 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 17:50:26 client-5 kernel: Lustre: 628:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 10 previous similar messages May 17 17:50:26 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff810319af2800: Connection restored to service lustre-MDT0000 using nid 192.168.4.128@o2ib. May 17 17:50:26 client-5 kernel: Lustre: Skipped 1 previous similar message May 17 17:50:26 client-5 xinetd[3391]: EXIT: shell status=0 pid=5891 duration=0(sec) May 17 17:50:26 client-5 xinetd[3391]: START: shell pid=5985 from=192.168.4.5 May 17 17:50:26 client-5 rshd[5987]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 17:50:26 client-5 xinetd[3391]: EXIT: shell status=0 pid=5985 duration=0(sec) May 17 17:50:27 client-5 kernel: LustreError: 4415:0:(client.c:1057:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@ffff81031f7b4000 x1369104143352929/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:/ffffffff/ffffffff rc 0/-1 May 17 17:50:27 client-5 kernel: LustreError: 4415:0:(client.c:1057:ptlrpc_import_delay_req()) Skipped 19 previous similar messages May 17 17:50:27 client-5 xinetd[3391]: START: shell pid=6158 from=192.168.4.5 May 17 17:50:27 client-5 rshd[6159]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-dual test 5: open, unlink \|X\| close ======================================================== 17:50:27 \(1305679827\)");echo XXRETCODE:$?' May 17 17:50:27 client-5 kernel: Lustre: DEBUG MARKER: == replay-dual test 5: open, unlink |X| close ======================================================== 17:50:27 (1305679827) May 17 17:50:27 client-5 xinetd[3391]: EXIT: shell status=0 pid=6158 duration=0(sec) May 17 17:50:27 client-5 xinetd[3391]: START: shell pid=6214 from=192.168.4.5 May 17 17:50:27 client-5 rshd[6215]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 17:50:27 client-5 xinetd[3391]: EXIT: shell status=0 pid=6214 duration=0(sec) May 17 17:50:27 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 17:50:40 client-5 kernel: Lustre: 630:0:(import.c:529:import_select_connection()) MGC192.168.4.128@o2ib: tried all connections, increasing latency to 21s May 17 17:50:46 client-5 xinetd[3391]: START: shell pid=6396 from=192.168.4.5 May 17 17:50:46 client-5 rshd[6397]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 17:50:46 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_statfs operation failed with -107 May 17 17:50:46 client-5 kernel: LustreError: Skipped 7 previous similar messages May 17 17:50:46 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff810319af2800: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 17:50:46 client-5 kernel: Lustre: Skipped 1 previous similar message May 17 17:50:50 client-5 xinetd[3391]: EXIT: shell status=0 pid=6396 duration=4(sec) May 17 17:50:50 client-5 xinetd[3391]: START: shell pid=6486 from=192.168.4.5 May 17 17:50:50 client-5 rshd[6487]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 17:50:50 client-5 xinetd[3391]: EXIT: shell status=0 pid=6486 duration=0(sec) May 17 17:50:51 client-5 xinetd[3391]: START: shell pid=6656 from=192.168.4.5 May 17 17:50:51 client-5 rshd[6659]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-dual test 6: open1, open2, unlink \|X\| close1 [fail mds1] close2 ============================ 17:50:51 \(1305679851\)");echo XXRETCODE:$?' May 17 17:50:51 client-5 kernel: Lustre: DEBUG MARKER: == replay-dual test 6: open1, open2, unlink |X| close1 [fail mds1] close2 ============================ 17:50:51 (1305679851) May 17 17:50:51 client-5 xinetd[3391]: EXIT: shell status=0 pid=6656 duration=0(sec) May 17 17:50:51 client-5 xinetd[3391]: START: shell pid=6716 from=192.168.4.5 May 17 17:50:51 client-5 rshd[6718]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 17:50:51 client-5 xinetd[3391]: EXIT: shell status=0 pid=6716 duration=0(sec) May 17 17:50:51 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 17:51:00 client-5 kernel: LustreError: 4415:0:(client.c:1057:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@ffff810289b3e800 x1369104143353070/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:/ffffffff/ffffffff rc 0/-1 May 17 17:51:00 client-5 kernel: LustreError: 4415:0:(client.c:1057:ptlrpc_import_delay_req()) Skipped 39 previous similar messages May 17 17:51:10 client-5 xinetd[3391]: START: shell pid=6903 from=192.168.4.5 May 17 17:51:10 client-5 rshd[6904]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 17:51:10 client-5 kernel: Lustre: 630:0:(import.c:529:import_select_connection()) MGC192.168.4.128@o2ib: tried all connections, increasing latency to 21s May 17 17:51:10 client-5 kernel: Lustre: 629:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf649536 to 0xfff98862bf64a5ac May 17 17:51:10 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 17:51:10 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Connection restored to service MGS using nid 192.168.4.128@o2ib. May 17 17:51:10 client-5 kernel: Lustre: Skipped 3 previous similar messages May 17 17:51:11 client-5 kernel: LustreError: 629:0:(client.c:2570:ptlrpc_replay_interpret()) @@@ status -116, old was 0 req@ffff8102e9af3000 x1369104143352944/t30064771083(30064771083) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:23/10 lens 360/424 e 0 to 0 dl 1305679877 ref 2 fl Interpret:R/ffffffff/ffffffff rc -116/-1 May 17 17:51:11 client-5 xinetd[3391]: EXIT: shell status=0 pid=6903 duration=1(sec) May 17 17:51:11 client-5 xinetd[3391]: START: shell pid=6993 from=192.168.4.5 May 17 17:51:11 client-5 rshd[6995]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 17:51:11 client-5 xinetd[3391]: EXIT: shell status=0 pid=6993 duration=0(sec) May 17 17:51:12 client-5 xinetd[3391]: START: shell pid=7166 from=192.168.4.5 May 17 17:51:12 client-5 rshd[7167]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-dual test 8: replay of resent request ====================================================== 17:51:12 \(1305679872\)");echo XXRETCODE:$?' May 17 17:51:12 client-5 kernel: Lustre: DEBUG MARKER: == replay-dual test 8: replay of resent request ====================================================== 17:51:12 (1305679872) May 17 17:51:12 client-5 xinetd[3391]: EXIT: shell status=0 pid=7166 duration=0(sec) May 17 17:51:12 client-5 xinetd[3391]: START: shell pid=7217 from=192.168.4.5 May 17 17:51:12 client-5 rshd[7218]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 17:51:12 client-5 xinetd[3391]: EXIT: shell status=0 pid=7217 duration=0(sec) May 17 17:51:12 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 17:51:31 client-5 kernel: Lustre: 628:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369104143353056 sent from lustre-MDT0000-mdc-ffff810319af2800 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305679860] [real_sent 1305679860] [current 1305679891] [deadline 31s] [delay 0s] req@ffff8102d1686000 x1369104143353056/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 192/192 e 0 to 1 dl 1305679891 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 17:51:31 client-5 kernel: Lustre: 628:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 9 previous similar messages May 17 17:51:43 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff810319af2800: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 17:51:43 client-5 kernel: Lustre: Skipped 3 previous similar messages May 17 17:52:02 client-5 xinetd[3391]: START: shell pid=7424 from=192.168.4.5 May 17 17:52:02 client-5 rshd[7425]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 17:52:03 client-5 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail. May 17 17:52:03 client-5 kernel: Lustre: 629:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf64a5ac to 0xfff98862bf64a8bc May 17 17:52:03 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 17:52:04 client-5 kernel: LustreError: 629:0:(client.c:2570:ptlrpc_replay_interpret()) @@@ status -116, old was 0 req@ffff81032fa3b000 x1369104143353039/t34359738377(34359738377) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:23/10 lens 360/424 e 0 to 0 dl 1305679956 ref 2 fl Interpret:R/ffffffff/ffffffff rc -116/-1 May 17 17:52:04 client-5 xinetd[3391]: EXIT: shell status=0 pid=7424 duration=2(sec) May 17 17:52:04 client-5 xinetd[3391]: START: shell pid=7517 from=192.168.4.5 May 17 17:52:04 client-5 rshd[7518]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 17:52:04 client-5 xinetd[3391]: EXIT: shell status=0 pid=7517 duration=0(sec) May 17 17:52:04 client-5 xinetd[3391]: START: shell pid=7689 from=192.168.4.5 May 17 17:52:04 client-5 rshd[7690]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-dual test 9: resending a replayed create =================================================== 17:52:04 \(1305679924\)");echo XXRETCODE:$?' May 17 17:52:04 client-5 kernel: Lustre: DEBUG MARKER: == replay-dual test 9: resending a replayed create =================================================== 17:52:04 (1305679924) May 17 17:52:04 client-5 xinetd[3391]: EXIT: shell status=0 pid=7689 duration=0(sec) May 17 17:52:04 client-5 xinetd[3391]: START: shell pid=7740 from=192.168.4.5 May 17 17:52:04 client-5 rshd[7741]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 17:52:04 client-5 xinetd[3391]: EXIT: shell status=0 pid=7740 duration=0(sec) May 17 17:52:05 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 17:52:23 client-5 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail. May 17 17:52:23 client-5 kernel: Lustre: 629:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf64a8bc to 0xfff98862bf64ab94 May 17 17:52:23 client-5 kernel: LustreError: 4415:0:(client.c:1057:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@ffff810312c52800 x1369104143353360/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:/ffffffff/ffffffff rc 0/-1 May 17 17:52:23 client-5 kernel: LustreError: 4415:0:(client.c:1057:ptlrpc_import_delay_req()) Skipped 14 previous similar messages May 17 17:52:23 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 17:52:23 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Connection restored to service MGS using nid 192.168.4.128@o2ib. May 17 17:52:23 client-5 kernel: Lustre: Skipped 6 previous similar messages May 17 17:52:23 client-5 xinetd[3391]: START: shell pid=7937 from=192.168.4.5 May 17 17:52:24 client-5 rshd[7938]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 17:53:00 client-5 xinetd[3391]: EXIT: shell status=0 pid=7937 duration=37(sec) May 17 17:53:00 client-5 xinetd[3391]: START: shell pid=8037 from=192.168.4.5 May 17 17:53:00 client-5 rshd[8040]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 17:53:00 client-5 xinetd[3391]: EXIT: shell status=0 pid=8037 duration=0(sec) May 17 17:53:00 client-5 xinetd[3391]: START: shell pid=8210 from=192.168.4.5 May 17 17:53:00 client-5 rshd[8212]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-dual test 10: resending a replayed unlink ================================================== 17:53:00 \(1305679980\)");echo XXRETCODE:$?' May 17 17:53:00 client-5 kernel: Lustre: DEBUG MARKER: == replay-dual test 10: resending a replayed unlink ================================================== 17:53:00 (1305679980) May 17 17:53:00 client-5 xinetd[3391]: EXIT: shell status=0 pid=8210 duration=0(sec) May 17 17:53:00 client-5 xinetd[3391]: START: shell pid=8263 from=192.168.4.5 May 17 17:53:00 client-5 rshd[8264]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 17:53:01 client-5 xinetd[3391]: EXIT: shell status=0 pid=8263 duration=1(sec) May 17 17:53:01 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 17:53:18 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The obd_ping operation failed with -107 May 17 17:53:18 client-5 kernel: LustreError: Skipped 9 previous similar messages May 17 17:53:18 client-5 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail. May 17 17:53:18 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff810319af2800: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 17:53:18 client-5 kernel: Lustre: Skipped 4 previous similar messages May 17 17:53:18 client-5 kernel: Lustre: 629:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf64ab94 to 0xfff98862bf64aef8 May 17 17:53:19 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 17:53:20 client-5 xinetd[3391]: START: shell pid=8456 from=192.168.4.5 May 17 17:53:20 client-5 rshd[8457]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 17:53:41 client-5 kernel: Lustre: 628:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369104143353495 sent from lustre-MDT0000-mdc-ffff810319af2800 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305679988] [real_sent 1305679988] [current 1305680021] [deadline 33s] [delay 0s] req@ffff810312e49400 x1369104143353495/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 192/192 e 0 to 1 dl 1305680021 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 17:53:41 client-5 kernel: Lustre: 628:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 13 previous similar messages May 17 17:53:51 client-5 kernel: LustreError: 629:0:(client.c:2515:ptlrpc_replay_interpret()) request replay timed out, restarting recovery May 17 17:53:51 client-5 xinetd[3391]: EXIT: shell status=0 pid=8456 duration=31(sec) May 17 17:53:51 client-5 xinetd[3391]: START: shell pid=8564 from=192.168.4.5 May 17 17:53:51 client-5 rshd[8565]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 17:53:51 client-5 xinetd[3391]: EXIT: shell status=0 pid=8564 duration=0(sec) May 17 17:53:51 client-5 xinetd[3391]: START: shell pid=8736 from=192.168.4.5 May 17 17:53:51 client-5 rshd[8737]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-dual test 11: both clients timeout during replay =========================================== 17:53:51 \(1305680031\)");echo XXRETCODE:$?' May 17 17:53:51 client-5 kernel: Lustre: DEBUG MARKER: == replay-dual test 11: both clients timeout during replay =========================================== 17:53:51 (1305680031) May 17 17:53:51 client-5 xinetd[3391]: EXIT: shell status=0 pid=8736 duration=0(sec) May 17 17:53:51 client-5 xinetd[3391]: START: shell pid=8787 from=192.168.4.5 May 17 17:53:51 client-5 rshd[8788]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 17:53:51 client-5 xinetd[3391]: EXIT: shell status=0 pid=8787 duration=0(sec) May 17 17:53:52 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 17:54:11 client-5 xinetd[3391]: EXIT: login status=0 pid=25656 duration=4820(sec) May 17 17:54:13 client-5 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail. May 17 17:54:13 client-5 kernel: Lustre: 629:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf64aef8 to 0xfff98862bf64b294 May 17 17:54:13 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 17:54:45 client-5 kernel: LustreError: 629:0:(client.c:2515:ptlrpc_replay_interpret()) request replay timed out, restarting recovery May 17 17:54:45 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff8102c8d33400: Connection restored to service lustre-MDT0000 using nid 192.168.4.128@o2ib. May 17 17:54:45 client-5 kernel: Lustre: Skipped 6 previous similar messages May 17 17:54:50 client-5 xinetd[3391]: START: shell pid=9047 from=192.168.4.5 May 17 17:54:50 client-5 rshd[9048]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 17:54:50 client-5 xinetd[3391]: EXIT: shell status=0 pid=9047 duration=0(sec) May 17 17:54:50 client-5 xinetd[3391]: START: shell pid=9219 from=192.168.4.5 May 17 17:54:50 client-5 rshd[9220]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-dual test 12: open resend timeout ========================================================== 17:54:50 \(1305680090\)");echo XXRETCODE:$?' May 17 17:54:50 client-5 kernel: Lustre: DEBUG MARKER: == replay-dual test 12: open resend timeout ========================================================== 17:54:50 (1305680090) May 17 17:54:50 client-5 xinetd[3391]: EXIT: shell status=0 pid=9219 duration=0(sec) May 17 17:54:50 client-5 xinetd[3391]: START: shell pid=9270 from=192.168.4.5 May 17 17:54:50 client-5 rshd[9271]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 17:54:51 client-5 xinetd[3391]: EXIT: shell status=0 pid=9270 duration=1(sec) May 17 17:54:51 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 17:55:08 client-5 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail. May 17 17:55:08 client-5 kernel: Lustre: 629:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf64b294 to 0xfff98862bf64b629 May 17 17:55:10 client-5 xinetd[3391]: START: shell pid=9475 from=192.168.4.5 May 17 17:55:10 client-5 rshd[9477]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 17:55:18 client-5 xinetd[3391]: EXIT: shell status=0 pid=9475 duration=8(sec) May 17 17:55:19 client-5 xinetd[3391]: START: shell pid=9572 from=192.168.4.5 May 17 17:55:19 client-5 rshd[9573]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 17:55:19 client-5 xinetd[3391]: EXIT: shell status=0 pid=9572 duration=0(sec) May 17 17:55:19 client-5 xinetd[3391]: START: shell pid=9744 from=192.168.4.5 May 17 17:55:19 client-5 rshd[9745]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-dual test 13: close resend timeout ========================================================= 17:55:19 \(1305680119\)");echo XXRETCODE:$?' May 17 17:55:19 client-5 kernel: Lustre: DEBUG MARKER: == replay-dual test 13: close resend timeout ========================================================= 17:55:19 (1305680119) May 17 17:55:19 client-5 xinetd[3391]: EXIT: shell status=0 pid=9744 duration=0(sec) May 17 17:55:19 client-5 xinetd[3391]: START: shell pid=9797 from=192.168.4.5 May 17 17:55:19 client-5 rshd[9799]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 17:55:19 client-5 xinetd[3391]: EXIT: shell status=0 pid=9797 duration=0(sec) May 17 17:55:19 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 17:55:38 client-5 xinetd[3391]: START: shell pid=9996 from=192.168.4.5 May 17 17:55:38 client-5 rshd[9997]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 17:55:38 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff810319af2800: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 17:55:38 client-5 kernel: Lustre: Skipped 5 previous similar messages May 17 17:55:38 client-5 kernel: Lustre: 629:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf64b629 to 0xfff98862bf64b978 May 17 17:55:38 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 17:55:38 client-5 kernel: Lustre: Skipped 1 previous similar message May 17 17:55:40 client-5 xinetd[3391]: EXIT: shell status=0 pid=9996 duration=2(sec) May 17 17:55:40 client-5 xinetd[3391]: START: shell pid=10088 from=192.168.4.5 May 17 17:55:40 client-5 rshd[10091]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 17:55:40 client-5 xinetd[3391]: EXIT: shell status=0 pid=10088 duration=0(sec) May 17 17:55:40 client-5 xinetd[3391]: START: shell pid=10262 from=192.168.4.5 May 17 17:55:40 client-5 rshd[10263]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-dual test 14b: delete ost orphans if gap occured in objids due to VBR ====================== 17:55:40 \(1305680140\)");echo XXRETCODE:$?' May 17 17:55:40 client-5 kernel: Lustre: DEBUG MARKER: == replay-dual test 14b: delete ost orphans if gap occured in objids due to VBR ====================== 17:55:40 (1305680140) May 17 17:55:40 client-5 xinetd[3391]: EXIT: shell status=0 pid=10262 duration=0(sec) May 17 17:55:43 client-5 xinetd[3391]: START: shell pid=10362 from=192.168.4.5 May 17 17:55:43 client-5 rshd[10363]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 17:55:44 client-5 xinetd[3391]: EXIT: shell status=0 pid=10362 duration=1(sec) May 17 17:55:44 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 17:55:44 client-5 kernel: LustreError: 10429:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 17:55:44 client-5 kernel: LustreError: 10429:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 17:55:44 client-5 kernel: Lustre: client ffff8102c8d33400 umount complete May 17 17:56:03 client-5 xinetd[3391]: START: shell pid=10548 from=192.168.4.5 May 17 17:56:03 client-5 rshd[10549]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 17:56:03 client-5 kernel: Lustre: 629:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf64b978 to 0xfff98862bf64be95 May 17 17:57:12 client-5 xinetd[3391]: START: login pid=10589 from=10.10.0.1 May 17 17:57:33 client-5 kernel: LustreError: 629:0:(client.c:2570:ptlrpc_replay_interpret()) @@@ status -116, old was 0 req@ffff810319873800 x1369104143354059/t64424509446(64424509446) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:23/10 lens 360/424 e 6 to 0 dl 1305680282 ref 2 fl Interpret:R/ffffffff/ffffffff rc -116/-1 May 17 17:57:33 client-5 kernel: LustreError: 629:0:(client.c:2570:ptlrpc_replay_interpret()) Skipped 1 previous similar message May 17 17:57:33 client-5 xinetd[3391]: EXIT: shell status=0 pid=10548 duration=90(sec) May 17 17:57:33 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 17:57:33 client-5 kernel: Lustre: 10667:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff8102c496d400->192.168.4.128@o2ib netid 50000: select flavor null May 17 17:57:33 client-5 kernel: Lustre: 10667:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 12 previous similar messages May 17 17:57:33 client-5 kernel: Lustre: Client lustre-client has started May 17 17:57:33 client-5 kernel: Lustre: Skipped 1 previous similar message May 17 17:57:33 client-5 kernel: Lustre: 10674:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 17:57:33 client-5 kernel: Lustre: 10674:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 3 previous similar messages May 17 17:57:37 client-5 xinetd[3391]: START: shell pid=10780 from=192.168.4.5 May 17 17:57:37 client-5 rshd[10781]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark before 2516856, after 2516856");echo XXRETCODE:$?' May 17 17:57:37 client-5 kernel: Lustre: DEBUG MARKER: before 2516856, after 2516856 May 17 17:57:37 client-5 xinetd[3391]: EXIT: shell status=0 pid=10780 duration=0(sec) May 17 17:57:37 client-5 xinetd[3391]: START: shell pid=10871 from=192.168.4.5 May 17 17:57:37 client-5 rshd[10872]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 17:57:37 client-5 xinetd[3391]: EXIT: shell status=0 pid=10871 duration=0(sec) May 17 17:57:37 client-5 xinetd[3391]: START: shell pid=11041 from=192.168.4.5 May 17 17:57:37 client-5 rshd[11044]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-dual test 15a: timeout waiting for lost client during replay, 1 client completes =========== 17:57:37 \(1305680257\)");echo XXRETCODE:$?' May 17 17:57:37 client-5 kernel: Lustre: DEBUG MARKER: == replay-dual test 15a: timeout waiting for lost client during replay, 1 client completes =========== 17:57:37 (1305680257) May 17 17:57:37 client-5 xinetd[3391]: EXIT: shell status=0 pid=11041 duration=0(sec) May 17 17:57:37 client-5 xinetd[3391]: START: shell pid=11094 from=192.168.4.5 May 17 17:57:37 client-5 rshd[11095]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 17:57:37 client-5 xinetd[3391]: EXIT: shell status=0 pid=11094 duration=0(sec) May 17 17:57:38 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 17:57:38 client-5 kernel: LustreError: 11160:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 17:57:38 client-5 kernel: LustreError: 11160:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 17:57:38 client-5 kernel: Lustre: client ffff8102c496d400 umount complete May 17 17:57:57 client-5 xinetd[3391]: START: shell pid=11279 from=192.168.4.5 May 17 17:57:57 client-5 rshd[11280]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 17:57:57 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_statfs operation failed with -107 May 17 17:57:57 client-5 kernel: LustreError: Skipped 15 previous similar messages May 17 17:57:58 client-5 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail. May 17 17:57:58 client-5 kernel: LustreError: Skipped 2 previous similar messages May 17 17:57:58 client-5 kernel: Lustre: 629:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf64be95 to 0xfff98862bf64c778 May 17 17:57:58 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 17:57:58 client-5 kernel: Lustre: Skipped 1 previous similar message May 17 17:58:16 client-5 kernel: Lustre: 628:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369104143354402 sent from lustre-MDT0000-mdc-ffff810319af2800 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305680263] [real_sent 1305680263] [current 1305680296] [deadline 33s] [delay 0s] req@ffff810319b2c000 x1369104143354402/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 192/192 e 0 to 1 dl 1305680296 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 17:58:16 client-5 kernel: Lustre: 628:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 20 previous similar messages May 17 17:59:28 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff810319af2800: Connection restored to service lustre-MDT0000 using nid 192.168.4.128@o2ib. May 17 17:59:28 client-5 kernel: Lustre: Skipped 10 previous similar messages May 17 17:59:28 client-5 xinetd[3391]: EXIT: shell status=0 pid=11279 duration=91(sec) May 17 17:59:28 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 17:59:28 client-5 kernel: Lustre: 11331:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff8103160f9c00->192.168.4.128@o2ib netid 50000: select flavor null May 17 17:59:28 client-5 kernel: Lustre: 11331:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 6 previous similar messages May 17 17:59:28 client-5 kernel: Lustre: Client lustre-client has started May 17 17:59:28 client-5 kernel: Lustre: 11338:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 17:59:28 client-5 kernel: Lustre: 11338:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message May 17 17:59:28 client-5 xinetd[3391]: START: shell pid=11397 from=192.168.4.5 May 17 17:59:28 client-5 rshd[11398]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 17:59:28 client-5 xinetd[3391]: EXIT: shell status=0 pid=11397 duration=0(sec) May 17 17:59:29 client-5 xinetd[3391]: START: shell pid=11560 from=192.168.4.5 May 17 17:59:29 client-5 rshd[11561]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark SKIP: replay-dual test_15c skipping ALWAYS excluded test 15c");echo XXRETCODE:$?' May 17 17:59:29 client-5 kernel: Lustre: DEBUG MARKER: SKIP: replay-dual test_15c skipping ALWAYS excluded test 15c May 17 17:59:29 client-5 xinetd[3391]: EXIT: shell status=0 pid=11560 duration=0(sec) May 17 17:59:29 client-5 xinetd[3391]: START: shell pid=11665 from=192.168.4.5 May 17 17:59:29 client-5 rshd[11666]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-dual test 16: fail MDS during recovery \(3571\) ============================================== 17:59:29 \(1305680369\)");echo XXRETCODE:$?' May 17 17:59:29 client-5 kernel: Lustre: DEBUG MARKER: == replay-dual test 16: fail MDS during recovery (3571) ============================================== 17:59:29 (1305680369) May 17 17:59:29 client-5 xinetd[3391]: EXIT: shell status=0 pid=11665 duration=0(sec) May 17 17:59:29 client-5 xinetd[3391]: START: shell pid=11716 from=192.168.4.5 May 17 17:59:29 client-5 rshd[11717]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 17:59:29 client-5 xinetd[3391]: EXIT: shell status=0 pid=11716 duration=0(sec) May 17 17:59:30 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 17:59:30 client-5 kernel: LustreError: 11782:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 17:59:30 client-5 kernel: LustreError: 11782:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 17:59:30 client-5 kernel: Lustre: client ffff8103160f9c00 umount complete May 17 17:59:48 client-5 kernel: Lustre: 629:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf64c778 to 0xfff98862bf64d491 May 17 18:00:13 client-5 kernel: Lustre: 630:0:(import.c:529:import_select_connection()) lustre-MDT0000-mdc-ffff810319af2800: tried all connections, increasing latency to 6s May 17 18:00:20 client-5 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail. May 17 18:00:20 client-5 kernel: LustreError: Skipped 1 previous similar message May 17 18:00:26 client-5 xinetd[3391]: START: shell pid=12022 from=192.168.4.5 May 17 18:00:26 client-5 rshd[12023]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 18:00:30 client-5 kernel: Lustre: 629:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf64d491 to 0xfff98862bf64d689 May 17 18:01:55 client-5 kernel: LustreError: 629:0:(client.c:2570:ptlrpc_replay_interpret()) @@@ status 0, old was -19 req@ffff8103191e3400 x1369104143354661/t73014444059(73014444059) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 464/416 e 4 to 0 dl 1305680544 ref 2 fl Interpret:R/ffffffff/ffffffff rc 0/-1 May 17 18:01:55 client-5 xinetd[3391]: EXIT: shell status=0 pid=12022 duration=89(sec) May 17 18:01:56 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 18:01:56 client-5 kernel: Lustre: 12079:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff8103193c9800->192.168.4.128@o2ib netid 50000: select flavor null May 17 18:01:56 client-5 kernel: Lustre: 12079:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 6 previous similar messages May 17 18:01:56 client-5 kernel: Lustre: Client lustre-client has started May 17 18:01:56 client-5 kernel: Lustre: 12086:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 18:01:56 client-5 kernel: Lustre: 12086:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message May 17 18:01:56 client-5 xinetd[3391]: START: shell pid=12143 from=192.168.4.5 May 17 18:01:56 client-5 rshd[12146]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 18:01:56 client-5 xinetd[3391]: EXIT: shell status=0 pid=12143 duration=0(sec) May 17 18:01:56 client-5 xinetd[3391]: START: shell pid=12317 from=192.168.4.5 May 17 18:01:56 client-5 rshd[12318]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-dual test 17: fail OST during recovery \(3571\) ============================================== 18:01:56 \(1305680516\)");echo XXRETCODE:$?' May 17 18:01:56 client-5 kernel: Lustre: DEBUG MARKER: == replay-dual test 17: fail OST during recovery (3571) ============================================== 18:01:56 (1305680516) May 17 18:01:56 client-5 xinetd[3391]: EXIT: shell status=0 pid=12317 duration=0(sec) May 17 18:01:56 client-5 xinetd[3391]: START: shell pid=12395 from=192.168.4.5 May 17 18:01:56 client-5 rshd[12396]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 18:01:57 client-5 xinetd[3391]: EXIT: shell status=0 pid=12395 duration=1(sec) May 17 18:01:57 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-OST0000 May 17 18:01:57 client-5 kernel: LustreError: 12459:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 18:01:57 client-5 kernel: LustreError: 12459:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 18:01:57 client-5 kernel: Lustre: client ffff8103193c9800 umount complete May 17 18:02:01 client-5 kernel: Lustre: lustre-OST0000-osc-ffff810319af2800: Connection to service lustre-OST0000 via nid 192.168.4.129@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 18:02:01 client-5 kernel: Lustre: Skipped 4 previous similar messages May 17 18:02:11 client-5 kernel: Lustre: 630:0:(import.c:529:import_select_connection()) lustre-OST0000-osc-ffff810319af2800: tried all connections, increasing latency to 6s May 17 18:02:11 client-5 kernel: Lustre: 630:0:(import.c:529:import_select_connection()) Skipped 2 previous similar messages May 17 18:02:43 client-5 xinetd[3391]: START: shell pid=12695 from=192.168.4.5 May 17 18:02:43 client-5 rshd[12696]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 18:04:11 client-5 xinetd[3391]: EXIT: shell status=0 pid=12695 duration=88(sec) May 17 18:04:11 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 18:04:11 client-5 kernel: Lustre: 12743:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff810314ec4000->192.168.4.128@o2ib netid 50000: select flavor null May 17 18:04:11 client-5 kernel: Lustre: 12743:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 6 previous similar messages May 17 18:04:11 client-5 kernel: Lustre: Client lustre-client has started May 17 18:04:11 client-5 kernel: Lustre: 12750:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 18:04:11 client-5 kernel: Lustre: 12750:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message May 17 18:04:12 client-5 xinetd[3391]: START: shell pid=12809 from=192.168.4.5 May 17 18:04:12 client-5 rshd[12810]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 18:04:12 client-5 xinetd[3391]: EXIT: shell status=0 pid=12809 duration=0(sec) May 17 18:04:12 client-5 xinetd[3391]: START: shell pid=12981 from=192.168.4.5 May 17 18:04:12 client-5 rshd[12982]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-dual test 18: ldlm_handle_enqueue succeeds on evicted export \(3822\) ======================== 18:04:12 \(1305680652\)");echo XXRETCODE:$?' May 17 18:04:12 client-5 kernel: Lustre: DEBUG MARKER: == replay-dual test 18: ldlm_handle_enqueue succeeds on evicted export (3822) ======================== 18:04:12 (1305680652) May 17 18:04:12 client-5 xinetd[3391]: EXIT: shell status=0 pid=12981 duration=0(sec) May 17 18:04:13 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc start May 17 18:04:13 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks mdc stop May 17 18:04:53 client-5 xinetd[3391]: START: shell pid=13123 from=192.168.4.5 May 17 18:04:53 client-5 rshd[13124]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 18:04:53 client-5 xinetd[3391]: EXIT: shell status=0 pid=13123 duration=0(sec) May 17 18:04:54 client-5 xinetd[3391]: START: shell pid=13295 from=192.168.4.5 May 17 18:04:54 client-5 rshd[13296]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-dual test 19: resend of open request ======================================================= 18:04:54 \(1305680694\)");echo XXRETCODE:$?' May 17 18:04:54 client-5 kernel: Lustre: DEBUG MARKER: == replay-dual test 19: resend of open request ======================================================= 18:04:54 (1305680694) May 17 18:04:54 client-5 xinetd[3391]: EXIT: shell status=0 pid=13295 duration=0(sec) May 17 18:04:54 client-5 xinetd[3391]: START: shell pid=13346 from=192.168.4.5 May 17 18:04:54 client-5 rshd[13347]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 18:04:54 client-5 xinetd[3391]: EXIT: shell status=0 pid=13346 duration=0(sec) May 17 18:04:54 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 18:06:38 client-5 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail. May 17 18:06:41 client-5 xinetd[3391]: START: shell pid=13554 from=192.168.4.5 May 17 18:06:41 client-5 rshd[13555]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 18:06:41 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_statfs operation failed with -107 May 17 18:06:41 client-5 kernel: LustreError: Skipped 7 previous similar messages May 17 18:06:44 client-5 kernel: LustreError: 629:0:(client.c:2570:ptlrpc_replay_interpret()) @@@ status -116, old was 0 req@ffff8102fa845000 x1369104143355403/t77309411439(77309411439) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:23/10 lens 360/424 e 1 to 0 dl 1305680855 ref 2 fl Interpret:R/ffffffff/ffffffff rc -116/-1 May 17 18:06:44 client-5 xinetd[3391]: EXIT: shell status=0 pid=13554 duration=3(sec) May 17 18:06:44 client-5 xinetd[3391]: START: shell pid=13645 from=192.168.4.5 May 17 18:06:44 client-5 rshd[13647]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 18:06:44 client-5 xinetd[3391]: EXIT: shell status=0 pid=13645 duration=0(sec) May 17 18:06:44 client-5 xinetd[3391]: START: shell pid=13817 from=192.168.4.5 May 17 18:06:44 client-5 rshd[13819]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-dual test 20: recovery time is not increasing ============================================== 18:06:44 \(1305680804\)");echo XXRETCODE:$?' May 17 18:06:44 client-5 kernel: Lustre: DEBUG MARKER: == replay-dual test 20: recovery time is not increasing ============================================== 18:06:44 (1305680804) May 17 18:06:44 client-5 xinetd[3391]: EXIT: shell status=0 pid=13817 duration=0(sec) May 17 18:06:44 client-5 xinetd[3391]: START: shell pid=13870 from=192.168.4.5 May 17 18:06:44 client-5 rshd[13871]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 18:06:44 client-5 xinetd[3391]: EXIT: shell status=0 pid=13870 duration=0(sec) May 17 18:06:45 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 18:06:45 client-5 kernel: LustreError: 13936:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 18:06:45 client-5 kernel: LustreError: 13936:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 18:06:45 client-5 kernel: Lustre: client ffff810314ec4000 umount complete May 17 18:06:49 client-5 kernel: Lustre: 629:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369104143355852 sent from MGC192.168.4.128@o2ib to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305680798] [real_sent 1305680798] [current 1305680809] [deadline 11s] [delay 0s] req@ffff810315cadc00 x1369104143355852/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 368/392 e 0 to 1 dl 1305680809 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 18:06:49 client-5 kernel: Lustre: 629:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 10 previous similar messages May 17 18:06:53 client-5 kernel: Lustre: 630:0:(import.c:529:import_select_connection()) MGC192.168.4.128@o2ib: tried all connections, increasing latency to 11s May 17 18:07:04 client-5 xinetd[3391]: START: shell pid=14055 from=192.168.4.5 May 17 18:07:04 client-5 rshd[14056]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 18:07:13 client-5 kernel: Lustre: 629:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf64d689 to 0xfff98862bf64edb2 May 17 18:07:13 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 18:07:14 client-5 kernel: Lustre: Skipped 2 previous similar messages May 17 18:09:19 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff810319af2800: Connection restored to service lustre-MDT0000 using nid 192.168.4.128@o2ib. May 17 18:09:19 client-5 kernel: Lustre: Skipped 8 previous similar messages May 17 18:09:19 client-5 xinetd[3391]: EXIT: shell status=0 pid=14055 duration=135(sec) May 17 18:09:19 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 18:09:19 client-5 kernel: Lustre: 14111:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff810314ec4000->192.168.4.128@o2ib netid 50000: select flavor null May 17 18:09:19 client-5 kernel: Lustre: 14111:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 6 previous similar messages May 17 18:09:19 client-5 kernel: Lustre: Client lustre-client has started May 17 18:09:19 client-5 kernel: Lustre: 14118:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 18:09:19 client-5 kernel: Lustre: 14118:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message May 17 18:09:19 client-5 xinetd[3391]: START: shell pid=14139 from=192.168.4.5 May 17 18:09:19 client-5 rshd[14140]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 18:09:19 client-5 xinetd[3391]: EXIT: shell status=0 pid=14139 duration=0(sec) May 17 18:09:19 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 18:09:20 client-5 kernel: LustreError: 14205:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 18:09:20 client-5 kernel: LustreError: 14205:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 18:09:20 client-5 kernel: Lustre: client ffff810314ec4000 umount complete May 17 18:09:38 client-5 xinetd[3391]: START: shell pid=14324 from=192.168.4.5 May 17 18:09:38 client-5 rshd[14325]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 18:09:39 client-5 kernel: Lustre: 629:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf64edb2 to 0xfff98862bf64f0b4 May 17 18:11:53 client-5 xinetd[3391]: EXIT: shell status=0 pid=14324 duration=135(sec) May 17 18:11:53 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 18:11:53 client-5 kernel: Lustre: Client lustre-client has started May 17 18:11:53 client-5 xinetd[3391]: START: shell pid=14450 from=192.168.4.5 May 17 18:11:53 client-5 rshd[14451]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 18:11:53 client-5 xinetd[3391]: EXIT: shell status=0 pid=14450 duration=0(sec) May 17 18:11:53 client-5 xinetd[3391]: START: shell pid=14620 from=192.168.4.5 May 17 18:11:53 client-5 rshd[14623]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-dual test 21a: commit on sharing =========================================================== 18:11:53 \(1305681113\)");echo XXRETCODE:$?' May 17 18:11:53 client-5 kernel: Lustre: DEBUG MARKER: == replay-dual test 21a: commit on sharing =========================================================== 18:11:53 (1305681113) May 17 18:11:53 client-5 xinetd[3391]: EXIT: shell status=0 pid=14620 duration=0(sec) May 17 18:11:54 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 18:11:54 client-5 kernel: LustreError: 14701:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 18:11:54 client-5 kernel: LustreError: 14701:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 18:11:54 client-5 kernel: Lustre: client ffff810315cad400 umount complete May 17 18:12:12 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff810319af2800: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 18:12:12 client-5 kernel: Lustre: Skipped 5 previous similar messages May 17 18:12:13 client-5 kernel: Lustre: 629:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf64f0b4 to 0xfff98862bf64f411 May 17 18:14:27 client-5 kernel: LustreError: 629:0:(client.c:2570:ptlrpc_replay_interpret()) @@@ status -116, old was 0 req@ffff8102c54ce400 x1369104143356425/t90194313220(90194313220) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:23/10 lens 360/424 e 6 to 0 dl 1305681301 ref 2 fl Interpret:R/ffffffff/ffffffff rc -116/-1 May 17 18:14:27 client-5 kernel: Lustre: 14836:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff810315cad400->192.168.4.128@o2ib netid 50000: select flavor null May 17 18:14:27 client-5 kernel: Lustre: 14836:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 13 previous similar messages May 17 18:14:27 client-5 kernel: Lustre: Client lustre-client has started May 17 18:14:27 client-5 xinetd[3391]: START: shell pid=14916 from=192.168.4.5 May 17 18:14:27 client-5 rshd[14918]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 18:14:27 client-5 xinetd[3391]: EXIT: shell status=0 pid=14916 duration=0(sec) May 17 18:14:28 client-5 xinetd[3391]: START: shell pid=15088 from=192.168.4.5 May 17 18:14:28 client-5 rshd[15090]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-dual test 21b: commit on sharing, two clients ============================================== 18:14:28 \(1305681268\)");echo XXRETCODE:$?' May 17 18:14:28 client-5 kernel: Lustre: DEBUG MARKER: == replay-dual test 21b: commit on sharing, two clients ============================================== 18:14:28 (1305681268) May 17 18:14:28 client-5 xinetd[3391]: EXIT: shell status=0 pid=15088 duration=0(sec) May 17 18:14:28 client-5 xinetd[3391]: START: shell pid=15130 from=192.168.4.5 May 17 18:14:28 client-5 rshd[15132]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:; lsof -t /mnt/lustre2 || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre2 | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre2 is still busy, wait one second && sleep 1; umount /mnt/lustre2; fi fi");echo XXRETCODE:$?' May 17 18:14:28 client-5 kernel: LustreError: 15173:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 18:14:28 client-5 kernel: LustreError: 15173:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 18:14:28 client-5 kernel: Lustre: client ffff810315cad400 umount complete May 17 18:14:28 client-5 xinetd[3391]: EXIT: shell status=0 pid=15130 duration=0(sec) May 17 18:14:28 client-5 xinetd[3391]: START: shell pid=15182 from=192.168.4.5 May 17 18:14:28 client-5 rshd[15183]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c " running=\$(mount | grep -c /mnt/lustre' '); rc=0; if [ \$running -eq 0 ] ; then mkdir -p /mnt/lustre; mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre; rc=\$?; fi; exit \$rc");echo XXRETCODE:$?' May 17 18:14:28 client-5 xinetd[3391]: EXIT: shell status=0 pid=15182 duration=0(sec) May 17 18:14:28 client-5 xinetd[3391]: START: shell pid=15227 from=192.168.4.5 May 17 18:14:28 client-5 rshd[15228]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mount | grep -w /mnt/lustre");echo XXRETCODE:$?' May 17 18:14:28 client-5 xinetd[3391]: EXIT: shell status=0 pid=15227 duration=0(sec) May 17 18:14:28 client-5 xinetd[3391]: START: shell pid=15271 from=192.168.4.5 May 17 18:14:28 client-5 rshd[15272]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param debug=\"-1\"; lctl set_param subsystem_debug=\"0xffb7e3ff\"; lctl set_param debug_mb=32;");echo XXRETCODE:$?' May 17 18:14:29 client-5 xinetd[3391]: EXIT: shell status=0 pid=15271 duration=1(sec) May 17 18:14:31 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 18:14:52 client-5 kernel: Lustre: 629:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf64f411 to 0xfff98862bf64f832 May 17 18:17:05 client-5 kernel: LustreError: 629:0:(client.c:2570:ptlrpc_replay_interpret()) @@@ status -116, old was 0 req@ffff8102c593ac00 x1369104143356726/t94489280519(94489280519) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:23/10 lens 360/424 e 4 to 0 dl 1305681445 ref 2 fl Interpret:R/ffffffff/ffffffff rc -116/-1 May 17 18:17:06 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 18:17:14 client-5 kernel: Lustre: 628:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369104143356973 sent from MGC192.168.4.128@o2ib to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305681427] [real_sent 1305681427] [current 1305681434] [deadline 7s] [delay 0s] req@ffff8103154d3c00 x1369104143356973/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 192/192 e 0 to 1 dl 1305681434 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 18:17:14 client-5 kernel: Lustre: 628:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 11 previous similar messages May 17 18:17:14 client-5 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail. May 17 18:17:14 client-5 kernel: LustreError: Skipped 3 previous similar messages May 17 18:17:24 client-5 kernel: Lustre: 630:0:(import.c:529:import_select_connection()) MGC192.168.4.128@o2ib: tried all connections, increasing latency to 6s May 17 18:17:24 client-5 kernel: Lustre: 630:0:(import.c:529:import_select_connection()) Skipped 1 previous similar message May 17 18:17:24 client-5 kernel: Lustre: 629:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf64f832 to 0xfff98862bf64fb5e May 17 18:17:24 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 18:17:24 client-5 kernel: Lustre: Skipped 3 previous similar messages May 17 18:17:24 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The obd_ping operation failed with -107 May 17 18:17:24 client-5 kernel: LustreError: Skipped 8 previous similar messages May 17 18:19:39 client-5 kernel: Lustre: 629:0:(client.c:2527:ptlrpc_replay_interpret()) @@@ Version mismatch during replay May 17 18:19:39 client-5 kernel: req@ffff8102e894d800 x1369104143356972/t98784247814(98784247814) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 472/424 e 0 to 0 dl 1305681655 ref 2 fl Interpret:R/ffffffff/ffffffff rc -75/-1 May 17 18:20:55 client-5 kernel: Lustre: 629:0:(import.c:1190:completed_replay_interpret()) lustre-MDT0000-mdc-ffff810319af2800: version recovery fails, reconnecting May 17 18:20:55 client-5 kernel: LustreError: 167-0: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. May 17 18:20:55 client-5 kernel: LustreError: 15725:0:(mdc_locks.c:722:mdc_enqueue()) ldlm_cli_enqueue: -4 May 17 18:20:55 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff810319af2800: Connection restored to service lustre-MDT0000 using nid 192.168.4.128@o2ib. May 17 18:20:55 client-5 kernel: Lustre: Skipped 7 previous similar messages May 17 18:20:56 client-5 xinetd[3391]: START: shell pid=15818 from=192.168.4.5 May 17 18:20:56 client-5 rshd[15819]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 18:20:56 client-5 xinetd[3391]: EXIT: shell status=0 pid=15818 duration=0(sec) May 17 18:20:56 client-5 xinetd[3391]: START: shell pid=15983 from=192.168.4.5 May 17 18:20:56 client-5 rshd[15984]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-dual replay-dual.sh test complete, duration 2197 sec ======================================= 18:20:56 \(1305681656\)");echo XXRETCODE:$?' May 17 18:20:56 client-5 kernel: Lustre: DEBUG MARKER: == replay-dual replay-dual.sh test complete, duration 2197 sec ======================================= 18:20:56 (1305681656) May 17 18:20:56 client-5 xinetd[3391]: EXIT: shell status=0 pid=15983 duration=0(sec) May 17 18:20:56 client-5 xinetd[3391]: START: shell pid=16035 from=192.168.4.5 May 17 18:20:56 client-5 rshd[16036]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:-f; lsof -t /mnt/lustre || need_kill=no; if [ x-f != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount -f /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre is still busy, wait one second && sleep 1; umount -f /mnt/lustre; fi fi");echo XXRETCODE:$?' May 17 18:20:57 client-5 kernel: Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request May 17 18:20:57 client-5 kernel: Lustre: Skipped 6 previous similar messages May 17 18:20:57 client-5 kernel: LustreError: 16077:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 18:20:57 client-5 kernel: LustreError: 16077:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 18:20:57 client-5 kernel: Lustre: client ffff810319af2800 umount complete May 17 18:20:57 client-5 xinetd[3391]: EXIT: shell status=0 pid=16035 duration=1(sec) May 17 18:20:57 client-5 xinetd[3391]: START: shell pid=16086 from=192.168.4.5 May 17 18:20:57 client-5 rshd[16087]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:-f; lsof -t /mnt/lustre2 || need_kill=no; if [ x-f != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre2 | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount -f /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre2 is still busy, wait one second && sleep 1; umount -f /mnt/lustre2; fi fi");echo XXRETCODE:$?' May 17 18:20:57 client-5 xinetd[3391]: EXIT: shell status=0 pid=16086 duration=0(sec) May 17 18:21:38 client-5 kernel: Lustre: Acceptor stopping May 17 18:21:39 client-5 kernel: Lustre: Removed LNI 10.10.4.5@tcp May 17 18:21:41 client-5 kernel: Lustre: Removed LNI 192.168.4.5@o2ib May 17 18:21:43 client-5 kernel: Lustre: 16710:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 18:21:43 client-5 kernel: Lustre: 16711:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 18:21:43 client-5 kernel: Lustre: OBD class driver, http://www.lustre.org/ May 17 18:21:43 client-5 kernel: Lustre: Lustre Version: 2.0.60 May 17 18:21:43 client-5 kernel: Lustre: Build Version: jenkins-g985ac5b-PRISTINE-2.6.18-194.17.1.el5 May 17 18:21:43 client-5 kernel: Lustre: Lustre LU module (ffffffff888eb1c0). May 17 18:21:43 client-5 kernel: Lustre: Register global MR array, MR size: 0xffffffffffffffff, array size: 1 May 17 18:21:43 client-5 kernel: Lustre: Added LNI 192.168.4.5@o2ib [8/64/0/180] May 17 18:21:43 client-5 kernel: Lustre: Added LNI 10.10.4.5@tcp [8/256/0/180] May 17 18:21:43 client-5 kernel: Lustre: Accept all, port 7988 May 17 18:21:43 client-5 kernel: Lustre: Lustre OSC module (ffffffff88b3c6c0). May 17 18:21:43 client-5 kernel: Lustre: Lustre LOV module (ffffffff88bab540). May 17 18:21:43 client-5 kernel: Lustre: Lustre client module (ffffffff88c643a0). May 17 18:21:43 client-5 xinetd[3391]: START: shell pid=17116 from=192.168.4.5 May 17 18:21:43 client-5 rshd[17117]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark -----============= acceptance-small: replay-vbr ============----- Tue May 17 18:21:43 PDT 2011");echo XXRETCODE:$?' May 17 18:21:43 client-5 kernel: Lustre: DEBUG MARKER: -----============= acceptance-small: replay-vbr ============----- Tue May 17 18:21:43 PDT 2011 May 17 18:21:43 client-5 xinetd[3391]: EXIT: shell status=0 pid=17116 duration=0(sec) May 17 18:21:44 client-5 xinetd[3391]: START: shell pid=17262 from=192.168.4.5 May 17 18:21:44 client-5 rshd[17264]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark excepting tests: 12a");echo XXRETCODE:$?' May 17 18:21:44 client-5 kernel: Lustre: DEBUG MARKER: excepting tests: 12a May 17 18:21:44 client-5 xinetd[3391]: EXIT: shell status=0 pid=17262 duration=0(sec) May 17 18:21:44 client-5 xinetd[3391]: START: shell pid=17316 from=192.168.4.5 May 17 18:21:44 client-5 rshd[17317]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:; lsof -t /mnt/lustre || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre is still busy, wait one second && sleep 1; umount /mnt/lustre; fi fi");echo XXRETCODE:$?' May 17 18:21:44 client-5 xinetd[3391]: EXIT: shell status=0 pid=17316 duration=0(sec) May 17 18:21:44 client-5 xinetd[3391]: START: shell pid=17360 from=192.168.4.5 May 17 18:21:44 client-5 rshd[17361]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:; lsof -t /mnt/lustre2 || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre2 | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre2 is still busy, wait one second && sleep 1; umount /mnt/lustre2; fi fi");echo XXRETCODE:$?' May 17 18:21:44 client-5 xinetd[3391]: EXIT: shell status=0 pid=17360 duration=0(sec) May 17 18:21:45 client-5 kernel: Lustre: 17563:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 18:22:44 client-5 xinetd[3391]: START: shell pid=17999 from=192.168.4.5 May 17 18:22:44 client-5 rshd[18000]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); mpts=\$(mount | grep -w -c /mnt/lustre); if [ \$running -ne \$mpts ]; then echo \$(hostname) env are INSANE!; exit 1; fi");echo XXRETCODE:$?' May 17 18:22:44 client-5 xinetd[3391]: EXIT: shell status=0 pid=17999 duration=0(sec) May 17 18:22:44 client-5 xinetd[3391]: START: shell pid=18046 from=192.168.4.5 May 17 18:22:44 client-5 rshd[18047]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); mpts=\$(mount | grep -w -c /mnt/lustre2); if [ \$running -ne \$mpts ]; then echo \$(hostname) env are INSANE!; exit 1; fi");echo XXRETCODE:$?' May 17 18:22:44 client-5 xinetd[3391]: EXIT: shell status=0 pid=18046 duration=0(sec) May 17 18:22:44 client-5 kernel: Lustre: 18089:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 18:22:44 client-5 kernel: Lustre: 18089:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message May 17 18:22:48 client-5 kernel: Lustre: 18426:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 18:22:48 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 18:22:48 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 18:22:48 client-5 kernel: Lustre: 18426:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff8102feea9800->192.168.4.128@o2ib netid 50000: select flavor null May 17 18:22:48 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 18:22:53 client-5 kernel: Lustre: 18426:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-OST0000-osc-ffff8102feea9800->192.168.4.129@o2ib netid 50000: select flavor null May 17 18:22:53 client-5 kernel: Lustre: Client lustre-client has started May 17 18:22:53 client-5 kernel: Lustre: 18443:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 18:22:53 client-5 kernel: Lustre: 18443:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message May 17 18:22:53 client-5 xinetd[3391]: START: shell pid=18453 from=192.168.4.5 May 17 18:22:53 client-5 rshd[18454]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c " running=\$(mount | grep -c /mnt/lustre' '); rc=0; if [ \$running -eq 0 ] ; then mkdir -p /mnt/lustre; mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre; rc=\$?; fi; exit \$rc");echo XXRETCODE:$?' May 17 18:22:53 client-5 xinetd[3391]: EXIT: shell status=0 pid=18453 duration=0(sec) May 17 18:22:53 client-5 xinetd[3391]: START: shell pid=18498 from=192.168.4.5 May 17 18:22:53 client-5 rshd[18499]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mount | grep -w /mnt/lustre");echo XXRETCODE:$?' May 17 18:22:53 client-5 xinetd[3391]: EXIT: shell status=0 pid=18498 duration=0(sec) May 17 18:22:53 client-5 xinetd[3391]: START: shell pid=18542 from=192.168.4.5 May 17 18:22:53 client-5 rshd[18543]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param debug=\"0x33f0406\"; lctl set_param subsystem_debug=\"0xffb7e3ff\"; lctl set_param debug_mb=32;");echo XXRETCODE:$?' May 17 18:22:54 client-5 xinetd[3391]: EXIT: shell status=0 pid=18542 duration=1(sec) May 17 18:22:54 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 18:22:54 client-5 kernel: Lustre: Client lustre-client has started May 17 18:22:54 client-5 xinetd[3391]: START: shell pid=18605 from=192.168.4.5 May 17 18:22:54 client-5 rshd[18606]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c " running=\$(mount | grep -c /mnt/lustre2' '); rc=0; if [ \$running -eq 0 ] ; then mkdir -p /mnt/lustre2; mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre2; rc=\$?; fi; exit \$rc");echo XXRETCODE:$?' May 17 18:22:54 client-5 xinetd[3391]: EXIT: shell status=0 pid=18605 duration=0(sec) May 17 18:22:54 client-5 xinetd[3391]: START: shell pid=18650 from=192.168.4.5 May 17 18:22:54 client-5 rshd[18651]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mount | grep -w /mnt/lustre2");echo XXRETCODE:$?' May 17 18:22:54 client-5 xinetd[3391]: EXIT: shell status=0 pid=18650 duration=0(sec) May 17 18:22:54 client-5 xinetd[3391]: START: shell pid=18694 from=192.168.4.5 May 17 18:22:54 client-5 rshd[18695]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param debug=\"0x33f0406\"; lctl set_param subsystem_debug=\"0xffb7e3ff\"; lctl set_param debug_mb=32;");echo XXRETCODE:$?' May 17 18:22:54 client-5 xinetd[3391]: EXIT: shell status=0 pid=18694 duration=0(sec) May 17 18:22:54 client-5 xinetd[3391]: START: shell pid=18864 from=192.168.4.5 May 17 18:22:54 client-5 rshd[18865]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark Using TIMEOUT=20");echo XXRETCODE:$?' May 17 18:22:54 client-5 kernel: Lustre: DEBUG MARKER: Using TIMEOUT=20 May 17 18:22:54 client-5 xinetd[3391]: EXIT: shell status=0 pid=18864 duration=0(sec) May 17 18:22:55 client-5 kernel: LustreError: 18918:0:(quota_ctl.c:328:client_quota_ctl()) ptlrpc_queue_wait failed, rc: -114 May 17 18:23:00 client-5 xinetd[3391]: START: shell pid=18938 from=192.168.4.5 May 17 18:23:00 client-5 rshd[18939]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "cat /proc/mounts");echo XXRETCODE:$?' May 17 18:23:00 client-5 xinetd[3391]: EXIT: shell status=0 pid=18938 duration=0(sec) May 17 18:23:01 client-5 xinetd[3391]: START: shell pid=19231 from=192.168.4.5 May 17 18:23:01 client-5 rshd[19232]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-vbr test 1a: open and close do not change versions ========================================= 18:23:01 \(1305681781\)");echo XXRETCODE:$?' May 17 18:23:01 client-5 kernel: Lustre: DEBUG MARKER: == replay-vbr test 1a: open and close do not change versions ========================================= 18:23:01 (1305681781) May 17 18:23:01 client-5 xinetd[3391]: EXIT: shell status=0 pid=19231 duration=0(sec) May 17 18:23:01 client-5 xinetd[3391]: START: shell pid=19358 from=192.168.4.5 May 17 18:23:01 client-5 rshd[19359]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 18:23:01 client-5 xinetd[3391]: EXIT: shell status=0 pid=19358 duration=0(sec) May 17 18:23:02 client-5 xinetd[3391]: START: shell pid=19530 from=192.168.4.5 May 17 18:23:02 client-5 rshd[19531]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-vbr test 1b: open \(O_CREAT\) checks version of parent ======================================= 18:23:02 \(1305681782\)");echo XXRETCODE:$?' May 17 18:23:02 client-5 kernel: Lustre: DEBUG MARKER: == replay-vbr test 1b: open (O_CREAT) checks version of parent ======================================= 18:23:02 (1305681782) May 17 18:23:02 client-5 xinetd[3391]: EXIT: shell status=0 pid=19530 duration=0(sec) May 17 18:23:02 client-5 xinetd[3391]: START: shell pid=19611 from=192.168.4.5 May 17 18:23:02 client-5 rshd[19612]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 18:23:02 client-5 xinetd[3391]: EXIT: shell status=0 pid=19611 duration=0(sec) May 17 18:23:03 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 18:23:16 client-5 kernel: Lustre: 16819:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369106497405022 sent from MGC192.168.4.128@o2ib to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305681789] [real_sent 1305681789] [current 1305681796] [deadline 7s] [delay 0s] req@ffff81031b332800 x1369106497405022/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 192/192 e 0 to 1 dl 1305681796 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 18:23:16 client-5 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail. May 17 18:23:16 client-5 kernel: Lustre: 16819:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369106497405023 sent from lustre-MDT0000-mdc-ffff8102feea9800 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305681789] [real_sent 1305681789] [current 1305681796] [deadline 7s] [delay 0s] req@ffff810321baec00 x1369106497405023/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 192/192 e 0 to 1 dl 1305681796 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 18:23:16 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff8102feea9800: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 18:23:16 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff810329823000: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 18:23:21 client-5 kernel: Lustre: 16819:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369106497405038 sent from lustre-MDT0000-mdc-ffff8102feea9800 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305681794] [real_sent 1305681794] [current 1305681801] [deadline 7s] [delay 0s] req@ffff810263d21000 x1369106497405038/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 192/192 e 0 to 1 dl 1305681801 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 18:23:21 client-5 kernel: Lustre: 16819:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 1 previous similar message May 17 18:23:22 client-5 xinetd[3391]: START: shell pid=19820 from=192.168.4.5 May 17 18:23:22 client-5 rshd[19821]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 18:23:26 client-5 kernel: Lustre: 16821:0:(import.c:529:import_select_connection()) MGC192.168.4.128@o2ib: tried all connections, increasing latency to 6s May 17 18:23:26 client-5 kernel: Lustre: 16821:0:(import.c:529:import_select_connection()) lustre-MDT0000-mdc-ffff8102feea9800: tried all connections, increasing latency to 6s May 17 18:23:26 client-5 kernel: Lustre: 16820:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf64fede to 0xfff98862bf6503ae May 17 18:23:26 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 18:23:26 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Connection restored to service MGS using nid 192.168.4.128@o2ib. May 17 18:24:32 client-5 kernel: Lustre: 16820:0:(client.c:2527:ptlrpc_replay_interpret()) @@@ Version mismatch during replay May 17 18:24:32 client-5 kernel: req@ffff81031d77fc00 x1369106497405012/t4294967312(4294967312) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 528/544 e 0 to 0 dl 1305681903 ref 2 fl Interpret:R/ffffffff/ffffffff rc -75/-1 May 17 18:24:32 client-5 kernel: LustreError: 16820:0:(client.c:2570:ptlrpc_replay_interpret()) @@@ status -116, old was 0 req@ffff8102f2fd7c00 x1369106497405013/t4294967313(4294967313) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:23/10 lens 360/424 e 0 to 0 dl 1305681884 ref 2 fl Interpret:R/ffffffff/ffffffff rc -116/-1 May 17 18:24:32 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff810329823000: Connection restored to service lustre-MDT0000 using nid 192.168.4.128@o2ib. May 17 18:25:03 client-5 kernel: Lustre: 16820:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369106497405249 sent from lustre-MDT0000-mdc-ffff8102feea9800 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305681872] [real_sent 1305681872] [current 1305681903] [deadline 31s] [delay 0s] req@ffff8102e82fa400 x1369106497405249/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 192/192 e 0 to 1 dl 1305681903 ref 1 fl Rpc:X/ffffffff/ffffffff rc 0/-1 May 17 18:25:03 client-5 kernel: Lustre: 16820:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 4 previous similar messages May 17 18:25:03 client-5 kernel: Lustre: 16820:0:(import.c:1190:completed_replay_interpret()) lustre-MDT0000-mdc-ffff8102feea9800: version recovery fails, reconnecting May 17 18:25:03 client-5 kernel: LustreError: 167-0: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. May 17 18:25:03 client-5 kernel: LustreError: 19853:0:(lmv_obd.c:1198:lmv_statfs()) can't stat MDS #0 (lustre-MDT0000-mdc-ffff8102feea9800), error -4 May 17 18:25:03 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff8102feea9800: Connection restored to service lustre-MDT0000 using nid 192.168.4.128@o2ib. May 17 18:25:03 client-5 kernel: LustreError: 19853:0:(llite_lib.c:1370:ll_statfs_internal()) md_statfs fails: rc = -4 May 17 18:25:03 client-5 xinetd[3391]: EXIT: shell status=0 pid=19820 duration=101(sec) May 17 18:25:03 client-5 xinetd[3391]: START: shell pid=19927 from=192.168.4.5 May 17 18:25:03 client-5 rshd[19929]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 18:25:03 client-5 xinetd[3391]: EXIT: shell status=0 pid=19927 duration=0(sec) May 17 18:25:03 client-5 xinetd[3391]: START: shell pid=20099 from=192.168.4.5 May 17 18:25:03 client-5 rshd[20101]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-vbr test 1c: open \(non O_CREAT\) does not checks versions =================================== 18:25:03 \(1305681903\)");echo XXRETCODE:$?' May 17 18:25:03 client-5 kernel: Lustre: DEBUG MARKER: == replay-vbr test 1c: open (non O_CREAT) does not checks versions =================================== 18:25:03 (1305681903) May 17 18:25:03 client-5 xinetd[3391]: EXIT: shell status=0 pid=20099 duration=0(sec) May 17 18:25:04 client-5 xinetd[3391]: START: shell pid=20189 from=192.168.4.5 May 17 18:25:04 client-5 rshd[20191]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 18:25:04 client-5 xinetd[3391]: EXIT: shell status=0 pid=20189 duration=0(sec) May 17 18:25:05 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 18:25:23 client-5 kernel: Lustre: 16819:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369106497405364 sent from MGC192.168.4.128@o2ib to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305681911] [real_sent 1305681911] [current 1305681923] [deadline 12s] [delay 0s] req@ffff81021f7fd400 x1369106497405364/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 192/192 e 0 to 1 dl 1305681923 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 18:25:23 client-5 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail. May 17 18:25:27 client-5 xinetd[3391]: START: shell pid=20430 from=192.168.4.5 May 17 18:25:27 client-5 rshd[20431]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 18:25:27 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_statfs operation failed with -107 May 17 18:25:27 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff8102feea9800: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 18:25:28 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The obd_ping operation failed with -107 May 17 18:25:34 client-5 kernel: Lustre: 16820:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369106497405409 sent from MGC192.168.4.128@o2ib to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305681923] [real_sent 1305681923] [current 1305681934] [deadline 11s] [delay 0s] req@ffff81031ecba800 x1369106497405409/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 368/392 e 0 to 1 dl 1305681934 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 18:25:38 client-5 kernel: Lustre: 16821:0:(import.c:529:import_select_connection()) MGC192.168.4.128@o2ib: tried all connections, increasing latency to 11s May 17 18:25:38 client-5 kernel: Lustre: 16821:0:(import.c:529:import_select_connection()) Skipped 1 previous similar message May 17 18:25:38 client-5 kernel: Lustre: 16820:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf6503ae to 0xfff98862bf6507ac May 17 18:25:38 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 18:25:38 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Connection restored to service MGS using nid 192.168.4.128@o2ib. May 17 18:25:42 client-5 kernel: Lustre: 16819:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369106497405365 sent from lustre-MDT0000-mdc-ffff8102feea9800 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305681911] [real_sent 1305681911] [current 1305681942] [deadline 31s] [delay 0s] req@ffff810322fca000 x1369106497405365/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 192/192 e 0 to 1 dl 1305681942 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 18:26:52 client-5 kernel: LustreError: 16820:0:(client.c:2570:ptlrpc_replay_interpret()) @@@ status -116, old was 0 req@ffff8102f2fd7400 x1369106497405341/t8589934595(8589934595) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:23/10 lens 360/424 e 6 to 0 dl 1305682045 ref 2 fl Interpret:R/ffffffff/ffffffff rc -116/-1 May 17 18:26:52 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff810329823000: Connection restored to service lustre-MDT0000 using nid 192.168.4.128@o2ib. May 17 18:26:52 client-5 xinetd[3391]: EXIT: shell status=0 pid=20430 duration=85(sec) May 17 18:26:52 client-5 xinetd[3391]: START: shell pid=20540 from=192.168.4.5 May 17 18:26:52 client-5 rshd[20542]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 18:26:52 client-5 xinetd[3391]: EXIT: shell status=0 pid=20540 duration=0(sec) May 17 18:26:52 client-5 xinetd[3391]: START: shell pid=20713 from=192.168.4.5 May 17 18:26:52 client-5 rshd[20714]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-vbr test 2a: create operations doesn\'t change version of parent ============================ 18:26:52 \(1305682012\)");echo XXRETCODE:$?' May 17 18:26:52 client-5 kernel: Lustre: DEBUG MARKER: == replay-vbr test 2a: create operations doesn't change version of parent ============================ 18:26:52 (1305682012) May 17 18:26:52 client-5 xinetd[3391]: EXIT: shell status=0 pid=20713 duration=0(sec) May 17 18:26:53 client-5 xinetd[3391]: START: shell pid=20940 from=192.168.4.5 May 17 18:26:53 client-5 rshd[20941]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 18:26:54 client-5 xinetd[3391]: EXIT: shell status=0 pid=20940 duration=1(sec) May 17 18:26:54 client-5 xinetd[3391]: START: shell pid=21112 from=192.168.4.5 May 17 18:26:54 client-5 rshd[21113]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-vbr test 2b: create checks version of parent =============================================== 18:26:54 \(1305682014\)");echo XXRETCODE:$?' May 17 18:26:54 client-5 kernel: Lustre: DEBUG MARKER: == replay-vbr test 2b: create checks version of parent =============================================== 18:26:54 (1305682014) May 17 18:26:54 client-5 xinetd[3391]: EXIT: shell status=0 pid=21112 duration=0(sec) May 17 18:26:55 client-5 xinetd[3391]: START: shell pid=21198 from=192.168.4.5 May 17 18:26:55 client-5 rshd[21199]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 18:26:55 client-5 xinetd[3391]: EXIT: shell status=0 pid=21198 duration=0(sec) May 17 18:26:55 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 18:27:13 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The obd_ping operation failed with -107 May 17 18:27:13 client-5 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail. May 17 18:27:13 client-5 kernel: Lustre: 16820:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf6507ac to 0xfff98862bf650bb8 May 17 18:27:13 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 18:27:13 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Connection restored to service MGS using nid 192.168.4.128@o2ib. May 17 18:27:13 client-5 kernel: Lustre: Skipped 1 previous similar message May 17 18:27:14 client-5 xinetd[3391]: START: shell pid=21409 from=192.168.4.5 May 17 18:27:14 client-5 rshd[21410]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 18:27:14 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff8102feea9800: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 18:27:14 client-5 kernel: Lustre: Skipped 1 previous similar message May 17 18:27:18 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The obd_ping operation failed with -107 May 17 18:27:18 client-5 kernel: LustreError: Skipped 1 previous similar message May 17 18:27:18 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff810329823000: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 18:27:35 client-5 kernel: Lustre: 16819:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369106497405681 sent from lustre-MDT0000-mdc-ffff810329823000 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305682018] [real_sent 1305682018] [current 1305682055] [deadline 37s] [delay 0s] req@ffff8103238f8800 x1369106497405681/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 192/192 e 0 to 1 dl 1305682055 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 18:27:35 client-5 kernel: Lustre: 16819:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 5 previous similar messages May 17 18:28:44 client-5 kernel: Lustre: 16820:0:(client.c:2527:ptlrpc_replay_interpret()) @@@ Version mismatch during replay May 17 18:28:44 client-5 kernel: req@ffff8103253a7c00 x1369106497405679/t12884901904(12884901904) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 456/416 e 0 to 0 dl 1305682162 ref 2 fl Interpret:R/ffffffff/ffffffff rc -75/-1 May 17 18:28:44 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff810329823000: Connection restored to service lustre-MDT0000 using nid 192.168.4.128@o2ib. May 17 18:29:22 client-5 kernel: Lustre: 16820:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369106497405981 sent from lustre-MDT0000-mdc-ffff8102feea9800 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305682124] [real_sent 1305682124] [current 1305682162] [deadline 38s] [delay 0s] req@ffff81032faa2800 x1369106497405981/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 192/192 e 0 to 1 dl 1305682162 ref 1 fl Rpc:X/ffffffff/ffffffff rc 0/-1 May 17 18:29:22 client-5 kernel: Lustre: 16820:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 6 previous similar messages May 17 18:29:22 client-5 kernel: Lustre: 16820:0:(import.c:1190:completed_replay_interpret()) lustre-MDT0000-mdc-ffff8102feea9800: version recovery fails, reconnecting May 17 18:29:22 client-5 kernel: LustreError: 167-0: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. May 17 18:29:22 client-5 kernel: LustreError: 21442:0:(lmv_obd.c:1198:lmv_statfs()) can't stat MDS #0 (lustre-MDT0000-mdc-ffff8102feea9800), error -4 May 17 18:29:22 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff8102feea9800: Connection restored to service lustre-MDT0000 using nid 192.168.4.128@o2ib. May 17 18:29:22 client-5 kernel: LustreError: 21442:0:(llite_lib.c:1370:ll_statfs_internal()) md_statfs fails: rc = -4 May 17 18:29:22 client-5 xinetd[3391]: EXIT: shell status=0 pid=21409 duration=128(sec) May 17 18:29:22 client-5 xinetd[3391]: START: shell pid=21515 from=192.168.4.5 May 17 18:29:22 client-5 rshd[21516]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 18:29:23 client-5 xinetd[3391]: EXIT: shell status=0 pid=21515 duration=1(sec) May 17 18:29:23 client-5 xinetd[3391]: START: shell pid=21687 from=192.168.4.5 May 17 18:29:23 client-5 rshd[21688]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-vbr test 3a: unlink doesn\'t change version of parent ======================================= 18:29:23 \(1305682163\)");echo XXRETCODE:$?' May 17 18:29:23 client-5 kernel: Lustre: DEBUG MARKER: == replay-vbr test 3a: unlink doesn't change version of parent ======================================= 18:29:23 (1305682163) May 17 18:29:23 client-5 xinetd[3391]: EXIT: shell status=0 pid=21687 duration=0(sec) May 17 18:29:23 client-5 xinetd[3391]: START: shell pid=21814 from=192.168.4.5 May 17 18:29:23 client-5 rshd[21815]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 18:29:23 client-5 xinetd[3391]: EXIT: shell status=0 pid=21814 duration=0(sec) May 17 18:29:23 client-5 xinetd[3391]: START: shell pid=21986 from=192.168.4.5 May 17 18:29:23 client-5 rshd[21987]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-vbr test 3b: unlink checks version of parent =============================================== 18:29:23 \(1305682163\)");echo XXRETCODE:$?' May 17 18:29:24 client-5 kernel: Lustre: DEBUG MARKER: == replay-vbr test 3b: unlink checks version of parent =============================================== 18:29:23 (1305682163) May 17 18:29:24 client-5 xinetd[3391]: EXIT: shell status=0 pid=21986 duration=1(sec) May 17 18:29:24 client-5 xinetd[3391]: START: shell pid=22076 from=192.168.4.5 May 17 18:29:24 client-5 rshd[22077]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 18:29:24 client-5 xinetd[3391]: EXIT: shell status=0 pid=22076 duration=0(sec) May 17 18:29:25 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 18:29:43 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The obd_ping operation failed with -107 May 17 18:29:43 client-5 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail. May 17 18:29:43 client-5 kernel: Lustre: 16820:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf650bb8 to 0xfff98862bf651049 May 17 18:29:43 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 18:29:44 client-5 xinetd[3391]: START: shell pid=22287 from=192.168.4.5 May 17 18:29:44 client-5 rshd[22288]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 18:29:44 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff8102feea9800: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 18:31:15 client-5 kernel: Lustre: 16820:0:(client.c:2527:ptlrpc_replay_interpret()) @@@ Version mismatch during replay May 17 18:31:15 client-5 kernel: req@ffff8102f388d400 x1369106497406123/t17179869195(17179869195) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 456/424 e 0 to 0 dl 1305682313 ref 2 fl Interpret:R/ffffffff/ffffffff rc -75/-1 May 17 18:31:15 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff810329823000: Connection restored to service lustre-MDT0000 using nid 192.168.4.128@o2ib. May 17 18:31:15 client-5 kernel: Lustre: Skipped 1 previous similar message May 17 18:31:53 client-5 kernel: Lustre: 16820:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369106497406425 sent from lustre-MDT0000-mdc-ffff8102feea9800 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305682275] [real_sent 1305682275] [current 1305682313] [deadline 38s] [delay 0s] req@ffff8103238f8c00 x1369106497406425/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 192/192 e 0 to 1 dl 1305682313 ref 1 fl Rpc:X/ffffffff/ffffffff rc 0/-1 May 17 18:31:53 client-5 kernel: Lustre: 16820:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 7 previous similar messages May 17 18:31:53 client-5 kernel: Lustre: 16820:0:(import.c:1190:completed_replay_interpret()) lustre-MDT0000-mdc-ffff8102feea9800: version recovery fails, reconnecting May 17 18:31:53 client-5 kernel: LustreError: 167-0: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. May 17 18:31:53 client-5 kernel: LustreError: 22320:0:(lmv_obd.c:1198:lmv_statfs()) can't stat MDS #0 (lustre-MDT0000-mdc-ffff8102feea9800), error -4 May 17 18:31:53 client-5 kernel: LustreError: 22320:0:(llite_lib.c:1370:ll_statfs_internal()) md_statfs fails: rc = -4 May 17 18:31:53 client-5 xinetd[3391]: EXIT: shell status=0 pid=22287 duration=129(sec) May 17 18:31:53 client-5 xinetd[3391]: START: shell pid=22393 from=192.168.4.5 May 17 18:31:53 client-5 rshd[22394]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 18:31:53 client-5 xinetd[3391]: EXIT: shell status=0 pid=22393 duration=0(sec) May 17 18:31:53 client-5 xinetd[3391]: START: shell pid=22565 from=192.168.4.5 May 17 18:31:53 client-5 rshd[22566]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-vbr test 4a: setattr of UID changes versions =============================================== 18:31:53 \(1305682313\)");echo XXRETCODE:$?' May 17 18:31:53 client-5 kernel: Lustre: DEBUG MARKER: == replay-vbr test 4a: setattr of UID changes versions =============================================== 18:31:53 (1305682313) May 17 18:31:53 client-5 xinetd[3391]: EXIT: shell status=0 pid=22565 duration=0(sec) May 17 18:31:54 client-5 xinetd[3391]: START: shell pid=22692 from=192.168.4.5 May 17 18:31:54 client-5 rshd[22693]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 18:31:54 client-5 xinetd[3391]: EXIT: shell status=0 pid=22692 duration=0(sec) May 17 18:31:54 client-5 xinetd[3391]: START: shell pid=22864 from=192.168.4.5 May 17 18:31:54 client-5 rshd[22865]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-vbr test 4b: setattr of GID changes versions =============================================== 18:31:54 \(1305682314\)");echo XXRETCODE:$?' May 17 18:31:54 client-5 kernel: Lustre: DEBUG MARKER: == replay-vbr test 4b: setattr of GID changes versions =============================================== 18:31:54 (1305682314) May 17 18:31:54 client-5 xinetd[3391]: EXIT: shell status=0 pid=22864 duration=0(sec) May 17 18:31:54 client-5 xinetd[3391]: START: shell pid=22991 from=192.168.4.5 May 17 18:31:54 client-5 rshd[22992]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 18:31:55 client-5 xinetd[3391]: EXIT: shell status=0 pid=22991 duration=1(sec) May 17 18:31:55 client-5 xinetd[3391]: START: shell pid=23163 from=192.168.4.5 May 17 18:31:55 client-5 rshd[23164]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-vbr test 4c: setattr of UID checks versions ================================================ 18:31:55 \(1305682315\)");echo XXRETCODE:$?' May 17 18:31:55 client-5 kernel: Lustre: DEBUG MARKER: == replay-vbr test 4c: setattr of UID checks versions ================================================ 18:31:55 (1305682315) May 17 18:31:55 client-5 xinetd[3391]: EXIT: shell status=0 pid=23163 duration=0(sec) May 17 18:31:56 client-5 xinetd[3391]: START: shell pid=23249 from=192.168.4.5 May 17 18:31:56 client-5 rshd[23250]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 18:31:56 client-5 xinetd[3391]: EXIT: shell status=0 pid=23249 duration=0(sec) May 17 18:31:56 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 18:32:15 client-5 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail. May 17 18:32:15 client-5 kernel: Lustre: 16820:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf651049 to 0xfff98862bf651551 May 17 18:32:15 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 18:32:15 client-5 xinetd[3391]: START: shell pid=23460 from=192.168.4.5 May 17 18:32:15 client-5 rshd[23461]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 18:32:15 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_statfs operation failed with -107 May 17 18:32:15 client-5 kernel: LustreError: Skipped 2 previous similar messages May 17 18:32:15 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff8102feea9800: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 18:32:15 client-5 kernel: Lustre: Skipped 1 previous similar message May 17 18:33:46 client-5 kernel: Lustre: 16820:0:(client.c:2527:ptlrpc_replay_interpret()) @@@ Version mismatch during replay May 17 18:33:46 client-5 kernel: req@ffff81033f776000 x1369106497406573/t21474836500(21474836500) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 456/424 e 0 to 0 dl 1305682459 ref 2 fl Interpret:R/ffffffff/ffffffff rc -75/-1 May 17 18:33:46 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff810329823000: Connection restored to service lustre-MDT0000 using nid 192.168.4.128@o2ib. May 17 18:33:46 client-5 kernel: Lustre: Skipped 2 previous similar messages May 17 18:34:19 client-5 kernel: Lustre: 16820:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369106497406875 sent from lustre-MDT0000-mdc-ffff8102feea9800 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305682426] [real_sent 1305682426] [current 1305682459] [deadline 33s] [delay 0s] req@ffff8102e82fa400 x1369106497406875/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 192/192 e 0 to 1 dl 1305682459 ref 1 fl Rpc:X/ffffffff/ffffffff rc 0/-1 May 17 18:34:19 client-5 kernel: Lustre: 16820:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 8 previous similar messages May 17 18:34:19 client-5 kernel: Lustre: 16820:0:(import.c:1190:completed_replay_interpret()) lustre-MDT0000-mdc-ffff8102feea9800: version recovery fails, reconnecting May 17 18:34:19 client-5 kernel: LustreError: 167-0: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. May 17 18:34:19 client-5 kernel: LustreError: 23493:0:(lmv_obd.c:1198:lmv_statfs()) can't stat MDS #0 (lustre-MDT0000-mdc-ffff8102feea9800), error -4 May 17 18:34:19 client-5 kernel: LustreError: 23493:0:(llite_lib.c:1370:ll_statfs_internal()) md_statfs fails: rc = -4 May 17 18:34:19 client-5 xinetd[3391]: EXIT: shell status=0 pid=23460 duration=124(sec) May 17 18:34:19 client-5 xinetd[3391]: START: shell pid=23564 from=192.168.4.5 May 17 18:34:19 client-5 rshd[23567]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 18:34:19 client-5 xinetd[3391]: EXIT: shell status=0 pid=23564 duration=0(sec) May 17 18:34:20 client-5 xinetd[3391]: START: shell pid=23737 from=192.168.4.5 May 17 18:34:20 client-5 rshd[23739]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-vbr test 4d: setattr of GID checks versions ================================================ 18:34:20 \(1305682460\)");echo XXRETCODE:$?' May 17 18:34:20 client-5 kernel: Lustre: DEBUG MARKER: == replay-vbr test 4d: setattr of GID checks versions ================================================ 18:34:20 (1305682460) May 17 18:34:20 client-5 xinetd[3391]: EXIT: shell status=0 pid=23737 duration=0(sec) May 17 18:34:21 client-5 xinetd[3391]: START: shell pid=23824 from=192.168.4.5 May 17 18:34:21 client-5 rshd[23825]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 18:34:21 client-5 xinetd[3391]: EXIT: shell status=0 pid=23824 duration=0(sec) May 17 18:34:21 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 18:34:40 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The obd_ping operation failed with -107 May 17 18:34:40 client-5 kernel: LustreError: Skipped 1 previous similar message May 17 18:34:40 client-5 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail. May 17 18:34:40 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff8102feea9800: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 18:34:40 client-5 kernel: Lustre: Skipped 1 previous similar message May 17 18:34:40 client-5 kernel: Lustre: 16820:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf651551 to 0xfff98862bf6518a0 May 17 18:34:40 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 18:34:40 client-5 xinetd[3391]: START: shell pid=24035 from=192.168.4.5 May 17 18:34:40 client-5 rshd[24036]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 18:36:11 client-5 kernel: Lustre: 16820:0:(client.c:2527:ptlrpc_replay_interpret()) @@@ Version mismatch during replay May 17 18:36:11 client-5 kernel: req@ffff810308d0d000 x1369106497406993/t25769803784(25769803784) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 456/424 e 0 to 0 dl 1305682604 ref 2 fl Interpret:R/ffffffff/ffffffff rc -75/-1 May 17 18:36:11 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff810329823000: Connection restored to service lustre-MDT0000 using nid 192.168.4.128@o2ib. May 17 18:36:11 client-5 kernel: Lustre: Skipped 2 previous similar messages May 17 18:36:44 client-5 kernel: Lustre: 16820:0:(import.c:1190:completed_replay_interpret()) lustre-MDT0000-mdc-ffff8102feea9800: version recovery fails, reconnecting May 17 18:36:44 client-5 kernel: LustreError: 167-0: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. May 17 18:36:44 client-5 kernel: LustreError: 24068:0:(lmv_obd.c:1198:lmv_statfs()) can't stat MDS #0 (lustre-MDT0000-mdc-ffff8102feea9800), error -4 May 17 18:36:44 client-5 kernel: LustreError: 24068:0:(llite_lib.c:1370:ll_statfs_internal()) md_statfs fails: rc = -4 May 17 18:36:44 client-5 xinetd[3391]: EXIT: shell status=0 pid=24035 duration=124(sec) May 17 18:36:44 client-5 xinetd[3391]: START: shell pid=24141 from=192.168.4.5 May 17 18:36:44 client-5 rshd[24142]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 18:36:44 client-5 xinetd[3391]: EXIT: shell status=0 pid=24141 duration=0(sec) May 17 18:36:44 client-5 xinetd[3391]: START: shell pid=24313 from=192.168.4.5 May 17 18:36:44 client-5 rshd[24314]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-vbr test 4e: setattr of permission changes versions ======================================== 18:36:44 \(1305682604\)");echo XXRETCODE:$?' May 17 18:36:44 client-5 kernel: Lustre: DEBUG MARKER: == replay-vbr test 4e: setattr of permission changes versions ======================================== 18:36:44 (1305682604) May 17 18:36:44 client-5 xinetd[3391]: EXIT: shell status=0 pid=24313 duration=0(sec) May 17 18:36:45 client-5 xinetd[3391]: START: shell pid=24440 from=192.168.4.5 May 17 18:36:45 client-5 rshd[24441]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 18:36:45 client-5 xinetd[3391]: EXIT: shell status=0 pid=24440 duration=0(sec) May 17 18:36:45 client-5 xinetd[3391]: START: shell pid=24612 from=192.168.4.5 May 17 18:36:45 client-5 rshd[24613]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-vbr test 4f: setattr of permission checks versions ========================================= 18:36:45 \(1305682605\)");echo XXRETCODE:$?' May 17 18:36:45 client-5 kernel: Lustre: DEBUG MARKER: == replay-vbr test 4f: setattr of permission checks versions ========================================= 18:36:45 (1305682605) May 17 18:36:45 client-5 xinetd[3391]: EXIT: shell status=0 pid=24612 duration=0(sec) May 17 18:36:46 client-5 xinetd[3391]: START: shell pid=24698 from=192.168.4.5 May 17 18:36:46 client-5 rshd[24699]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 18:36:46 client-5 xinetd[3391]: EXIT: shell status=0 pid=24698 duration=0(sec) May 17 18:36:46 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 18:36:57 client-5 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail. May 17 18:37:06 client-5 xinetd[3391]: START: shell pid=24907 from=192.168.4.5 May 17 18:37:06 client-5 rshd[24908]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 18:37:06 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_statfs operation failed with -107 May 17 18:37:06 client-5 kernel: LustreError: Skipped 2 previous similar messages May 17 18:37:06 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff8102feea9800: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 18:37:06 client-5 kernel: Lustre: Skipped 1 previous similar message May 17 18:37:07 client-5 kernel: Lustre: 16821:0:(import.c:529:import_select_connection()) MGC192.168.4.128@o2ib: tried all connections, increasing latency to 6s May 17 18:37:07 client-5 kernel: Lustre: 16820:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf6518a0 to 0xfff98862bf651c89 May 17 18:37:07 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 18:38:37 client-5 kernel: LustreError: 16820:0:(client.c:2570:ptlrpc_replay_interpret()) @@@ status -116, old was 0 req@ffff8103238f8c00 x1369106497407407/t30064771082(30064771082) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:23/10 lens 360/424 e 6 to 0 dl 1305682745 ref 2 fl Interpret:R/ffffffff/ffffffff rc -116/-1 May 17 18:38:37 client-5 kernel: Lustre: 16820:0:(client.c:2527:ptlrpc_replay_interpret()) @@@ Version mismatch during replay May 17 18:38:37 client-5 kernel: req@ffff8102f71d1400 x1369106497407421/t30064771088(30064771088) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 456/424 e 0 to 0 dl 1305682750 ref 2 fl Interpret:R/ffffffff/ffffffff rc -75/-1 May 17 18:39:10 client-5 kernel: Lustre: 16820:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369106497407707 sent from lustre-MDT0000-mdc-ffff8102feea9800 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305682717] [real_sent 1305682717] [current 1305682750] [deadline 33s] [delay 0s] req@ffff8103066be000 x1369106497407707/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 192/192 e 0 to 1 dl 1305682750 ref 1 fl Rpc:X/ffffffff/ffffffff rc 0/-1 May 17 18:39:10 client-5 kernel: Lustre: 16820:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 13 previous similar messages May 17 18:39:10 client-5 kernel: Lustre: 16820:0:(import.c:1190:completed_replay_interpret()) lustre-MDT0000-mdc-ffff8102feea9800: version recovery fails, reconnecting May 17 18:39:10 client-5 kernel: LustreError: 167-0: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. May 17 18:39:10 client-5 kernel: LustreError: 24940:0:(lmv_obd.c:1198:lmv_statfs()) can't stat MDS #0 (lustre-MDT0000-mdc-ffff8102feea9800), error -4 May 17 18:39:10 client-5 kernel: LustreError: 24940:0:(llite_lib.c:1370:ll_statfs_internal()) md_statfs fails: rc = -4 May 17 18:39:10 client-5 xinetd[3391]: EXIT: shell status=0 pid=24907 duration=124(sec) May 17 18:39:10 client-5 xinetd[3391]: START: shell pid=25015 from=192.168.4.5 May 17 18:39:10 client-5 rshd[25016]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 18:39:10 client-5 xinetd[3391]: EXIT: shell status=0 pid=25015 duration=0(sec) May 17 18:39:10 client-5 xinetd[3391]: START: shell pid=25187 from=192.168.4.5 May 17 18:39:10 client-5 rshd[25188]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-vbr test 4g: setattr of flags changes versions ============================================= 18:39:10 \(1305682750\)");echo XXRETCODE:$?' May 17 18:39:10 client-5 kernel: Lustre: DEBUG MARKER: == replay-vbr test 4g: setattr of flags changes versions ============================================= 18:39:10 (1305682750) May 17 18:39:10 client-5 xinetd[3391]: EXIT: shell status=0 pid=25187 duration=0(sec) May 17 18:39:11 client-5 xinetd[3391]: START: shell pid=25318 from=192.168.4.5 May 17 18:39:11 client-5 rshd[25319]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 18:39:11 client-5 xinetd[3391]: EXIT: shell status=0 pid=25318 duration=0(sec) May 17 18:39:11 client-5 xinetd[3391]: START: shell pid=25489 from=192.168.4.5 May 17 18:39:11 client-5 rshd[25491]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-vbr test 4h: setattr of flags checks versions ============================================== 18:39:11 \(1305682751\)");echo XXRETCODE:$?' May 17 18:39:11 client-5 kernel: Lustre: DEBUG MARKER: == replay-vbr test 4h: setattr of flags checks versions ============================================== 18:39:11 (1305682751) May 17 18:39:11 client-5 xinetd[3391]: EXIT: shell status=0 pid=25489 duration=0(sec) May 17 18:39:12 client-5 xinetd[3391]: START: shell pid=25576 from=192.168.4.5 May 17 18:39:12 client-5 rshd[25577]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 18:39:12 client-5 xinetd[3391]: EXIT: shell status=0 pid=25576 duration=0(sec) May 17 18:39:12 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 18:39:29 client-5 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail. May 17 18:39:31 client-5 xinetd[3391]: START: shell pid=25785 from=192.168.4.5 May 17 18:39:31 client-5 rshd[25786]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 18:39:32 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_statfs operation failed with -107 May 17 18:39:32 client-5 kernel: LustreError: Skipped 1 previous similar message May 17 18:39:32 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff8102feea9800: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 18:39:32 client-5 kernel: Lustre: Skipped 1 previous similar message May 17 18:39:44 client-5 kernel: Lustre: 16821:0:(import.c:529:import_select_connection()) MGC192.168.4.128@o2ib: tried all connections, increasing latency to 11s May 17 18:39:44 client-5 kernel: Lustre: 16820:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf651c89 to 0xfff98862bf652144 May 17 18:39:44 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 18:41:03 client-5 kernel: LustreError: 16820:0:(client.c:2570:ptlrpc_replay_interpret()) @@@ status -116, old was 0 req@ffff81032faa2c00 x1369106497407804/t34359738386(34359738386) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:23/10 lens 360/424 e 4 to 0 dl 1305682883 ref 2 fl Interpret:R/ffffffff/ffffffff rc -116/-1 May 17 18:41:03 client-5 kernel: Lustre: 16820:0:(client.c:2527:ptlrpc_replay_interpret()) @@@ Version mismatch during replay May 17 18:41:03 client-5 kernel: req@ffff810308d0d000 x1369106497407836/t34359738395(34359738395) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 456/424 e 0 to 0 dl 1305682896 ref 2 fl Interpret:R/ffffffff/ffffffff rc -75/-1 May 17 18:41:03 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff810329823000: Connection restored to service lustre-MDT0000 using nid 192.168.4.128@o2ib. May 17 18:41:03 client-5 kernel: Lustre: Skipped 5 previous similar messages May 17 18:41:36 client-5 kernel: Lustre: 16820:0:(import.c:1190:completed_replay_interpret()) lustre-MDT0000-mdc-ffff8102feea9800: version recovery fails, reconnecting May 17 18:41:36 client-5 kernel: LustreError: 167-0: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. May 17 18:41:36 client-5 kernel: LustreError: 25818:0:(lmv_obd.c:1198:lmv_statfs()) can't stat MDS #0 (lustre-MDT0000-mdc-ffff8102feea9800), error -4 May 17 18:41:36 client-5 kernel: LustreError: 25818:0:(llite_lib.c:1370:ll_statfs_internal()) md_statfs fails: rc = -4 May 17 18:41:36 client-5 xinetd[3391]: EXIT: shell status=0 pid=25785 duration=125(sec) May 17 18:41:36 client-5 xinetd[3391]: START: shell pid=25900 from=192.168.4.5 May 17 18:41:36 client-5 rshd[25901]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 18:41:36 client-5 xinetd[3391]: EXIT: shell status=0 pid=25900 duration=0(sec) May 17 18:41:36 client-5 xinetd[3391]: START: shell pid=26071 from=192.168.4.5 May 17 18:41:36 client-5 rshd[26073]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-vbr test 4i: setattr of times does not change versions ===================================== 18:41:36 \(1305682896\)");echo XXRETCODE:$?' May 17 18:41:36 client-5 kernel: Lustre: DEBUG MARKER: == replay-vbr test 4i: setattr of times does not change versions ===================================== 18:41:36 (1305682896) May 17 18:41:36 client-5 xinetd[3391]: EXIT: shell status=0 pid=26071 duration=0(sec) May 17 18:41:37 client-5 xinetd[3391]: START: shell pid=26224 from=192.168.4.5 May 17 18:41:37 client-5 rshd[26225]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 18:41:37 client-5 xinetd[3391]: EXIT: shell status=0 pid=26224 duration=0(sec) May 17 18:41:37 client-5 xinetd[3391]: START: shell pid=26396 from=192.168.4.5 May 17 18:41:37 client-5 rshd[26397]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-vbr test 4j: setattr of size does not change versions ====================================== 18:41:37 \(1305682897\)");echo XXRETCODE:$?' May 17 18:41:37 client-5 kernel: Lustre: DEBUG MARKER: == replay-vbr test 4j: setattr of size does not change versions ====================================== 18:41:37 (1305682897) May 17 18:41:37 client-5 xinetd[3391]: EXIT: shell status=0 pid=26396 duration=0(sec) May 17 18:41:38 client-5 xinetd[3391]: START: shell pid=26523 from=192.168.4.5 May 17 18:41:38 client-5 rshd[26524]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 18:41:38 client-5 xinetd[3391]: EXIT: shell status=0 pid=26523 duration=0(sec) May 17 18:41:38 client-5 xinetd[3391]: START: shell pid=26694 from=192.168.4.5 May 17 18:41:38 client-5 rshd[26696]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-vbr test 4k: setattr of times and size does not check versions ============================= 18:41:38 \(1305682898\)");echo XXRETCODE:$?' May 17 18:41:38 client-5 kernel: Lustre: DEBUG MARKER: == replay-vbr test 4k: setattr of times and size does not check versions ============================= 18:41:38 (1305682898) May 17 18:41:38 client-5 xinetd[3391]: EXIT: shell status=0 pid=26694 duration=0(sec) May 17 18:41:39 client-5 xinetd[3391]: START: shell pid=26789 from=192.168.4.5 May 17 18:41:39 client-5 rshd[26790]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 18:41:39 client-5 xinetd[3391]: EXIT: shell status=0 pid=26789 duration=0(sec) May 17 18:41:39 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 18:42:01 client-5 xinetd[3391]: START: shell pid=27014 from=192.168.4.5 May 17 18:42:01 client-5 rshd[27015]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 18:42:01 client-5 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail. May 17 18:42:01 client-5 kernel: Lustre: 16820:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf652144 to 0xfff98862bf65264c May 17 18:42:01 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 18:42:01 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_statfs operation failed with -107 May 17 18:42:01 client-5 kernel: LustreError: Skipped 1 previous similar message May 17 18:42:01 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff8102feea9800: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 18:42:01 client-5 kernel: Lustre: Skipped 1 previous similar message May 17 18:43:32 client-5 xinetd[3391]: EXIT: shell status=0 pid=27014 duration=91(sec) May 17 18:43:32 client-5 xinetd[3391]: START: shell pid=27122 from=192.168.4.5 May 17 18:43:32 client-5 rshd[27123]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 18:43:32 client-5 xinetd[3391]: EXIT: shell status=0 pid=27122 duration=0(sec) May 17 18:43:32 client-5 xinetd[3391]: START: shell pid=27294 from=192.168.4.5 May 17 18:43:32 client-5 rshd[27295]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-vbr test 5a: link changes versions of source but not target parent ========================= 18:43:32 \(1305683012\)");echo XXRETCODE:$?' May 17 18:43:32 client-5 kernel: Lustre: DEBUG MARKER: == replay-vbr test 5a: link changes versions of source but not target parent ========================= 18:43:32 (1305683012) May 17 18:43:32 client-5 xinetd[3391]: EXIT: shell status=0 pid=27294 duration=0(sec) May 17 18:43:33 client-5 xinetd[3391]: START: shell pid=27453 from=192.168.4.5 May 17 18:43:33 client-5 rshd[27454]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 18:43:33 client-5 xinetd[3391]: EXIT: shell status=0 pid=27453 duration=0(sec) May 17 18:43:33 client-5 xinetd[3391]: START: shell pid=27625 from=192.168.4.5 May 17 18:43:33 client-5 rshd[27626]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-vbr test 5b: link checks version of target parent ========================================== 18:43:33 \(1305683013\)");echo XXRETCODE:$?' May 17 18:43:33 client-5 kernel: Lustre: DEBUG MARKER: == replay-vbr test 5b: link checks version of target parent ========================================== 18:43:33 (1305683013) May 17 18:43:33 client-5 xinetd[3391]: EXIT: shell status=0 pid=27625 duration=0(sec) May 17 18:43:34 client-5 xinetd[3391]: START: shell pid=27715 from=192.168.4.5 May 17 18:43:34 client-5 rshd[27716]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 18:43:34 client-5 xinetd[3391]: EXIT: shell status=0 pid=27715 duration=0(sec) May 17 18:43:34 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 18:43:53 client-5 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail. May 17 18:43:53 client-5 kernel: Lustre: 16820:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf65264c to 0xfff98862bf652b2a May 17 18:43:54 client-5 xinetd[3391]: START: shell pid=27931 from=192.168.4.5 May 17 18:43:54 client-5 rshd[27932]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 18:45:25 client-5 kernel: Lustre: 16820:0:(client.c:2527:ptlrpc_replay_interpret()) @@@ Version mismatch during replay May 17 18:45:25 client-5 kernel: req@ffff8102f57da400 x1369106497408620/t42949672978(42949672978) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 456/408 e 0 to 0 dl 1305683158 ref 2 fl Interpret:R/ffffffff/ffffffff rc -75/-1 May 17 18:45:58 client-5 kernel: Lustre: 16820:0:(import.c:1190:completed_replay_interpret()) lustre-MDT0000-mdc-ffff8102feea9800: version recovery fails, reconnecting May 17 18:45:58 client-5 kernel: LustreError: 167-0: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. May 17 18:45:58 client-5 kernel: LustreError: 27964:0:(lmv_obd.c:1198:lmv_statfs()) can't stat MDS #0 (lustre-MDT0000-mdc-ffff8102feea9800), error -4 May 17 18:45:58 client-5 kernel: LustreError: 27964:0:(llite_lib.c:1370:ll_statfs_internal()) md_statfs fails: rc = -4 May 17 18:45:58 client-5 xinetd[3391]: EXIT: shell status=0 pid=27931 duration=124(sec) May 17 18:45:58 client-5 xinetd[3391]: START: shell pid=28033 from=192.168.4.5 May 17 18:45:58 client-5 rshd[28034]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 18:45:58 client-5 xinetd[3391]: EXIT: shell status=0 pid=28033 duration=0(sec) May 17 18:45:58 client-5 xinetd[3391]: START: shell pid=28205 from=192.168.4.5 May 17 18:45:58 client-5 rshd[28206]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-vbr test 5c: link checks version of source ================================================= 18:45:58 \(1305683158\)");echo XXRETCODE:$?' May 17 18:45:58 client-5 kernel: Lustre: DEBUG MARKER: == replay-vbr test 5c: link checks version of source ================================================= 18:45:58 (1305683158) May 17 18:45:58 client-5 xinetd[3391]: EXIT: shell status=0 pid=28205 duration=0(sec) May 17 18:45:59 client-5 xinetd[3391]: START: shell pid=28295 from=192.168.4.5 May 17 18:45:59 client-5 rshd[28296]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 18:45:59 client-5 xinetd[3391]: EXIT: shell status=0 pid=28295 duration=0(sec) May 17 18:45:59 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 18:46:18 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The obd_ping operation failed with -107 May 17 18:46:18 client-5 kernel: LustreError: Skipped 3 previous similar messages May 17 18:46:18 client-5 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail. May 17 18:46:18 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff8102feea9800: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 18:46:18 client-5 kernel: Lustre: Skipped 3 previous similar messages May 17 18:46:18 client-5 kernel: Lustre: 16820:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf652b2a to 0xfff98862bf652ecd May 17 18:46:18 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 18:46:18 client-5 kernel: Lustre: Skipped 1 previous similar message May 17 18:46:18 client-5 xinetd[3391]: START: shell pid=28510 from=192.168.4.5 May 17 18:46:18 client-5 rshd[28511]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 18:47:49 client-5 kernel: LustreError: 16820:0:(client.c:2570:ptlrpc_replay_interpret()) @@@ status -116, old was 0 req@ffff8102fc258800 x1369106497409029/t47244640259(47244640259) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:23/10 lens 360/424 e 6 to 0 dl 1305683297 ref 2 fl Interpret:R/ffffffff/ffffffff rc -116/-1 May 17 18:47:49 client-5 kernel: Lustre: 16820:0:(client.c:2527:ptlrpc_replay_interpret()) @@@ Version mismatch during replay May 17 18:47:49 client-5 kernel: req@ffff810330ec9000 x1369106497409042/t47244640265(47244640265) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 456/408 e 0 to 0 dl 1305683302 ref 2 fl Interpret:R/ffffffff/ffffffff rc -75/-1 May 17 18:48:22 client-5 kernel: Lustre: 16820:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369106497409343 sent from lustre-MDT0000-mdc-ffff8102feea9800 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305683269] [real_sent 1305683269] [current 1305683302] [deadline 33s] [delay 0s] req@ffff8103238f8c00 x1369106497409343/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 192/192 e 0 to 1 dl 1305683302 ref 1 fl Rpc:X/ffffffff/ffffffff rc 0/-1 May 17 18:48:22 client-5 kernel: Lustre: 16820:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 30 previous similar messages May 17 18:48:22 client-5 kernel: Lustre: 16820:0:(import.c:1190:completed_replay_interpret()) lustre-MDT0000-mdc-ffff8102feea9800: version recovery fails, reconnecting May 17 18:48:22 client-5 kernel: LustreError: 167-0: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. May 17 18:48:22 client-5 kernel: LustreError: 28543:0:(lmv_obd.c:1198:lmv_statfs()) can't stat MDS #0 (lustre-MDT0000-mdc-ffff8102feea9800), error -4 May 17 18:48:22 client-5 kernel: LustreError: 28543:0:(llite_lib.c:1370:ll_statfs_internal()) md_statfs fails: rc = -4 May 17 18:48:22 client-5 xinetd[3391]: EXIT: shell status=0 pid=28510 duration=124(sec) May 17 18:48:22 client-5 xinetd[3391]: START: shell pid=28612 from=192.168.4.5 May 17 18:48:22 client-5 rshd[28613]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 18:48:22 client-5 xinetd[3391]: EXIT: shell status=0 pid=28612 duration=0(sec) May 17 18:48:22 client-5 xinetd[3391]: START: shell pid=28784 from=192.168.4.5 May 17 18:48:22 client-5 rshd[28785]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-vbr test 6a: rename doesn\'t change versions of source parent and target parent ============= 18:48:22 \(1305683302\)");echo XXRETCODE:$?' May 17 18:48:23 client-5 kernel: Lustre: DEBUG MARKER: == replay-vbr test 6a: rename doesn't change versions of source parent and target parent ============= 18:48:22 (1305683302) May 17 18:48:23 client-5 xinetd[3391]: EXIT: shell status=0 pid=28784 duration=1(sec) May 17 18:48:23 client-5 xinetd[3391]: START: shell pid=28943 from=192.168.4.5 May 17 18:48:23 client-5 rshd[28944]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 18:48:23 client-5 xinetd[3391]: EXIT: shell status=0 pid=28943 duration=0(sec) May 17 18:48:23 client-5 xinetd[3391]: START: shell pid=29115 from=192.168.4.5 May 17 18:48:23 client-5 rshd[29116]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-vbr test 6b: rename within same dir doesn\'t change version of parent ======================= 18:48:23 \(1305683303\)");echo XXRETCODE:$?' May 17 18:48:24 client-5 kernel: Lustre: DEBUG MARKER: == replay-vbr test 6b: rename within same dir doesn't change version of parent ======================= 18:48:23 (1305683303) May 17 18:48:24 client-5 xinetd[3391]: EXIT: shell status=0 pid=29115 duration=1(sec) May 17 18:48:24 client-5 xinetd[3391]: START: shell pid=29242 from=192.168.4.5 May 17 18:48:24 client-5 rshd[29243]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 18:48:24 client-5 xinetd[3391]: EXIT: shell status=0 pid=29242 duration=0(sec) May 17 18:48:24 client-5 xinetd[3391]: START: shell pid=29414 from=192.168.4.5 May 17 18:48:24 client-5 rshd[29415]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-vbr test 6c: rename checks version of source parent ======================================== 18:48:24 \(1305683304\)");echo XXRETCODE:$?' May 17 18:48:24 client-5 kernel: Lustre: DEBUG MARKER: == replay-vbr test 6c: rename checks version of source parent ======================================== 18:48:24 (1305683304) May 17 18:48:24 client-5 xinetd[3391]: EXIT: shell status=0 pid=29414 duration=0(sec) May 17 18:48:25 client-5 xinetd[3391]: START: shell pid=29504 from=192.168.4.5 May 17 18:48:25 client-5 rshd[29505]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 18:48:25 client-5 xinetd[3391]: EXIT: shell status=0 pid=29504 duration=0(sec) May 17 18:48:25 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 18:48:45 client-5 xinetd[3391]: START: shell pid=29717 from=192.168.4.5 May 17 18:48:45 client-5 rshd[29718]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 18:48:45 client-5 kernel: Lustre: 16821:0:(import.c:529:import_select_connection()) MGC192.168.4.128@o2ib: tried all connections, increasing latency to 6s May 17 18:48:45 client-5 kernel: Lustre: 16820:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf652ecd to 0xfff98862bf653484 May 17 18:50:16 client-5 kernel: Lustre: 16820:0:(client.c:2527:ptlrpc_replay_interpret()) @@@ Version mismatch during replay May 17 18:50:16 client-5 kernel: req@ffff8102f2fd2c00 x1369106497409486/t51539607569(51539607569) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 472/424 e 0 to 0 dl 1305683449 ref 2 fl Interpret:R/ffffffff/ffffffff rc -75/-1 May 17 18:50:16 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff810329823000: Connection restored to service lustre-MDT0000 using nid 192.168.4.128@o2ib. May 17 18:50:16 client-5 kernel: Lustre: Skipped 11 previous similar messages May 17 18:50:49 client-5 kernel: Lustre: 16820:0:(import.c:1190:completed_replay_interpret()) lustre-MDT0000-mdc-ffff8102feea9800: version recovery fails, reconnecting May 17 18:50:49 client-5 kernel: LustreError: 167-0: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. May 17 18:50:49 client-5 kernel: LustreError: 29750:0:(lmv_obd.c:1198:lmv_statfs()) can't stat MDS #0 (lustre-MDT0000-mdc-ffff8102feea9800), error -4 May 17 18:50:49 client-5 kernel: LustreError: 29750:0:(llite_lib.c:1370:ll_statfs_internal()) md_statfs fails: rc = -4 May 17 18:50:49 client-5 xinetd[3391]: EXIT: shell status=0 pid=29717 duration=124(sec) May 17 18:50:49 client-5 xinetd[3391]: START: shell pid=29821 from=192.168.4.5 May 17 18:50:49 client-5 rshd[29822]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 18:50:49 client-5 xinetd[3391]: EXIT: shell status=0 pid=29821 duration=0(sec) May 17 18:50:49 client-5 xinetd[3391]: START: shell pid=29993 from=192.168.4.5 May 17 18:50:49 client-5 rshd[29994]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-vbr test 6d: rename checks version of target parent ======================================== 18:50:49 \(1305683449\)");echo XXRETCODE:$?' May 17 18:50:49 client-5 kernel: Lustre: DEBUG MARKER: == replay-vbr test 6d: rename checks version of target parent ======================================== 18:50:49 (1305683449) May 17 18:50:49 client-5 xinetd[3391]: EXIT: shell status=0 pid=29993 duration=0(sec) May 17 18:50:50 client-5 xinetd[3391]: START: shell pid=30083 from=192.168.4.5 May 17 18:50:50 client-5 rshd[30084]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 18:50:50 client-5 xinetd[3391]: EXIT: shell status=0 pid=30083 duration=0(sec) May 17 18:50:50 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 18:51:07 client-5 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail. May 17 18:51:07 client-5 kernel: LustreError: Skipped 1 previous similar message May 17 18:51:10 client-5 xinetd[3391]: START: shell pid=30296 from=192.168.4.5 May 17 18:51:10 client-5 rshd[30297]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 18:51:22 client-5 kernel: Lustre: 16821:0:(import.c:529:import_select_connection()) MGC192.168.4.128@o2ib: tried all connections, increasing latency to 11s May 17 18:51:22 client-5 kernel: Lustre: 16820:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf653484 to 0xfff98862bf6538dd May 17 18:51:22 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 18:51:22 client-5 kernel: Lustre: Skipped 1 previous similar message May 17 18:52:41 client-5 kernel: Lustre: 16820:0:(client.c:2527:ptlrpc_replay_interpret()) @@@ Version mismatch during replay May 17 18:52:41 client-5 kernel: req@ffff8102fe64b800 x1369106497409898/t55834574856(55834574856) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 472/424 e 0 to 0 dl 1305683594 ref 2 fl Interpret:R/ffffffff/ffffffff rc -75/-1 May 17 18:53:14 client-5 kernel: Lustre: 16820:0:(import.c:1190:completed_replay_interpret()) lustre-MDT0000-mdc-ffff8102feea9800: version recovery fails, reconnecting May 17 18:53:14 client-5 kernel: LustreError: 167-0: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. May 17 18:53:14 client-5 kernel: LustreError: 30329:0:(lmv_obd.c:1198:lmv_statfs()) can't stat MDS #0 (lustre-MDT0000-mdc-ffff8102feea9800), error -4 May 17 18:53:14 client-5 kernel: LustreError: 30329:0:(llite_lib.c:1370:ll_statfs_internal()) md_statfs fails: rc = -4 May 17 18:53:14 client-5 xinetd[3391]: EXIT: shell status=0 pid=30296 duration=124(sec) May 17 18:53:14 client-5 xinetd[3391]: START: shell pid=30398 from=192.168.4.5 May 17 18:53:14 client-5 rshd[30401]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 18:53:14 client-5 xinetd[3391]: EXIT: shell status=0 pid=30398 duration=0(sec) May 17 18:53:14 client-5 xinetd[3391]: START: shell pid=30572 from=192.168.4.5 May 17 18:53:14 client-5 rshd[30573]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-vbr test 7a: create, {lost}, create ======================================================== 18:53:14 \(1305683594\)");echo XXRETCODE:$?' May 17 18:53:14 client-5 kernel: Lustre: DEBUG MARKER: == replay-vbr test 7a: create, {lost}, create ======================================================== 18:53:14 (1305683594) May 17 18:53:14 client-5 xinetd[3391]: EXIT: shell status=0 pid=30572 duration=0(sec) May 17 18:53:15 client-5 xinetd[3391]: START: shell pid=30658 from=192.168.4.5 May 17 18:53:15 client-5 rshd[30659]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 18:53:15 client-5 xinetd[3391]: EXIT: shell status=0 pid=30658 duration=0(sec) May 17 18:53:15 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 18:53:34 client-5 kernel: Lustre: 16820:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf6538dd to 0xfff98862bf653ccd May 17 18:53:35 client-5 xinetd[3391]: START: shell pid=30873 from=192.168.4.5 May 17 18:53:35 client-5 rshd[30874]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 18:55:06 client-5 kernel: LustreError: 16820:0:(client.c:2570:ptlrpc_replay_interpret()) @@@ status -116, old was 0 req@ffff8102fade1000 x1369106497410292/t60129542155(60129542155) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:23/10 lens 360/424 e 0 to 0 dl 1305683724 ref 2 fl Interpret:R/ffffffff/ffffffff rc -116/-1 May 17 18:55:39 client-5 kernel: Lustre: 16820:0:(import.c:1190:completed_replay_interpret()) lustre-MDT0000-mdc-ffff8102feea9800: version recovery fails, reconnecting May 17 18:55:39 client-5 xinetd[3391]: EXIT: shell status=0 pid=30873 duration=124(sec) May 17 18:55:42 client-5 xinetd[3391]: START: shell pid=31025 from=192.168.4.5 May 17 18:55:42 client-5 rshd[31026]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 18:55:42 client-5 xinetd[3391]: EXIT: shell status=0 pid=31025 duration=0(sec) May 17 18:55:42 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 18:56:01 client-5 kernel: Lustre: 16820:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf653ccd to 0xfff98862bf654173 May 17 18:56:01 client-5 xinetd[3391]: START: shell pid=31240 from=192.168.4.5 May 17 18:56:01 client-5 rshd[31241]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 18:56:01 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_statfs operation failed with -107 May 17 18:56:01 client-5 kernel: LustreError: Skipped 8 previous similar messages May 17 18:56:01 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff8102feea9800: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 18:56:01 client-5 kernel: Lustre: Skipped 7 previous similar messages May 17 18:57:32 client-5 kernel: Lustre: 16820:0:(client.c:2527:ptlrpc_replay_interpret()) @@@ Version mismatch during replay May 17 18:57:32 client-5 kernel: req@ffff8103238f8800 x1369106497410724/t64424509454(64424509454) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 456/416 e 0 to 0 dl 1305683885 ref 2 fl Interpret:R/ffffffff/ffffffff rc -75/-1 May 17 18:57:32 client-5 kernel: Lustre: 16820:0:(client.c:2527:ptlrpc_replay_interpret()) Skipped 1 previous similar message May 17 18:58:05 client-5 kernel: Lustre: 16820:0:(import.c:1190:completed_replay_interpret()) lustre-MDT0000-mdc-ffff8102feea9800: version recovery fails, reconnecting May 17 18:58:05 client-5 kernel: LustreError: 167-0: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. May 17 18:58:05 client-5 kernel: LustreError: Skipped 1 previous similar message May 17 18:58:05 client-5 kernel: LustreError: 31273:0:(lmv_obd.c:1198:lmv_statfs()) can't stat MDS #0 (lustre-MDT0000-mdc-ffff8102feea9800), error -4 May 17 18:58:05 client-5 kernel: LustreError: 31273:0:(lmv_obd.c:1198:lmv_statfs()) Skipped 1 previous similar message May 17 18:58:05 client-5 kernel: LustreError: 31273:0:(llite_lib.c:1370:ll_statfs_internal()) md_statfs fails: rc = -4 May 17 18:58:05 client-5 kernel: LustreError: 31273:0:(llite_lib.c:1370:ll_statfs_internal()) Skipped 1 previous similar message May 17 18:58:05 client-5 xinetd[3391]: EXIT: shell status=0 pid=31240 duration=124(sec) May 17 18:58:08 client-5 xinetd[3391]: START: shell pid=31392 from=192.168.4.5 May 17 18:58:08 client-5 rshd[31393]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 18:58:08 client-5 xinetd[3391]: EXIT: shell status=0 pid=31392 duration=0(sec) May 17 18:58:09 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 18:58:28 client-5 kernel: Lustre: 16819:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369106497411158 sent from MGC192.168.4.128@o2ib to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305683891] [real_sent 1305683891] [current 1305683908] [deadline 17s] [delay 0s] req@ffff810339244800 x1369106497411158/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 192/192 e 0 to 1 dl 1305683908 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 18:58:28 client-5 kernel: Lustre: 16819:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 34 previous similar messages May 17 18:58:28 client-5 kernel: Lustre: 16820:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf654173 to 0xfff98862bf65466d May 17 18:58:28 client-5 xinetd[3391]: START: shell pid=31607 from=192.168.4.5 May 17 18:58:28 client-5 rshd[31608]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 18:59:59 client-5 kernel: LustreError: 16820:0:(client.c:2570:ptlrpc_replay_interpret()) @@@ status -116, old was 0 req@ffff81016d0a9c00 x1369106497411157/t68719476750(68719476750) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:23/10 lens 360/424 e 0 to 0 dl 1305684006 ref 2 fl Interpret:R/ffffffff/ffffffff rc -116/-1 May 17 19:00:32 client-5 kernel: Lustre: 16820:0:(import.c:1190:completed_replay_interpret()) lustre-MDT0000-mdc-ffff8102feea9800: version recovery fails, reconnecting May 17 19:00:32 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff8102feea9800: Connection restored to service lustre-MDT0000 using nid 192.168.4.128@o2ib. May 17 19:00:32 client-5 kernel: Lustre: Skipped 12 previous similar messages May 17 19:00:32 client-5 xinetd[3391]: EXIT: shell status=0 pid=31607 duration=124(sec) May 17 19:00:34 client-5 xinetd[3391]: START: shell pid=31763 from=192.168.4.5 May 17 19:00:34 client-5 rshd[31766]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 19:00:35 client-5 xinetd[3391]: EXIT: shell status=0 pid=31763 duration=1(sec) May 17 19:00:35 client-5 xinetd[3391]: START: shell pid=31935 from=192.168.4.5 May 17 19:00:35 client-5 rshd[31938]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-vbr test 7b: create, {lost}, unlink ======================================================== 19:00:35 \(1305684035\)");echo XXRETCODE:$?' May 17 19:00:35 client-5 kernel: Lustre: DEBUG MARKER: == replay-vbr test 7b: create, {lost}, unlink ======================================================== 19:00:35 (1305684035) May 17 19:00:35 client-5 xinetd[3391]: EXIT: shell status=0 pid=31935 duration=0(sec) May 17 19:00:36 client-5 xinetd[3391]: START: shell pid=32023 from=192.168.4.5 May 17 19:00:36 client-5 rshd[32024]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 19:00:36 client-5 xinetd[3391]: EXIT: shell status=0 pid=32023 duration=0(sec) May 17 19:00:36 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 19:00:45 client-5 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail. May 17 19:00:45 client-5 kernel: LustreError: Skipped 3 previous similar messages May 17 19:00:55 client-5 kernel: Lustre: 16821:0:(import.c:529:import_select_connection()) MGC192.168.4.128@o2ib: tried all connections, increasing latency to 6s May 17 19:00:55 client-5 kernel: Lustre: 16820:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf65466d to 0xfff98862bf654b6e May 17 19:00:55 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 19:00:55 client-5 kernel: Lustre: Skipped 3 previous similar messages May 17 19:00:55 client-5 xinetd[3391]: START: shell pid=32238 from=192.168.4.5 May 17 19:00:55 client-5 rshd[32239]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 19:02:59 client-5 kernel: Lustre: 16820:0:(import.c:1190:completed_replay_interpret()) lustre-MDT0000-mdc-ffff8102feea9800: version recovery fails, reconnecting May 17 19:02:59 client-5 xinetd[3391]: EXIT: shell status=0 pid=32238 duration=124(sec) May 17 19:03:02 client-5 xinetd[3391]: START: shell pid=32396 from=192.168.4.5 May 17 19:03:02 client-5 rshd[32397]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 19:03:02 client-5 xinetd[3391]: EXIT: shell status=0 pid=32396 duration=0(sec) May 17 19:03:02 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 19:03:22 client-5 xinetd[3391]: START: shell pid=32609 from=192.168.4.5 May 17 19:03:22 client-5 rshd[32610]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 19:03:32 client-5 kernel: Lustre: 16821:0:(import.c:529:import_select_connection()) MGC192.168.4.128@o2ib: tried all connections, increasing latency to 11s May 17 19:03:32 client-5 kernel: Lustre: 16820:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf654b6e to 0xfff98862bf655099 May 17 19:05:26 client-5 kernel: Lustre: 16820:0:(import.c:1190:completed_replay_interpret()) lustre-MDT0000-mdc-ffff8102feea9800: version recovery fails, reconnecting May 17 19:05:26 client-5 xinetd[3391]: EXIT: shell status=0 pid=32609 duration=124(sec) May 17 19:05:29 client-5 xinetd[3391]: START: shell pid=32763 from=192.168.4.5 May 17 19:05:29 client-5 rshd[32764]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 19:05:29 client-5 xinetd[3391]: EXIT: shell status=0 pid=32763 duration=0(sec) May 17 19:05:29 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 19:05:49 client-5 xinetd[3391]: START: shell pid=518 from=192.168.4.5 May 17 19:05:49 client-5 rshd[519]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 19:05:49 client-5 kernel: Lustre: 16820:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf655099 to 0xfff98862bf655569 May 17 19:07:20 client-5 kernel: Lustre: 16820:0:(client.c:2527:ptlrpc_replay_interpret()) @@@ Version mismatch during replay May 17 19:07:20 client-5 kernel: req@ffff810300fbc800 x1369106497412437/t81604378639(81604378639) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 456/424 e 0 to 0 dl 1305684473 ref 2 fl Interpret:R/ffffffff/ffffffff rc -75/-1 May 17 19:07:20 client-5 kernel: Lustre: 16820:0:(client.c:2527:ptlrpc_replay_interpret()) Skipped 3 previous similar messages May 17 19:07:53 client-5 kernel: Lustre: 16820:0:(import.c:1190:completed_replay_interpret()) lustre-MDT0000-mdc-ffff8102feea9800: version recovery fails, reconnecting May 17 19:07:53 client-5 kernel: LustreError: 167-0: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. May 17 19:07:53 client-5 kernel: LustreError: Skipped 3 previous similar messages May 17 19:07:53 client-5 kernel: LustreError: 554:0:(lmv_obd.c:1198:lmv_statfs()) can't stat MDS #0 (lustre-MDT0000-mdc-ffff8102feea9800), error -4 May 17 19:07:53 client-5 kernel: LustreError: 554:0:(lmv_obd.c:1198:lmv_statfs()) Skipped 3 previous similar messages May 17 19:07:53 client-5 kernel: LustreError: 554:0:(llite_lib.c:1370:ll_statfs_internal()) md_statfs fails: rc = -4 May 17 19:07:53 client-5 kernel: LustreError: 554:0:(llite_lib.c:1370:ll_statfs_internal()) Skipped 3 previous similar messages May 17 19:07:53 client-5 xinetd[3391]: EXIT: shell status=0 pid=518 duration=124(sec) May 17 19:07:55 client-5 xinetd[3391]: START: shell pid=691 from=192.168.4.5 May 17 19:07:55 client-5 rshd[694]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 19:07:55 client-5 xinetd[3391]: EXIT: shell status=0 pid=691 duration=0(sec) May 17 19:07:55 client-5 xinetd[3391]: START: shell pid=865 from=192.168.4.5 May 17 19:07:55 client-5 rshd[866]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-vbr test 7c: create, {lost}, rename ======================================================== 19:07:55 \(1305684475\)");echo XXRETCODE:$?' May 17 19:07:55 client-5 kernel: Lustre: DEBUG MARKER: == replay-vbr test 7c: create, {lost}, rename ======================================================== 19:07:55 (1305684475) May 17 19:07:55 client-5 xinetd[3391]: EXIT: shell status=0 pid=865 duration=0(sec) May 17 19:07:56 client-5 xinetd[3391]: START: shell pid=952 from=192.168.4.5 May 17 19:07:56 client-5 rshd[953]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 19:07:56 client-5 xinetd[3391]: EXIT: shell status=0 pid=952 duration=0(sec) May 17 19:07:57 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 19:08:16 client-5 xinetd[3391]: START: shell pid=1165 from=192.168.4.5 May 17 19:08:16 client-5 rshd[1166]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 19:08:16 client-5 kernel: Lustre: 16820:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf655569 to 0xfff98862bf655aa2 May 17 19:08:16 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_statfs operation failed with -107 May 17 19:08:16 client-5 kernel: LustreError: Skipped 9 previous similar messages May 17 19:08:16 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff8102feea9800: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 19:08:16 client-5 kernel: Lustre: Skipped 9 previous similar messages May 17 19:08:32 client-5 kernel: Lustre: 16819:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369106497412878 sent from lustre-MDT0000-mdc-ffff810329823000 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305684479] [real_sent 1305684479] [current 1305684512] [deadline 33s] [delay 0s] req@ffff81033f76a000 x1369106497412878/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 192/192 e 0 to 1 dl 1305684512 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 19:08:32 client-5 kernel: Lustre: 16819:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 34 previous similar messages May 17 19:10:20 client-5 kernel: Lustre: 16820:0:(import.c:1190:completed_replay_interpret()) lustre-MDT0000-mdc-ffff8102feea9800: version recovery fails, reconnecting May 17 19:10:20 client-5 xinetd[3391]: EXIT: shell status=0 pid=1165 duration=124(sec) May 17 19:10:23 client-5 xinetd[3391]: START: shell pid=1319 from=192.168.4.5 May 17 19:10:23 client-5 rshd[1320]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 19:10:23 client-5 xinetd[3391]: EXIT: shell status=0 pid=1319 duration=0(sec) May 17 19:10:23 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 19:10:43 client-5 xinetd[3391]: START: shell pid=1533 from=192.168.4.5 May 17 19:10:43 client-5 rshd[1534]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 19:10:43 client-5 kernel: Lustre: 16820:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf655aa2 to 0xfff98862bf655ff7 May 17 19:10:43 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Connection restored to service MGS using nid 192.168.4.128@o2ib. May 17 19:10:43 client-5 kernel: Lustre: Skipped 12 previous similar messages May 17 19:12:46 client-5 kernel: Lustre: 16820:0:(import.c:1190:completed_replay_interpret()) lustre-MDT0000-mdc-ffff8102feea9800: version recovery fails, reconnecting May 17 19:12:46 client-5 xinetd[3391]: EXIT: shell status=0 pid=1533 duration=123(sec) May 17 19:12:50 client-5 xinetd[3391]: START: shell pid=1691 from=192.168.4.5 May 17 19:12:50 client-5 rshd[1692]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 19:12:50 client-5 xinetd[3391]: EXIT: shell status=0 pid=1691 duration=0(sec) May 17 19:12:50 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 19:13:00 client-5 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail. May 17 19:13:00 client-5 kernel: LustreError: Skipped 4 previous similar messages May 17 19:13:09 client-5 xinetd[3391]: START: shell pid=1913 from=192.168.4.5 May 17 19:13:09 client-5 rshd[1914]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 19:13:10 client-5 kernel: Lustre: 16821:0:(import.c:529:import_select_connection()) MGC192.168.4.128@o2ib: tried all connections, increasing latency to 6s May 17 19:13:10 client-5 kernel: Lustre: 16820:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf655ff7 to 0xfff98862bf656568 May 17 19:13:10 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 19:13:10 client-5 kernel: Lustre: Skipped 4 previous similar messages May 17 19:15:13 client-5 kernel: Lustre: 16820:0:(import.c:1190:completed_replay_interpret()) lustre-MDT0000-mdc-ffff8102feea9800: version recovery fails, reconnecting May 17 19:15:13 client-5 xinetd[3391]: EXIT: shell status=0 pid=1913 duration=124(sec) May 17 19:15:16 client-5 xinetd[3391]: START: shell pid=2069 from=192.168.4.5 May 17 19:15:16 client-5 rshd[2070]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 19:15:16 client-5 xinetd[3391]: EXIT: shell status=0 pid=2069 duration=0(sec) May 17 19:15:17 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 19:15:36 client-5 xinetd[3391]: START: shell pid=2296 from=192.168.4.5 May 17 19:15:36 client-5 rshd[2297]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 19:15:47 client-5 kernel: Lustre: 16821:0:(import.c:529:import_select_connection()) MGC192.168.4.128@o2ib: tried all connections, increasing latency to 11s May 17 19:15:47 client-5 kernel: Lustre: 16820:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf656568 to 0xfff98862bf656b7a May 17 19:17:40 client-5 kernel: Lustre: 16820:0:(import.c:1190:completed_replay_interpret()) lustre-MDT0000-mdc-ffff8102feea9800: version recovery fails, reconnecting May 17 19:17:40 client-5 xinetd[3391]: EXIT: shell status=0 pid=2296 duration=124(sec) May 17 19:17:42 client-5 xinetd[3391]: START: shell pid=2466 from=192.168.4.5 May 17 19:17:43 client-5 rshd[2467]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 19:17:43 client-5 xinetd[3391]: EXIT: shell status=0 pid=2466 duration=1(sec) May 17 19:17:43 client-5 xinetd[3391]: START: shell pid=2644 from=192.168.4.5 May 17 19:17:43 client-5 rshd[2645]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-vbr test 7d: unlink, {lost}, create ======================================================== 19:17:43 \(1305685063\)");echo XXRETCODE:$?' May 17 19:17:43 client-5 kernel: Lustre: DEBUG MARKER: == replay-vbr test 7d: unlink, {lost}, create ======================================================== 19:17:43 (1305685063) May 17 19:17:43 client-5 xinetd[3391]: EXIT: shell status=0 pid=2644 duration=0(sec) May 17 19:17:44 client-5 xinetd[3391]: START: shell pid=2730 from=192.168.4.5 May 17 19:17:44 client-5 rshd[2731]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 19:17:44 client-5 xinetd[3391]: EXIT: shell status=0 pid=2730 duration=0(sec) May 17 19:17:44 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 19:18:04 client-5 xinetd[3391]: START: shell pid=2945 from=192.168.4.5 May 17 19:18:04 client-5 rshd[2946]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 19:18:04 client-5 kernel: Lustre: 16820:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf656b7a to 0xfff98862bf657074 May 17 19:18:35 client-5 kernel: Lustre: 16819:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369106497414652 sent from lustre-MDT0000-mdc-ffff8102feea9800 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305685082] [real_sent 1305685082] [current 1305685115] [deadline 33s] [delay 0s] req@ffff810333a94000 x1369106497414652/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 192/192 e 0 to 1 dl 1305685115 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 19:18:35 client-5 kernel: Lustre: 16819:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 38 previous similar messages May 17 19:19:35 client-5 xinetd[3391]: EXIT: shell status=0 pid=2945 duration=91(sec) May 17 19:19:38 client-5 xinetd[3391]: START: shell pid=3106 from=192.168.4.5 May 17 19:19:38 client-5 rshd[3107]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 19:19:38 client-5 xinetd[3391]: EXIT: shell status=0 pid=3106 duration=0(sec) May 17 19:19:38 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 19:19:57 client-5 xinetd[3391]: START: shell pid=3343 from=192.168.4.5 May 17 19:19:58 client-5 rshd[3347]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 19:19:58 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_statfs operation failed with -107 May 17 19:19:58 client-5 kernel: LustreError: Skipped 9 previous similar messages May 17 19:19:58 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff8102feea9800: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 19:19:58 client-5 kernel: Lustre: Skipped 9 previous similar messages May 17 19:19:59 client-5 kernel: Lustre: 16820:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf657074 to 0xfff98862bf65757c May 17 19:21:28 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff810329823000: Connection restored to service lustre-MDT0000 using nid 192.168.4.128@o2ib. May 17 19:21:28 client-5 kernel: Lustre: Skipped 12 previous similar messages May 17 19:21:28 client-5 xinetd[3391]: EXIT: shell status=0 pid=3343 duration=91(sec) May 17 19:21:31 client-5 xinetd[3391]: START: shell pid=3521 from=192.168.4.5 May 17 19:21:31 client-5 rshd[3522]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 19:21:32 client-5 xinetd[3391]: EXIT: shell status=0 pid=3521 duration=1(sec) May 17 19:21:32 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 19:21:51 client-5 kernel: Lustre: 16820:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf65757c to 0xfff98862bf657aca May 17 19:21:51 client-5 xinetd[3391]: START: shell pid=3744 from=192.168.4.5 May 17 19:21:51 client-5 rshd[3745]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 19:23:22 client-5 xinetd[3391]: EXIT: shell status=0 pid=3744 duration=91(sec) May 17 19:23:25 client-5 xinetd[3391]: START: shell pid=3894 from=192.168.4.5 May 17 19:23:25 client-5 rshd[3897]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 19:23:25 client-5 xinetd[3391]: EXIT: shell status=0 pid=3894 duration=0(sec) May 17 19:23:25 client-5 xinetd[3391]: START: shell pid=4068 from=192.168.4.5 May 17 19:23:25 client-5 rshd[4069]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-vbr test 7e: unlink, {lost}, unlink ======================================================== 19:23:25 \(1305685405\)");echo XXRETCODE:$?' May 17 19:23:25 client-5 kernel: Lustre: DEBUG MARKER: == replay-vbr test 7e: unlink, {lost}, unlink ======================================================== 19:23:25 (1305685405) May 17 19:23:25 client-5 xinetd[3391]: EXIT: shell status=0 pid=4068 duration=0(sec) May 17 19:23:26 client-5 xinetd[3391]: START: shell pid=4154 from=192.168.4.5 May 17 19:23:26 client-5 rshd[4155]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 19:23:26 client-5 xinetd[3391]: EXIT: shell status=0 pid=4154 duration=0(sec) May 17 19:23:26 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 19:23:45 client-5 xinetd[3391]: START: shell pid=4373 from=192.168.4.5 May 17 19:23:45 client-5 rshd[4374]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 19:23:46 client-5 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail. May 17 19:23:46 client-5 kernel: LustreError: Skipped 4 previous similar messages May 17 19:23:46 client-5 kernel: Lustre: 16820:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf657aca to 0xfff98862bf658026 May 17 19:23:46 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 19:23:46 client-5 kernel: Lustre: Skipped 4 previous similar messages May 17 19:25:16 client-5 kernel: Lustre: 16820:0:(client.c:2527:ptlrpc_replay_interpret()) @@@ Version mismatch during replay May 17 19:25:16 client-5 kernel: req@ffff8102f388d400 x1369106497415664/t115964117011(115964117011) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 456/424 e 0 to 0 dl 1305685549 ref 2 fl Interpret:R/ffffffff/ffffffff rc -75/-1 May 17 19:25:16 client-5 kernel: Lustre: 16820:0:(client.c:2527:ptlrpc_replay_interpret()) Skipped 4 previous similar messages May 17 19:25:49 client-5 kernel: Lustre: 16820:0:(import.c:1190:completed_replay_interpret()) lustre-MDT0000-mdc-ffff8102feea9800: version recovery fails, reconnecting May 17 19:25:49 client-5 kernel: LustreError: 167-0: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. May 17 19:25:49 client-5 kernel: LustreError: Skipped 4 previous similar messages May 17 19:25:49 client-5 kernel: LustreError: 4406:0:(lmv_obd.c:1198:lmv_statfs()) can't stat MDS #0 (lustre-MDT0000-mdc-ffff8102feea9800), error -4 May 17 19:25:49 client-5 kernel: LustreError: 4406:0:(lmv_obd.c:1198:lmv_statfs()) Skipped 4 previous similar messages May 17 19:25:49 client-5 kernel: LustreError: 4406:0:(llite_lib.c:1370:ll_statfs_internal()) md_statfs fails: rc = -4 May 17 19:25:49 client-5 kernel: LustreError: 4406:0:(llite_lib.c:1370:ll_statfs_internal()) Skipped 4 previous similar messages May 17 19:25:49 client-5 xinetd[3391]: EXIT: shell status=0 pid=4373 duration=124(sec) May 17 19:25:52 client-5 xinetd[3391]: START: shell pid=4523 from=192.168.4.5 May 17 19:25:52 client-5 rshd[4524]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 19:25:52 client-5 xinetd[3391]: EXIT: shell status=0 pid=4523 duration=0(sec) May 17 19:25:53 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 19:26:12 client-5 xinetd[3391]: START: shell pid=4742 from=192.168.4.5 May 17 19:26:12 client-5 rshd[4743]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 19:26:13 client-5 kernel: Lustre: 16821:0:(import.c:529:import_select_connection()) MGC192.168.4.128@o2ib: tried all connections, increasing latency to 6s May 17 19:26:13 client-5 kernel: Lustre: 16820:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf658026 to 0xfff98862bf65850b May 17 19:28:16 client-5 kernel: Lustre: 16820:0:(import.c:1190:completed_replay_interpret()) lustre-MDT0000-mdc-ffff8102feea9800: version recovery fails, reconnecting May 17 19:28:16 client-5 xinetd[3391]: EXIT: shell status=0 pid=4742 duration=124(sec) May 17 19:28:19 client-5 xinetd[3391]: START: shell pid=4892 from=192.168.4.5 May 17 19:28:19 client-5 rshd[4893]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 19:28:19 client-5 xinetd[3391]: EXIT: shell status=0 pid=4892 duration=0(sec) May 17 19:28:19 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 19:28:39 client-5 xinetd[3391]: START: shell pid=5111 from=192.168.4.5 May 17 19:28:39 client-5 rshd[5112]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 19:28:46 client-5 kernel: Lustre: 16820:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369106497416562 sent from MGC192.168.4.128@o2ib to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305685715] [real_sent 1305685715] [current 1305685726] [deadline 11s] [delay 0s] req@ffff81031ecba400 x1369106497416562/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 368/392 e 0 to 1 dl 1305685726 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 19:28:46 client-5 kernel: Lustre: 16820:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 30 previous similar messages May 17 19:28:50 client-5 kernel: Lustre: 16821:0:(import.c:529:import_select_connection()) MGC192.168.4.128@o2ib: tried all connections, increasing latency to 11s May 17 19:28:50 client-5 kernel: Lustre: 16820:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf65850b to 0xfff98862bf6589d4 May 17 19:30:43 client-5 kernel: Lustre: 16820:0:(import.c:1190:completed_replay_interpret()) lustre-MDT0000-mdc-ffff8102feea9800: version recovery fails, reconnecting May 17 19:30:43 client-5 xinetd[3391]: EXIT: shell status=0 pid=5111 duration=124(sec) May 17 19:30:46 client-5 xinetd[3391]: START: shell pid=5261 from=192.168.4.5 May 17 19:30:46 client-5 rshd[5262]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 19:30:46 client-5 xinetd[3391]: EXIT: shell status=0 pid=5261 duration=0(sec) May 17 19:30:47 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 19:31:05 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The obd_ping operation failed with -107 May 17 19:31:05 client-5 kernel: LustreError: Skipped 11 previous similar messages May 17 19:31:05 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff8102feea9800: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 19:31:05 client-5 kernel: Lustre: Skipped 9 previous similar messages May 17 19:31:05 client-5 kernel: Lustre: 16820:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf6589d4 to 0xfff98862bf658e11 May 17 19:31:06 client-5 xinetd[3391]: START: shell pid=5478 from=192.168.4.5 May 17 19:31:06 client-5 rshd[5479]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 19:32:39 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff810329823000: Connection restored to service lustre-MDT0000 using nid 192.168.4.128@o2ib. May 17 19:32:39 client-5 kernel: Lustre: Skipped 14 previous similar messages May 17 19:33:12 client-5 kernel: Lustre: 16820:0:(import.c:1190:completed_replay_interpret()) lustre-MDT0000-mdc-ffff8102feea9800: version recovery fails, reconnecting May 17 19:33:12 client-5 xinetd[3391]: EXIT: shell status=0 pid=5478 duration=126(sec) May 17 19:33:15 client-5 xinetd[3391]: START: shell pid=5630 from=192.168.4.5 May 17 19:33:15 client-5 rshd[5631]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 19:33:15 client-5 xinetd[3391]: EXIT: shell status=0 pid=5630 duration=0(sec) May 17 19:33:15 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 19:33:35 client-5 xinetd[3391]: START: shell pid=5845 from=192.168.4.5 May 17 19:33:35 client-5 rshd[5846]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 19:33:35 client-5 kernel: Lustre: 16820:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf658e11 to 0xfff98862bf6593b3 May 17 19:35:39 client-5 kernel: Lustre: 16820:0:(import.c:1190:completed_replay_interpret()) lustre-MDT0000-mdc-ffff8102feea9800: version recovery fails, reconnecting May 17 19:35:39 client-5 xinetd[3391]: EXIT: shell status=0 pid=5845 duration=124(sec) May 17 19:35:41 client-5 xinetd[3391]: START: shell pid=6004 from=192.168.4.5 May 17 19:35:41 client-5 rshd[6005]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 19:35:41 client-5 xinetd[3391]: EXIT: shell status=0 pid=6004 duration=0(sec) May 17 19:35:41 client-5 xinetd[3391]: START: shell pid=6175 from=192.168.4.5 May 17 19:35:41 client-5 rshd[6177]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-vbr test 7f: unlink, {lost}, rename ======================================================== 19:35:41 \(1305686141\)");echo XXRETCODE:$?' May 17 19:35:42 client-5 kernel: Lustre: DEBUG MARKER: == replay-vbr test 7f: unlink, {lost}, rename ======================================================== 19:35:41 (1305686141) May 17 19:35:42 client-5 xinetd[3391]: EXIT: shell status=0 pid=6175 duration=1(sec) May 17 19:35:42 client-5 xinetd[3391]: START: shell pid=6262 from=192.168.4.5 May 17 19:35:42 client-5 rshd[6263]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 19:35:42 client-5 xinetd[3391]: EXIT: shell status=0 pid=6262 duration=0(sec) May 17 19:35:43 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 19:36:02 client-5 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail. May 17 19:36:02 client-5 kernel: LustreError: Skipped 4 previous similar messages May 17 19:36:02 client-5 kernel: Lustre: 16820:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf6593b3 to 0xfff98862bf659916 May 17 19:36:02 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 19:36:02 client-5 kernel: Lustre: Skipped 4 previous similar messages May 17 19:36:02 client-5 xinetd[3391]: START: shell pid=6479 from=192.168.4.5 May 17 19:36:02 client-5 rshd[6480]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 19:37:33 client-5 kernel: Lustre: 16820:0:(client.c:2527:ptlrpc_replay_interpret()) @@@ Version mismatch during replay May 17 19:37:33 client-5 kernel: req@ffff8103238f8000 x1369106497417835/t137438953488(137438953488) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 472/424 e 0 to 0 dl 1305686286 ref 2 fl Interpret:R/ffffffff/ffffffff rc -75/-1 May 17 19:37:33 client-5 kernel: Lustre: 16820:0:(client.c:2527:ptlrpc_replay_interpret()) Skipped 4 previous similar messages May 17 19:38:06 client-5 kernel: Lustre: 16820:0:(import.c:1190:completed_replay_interpret()) lustre-MDT0000-mdc-ffff8102feea9800: version recovery fails, reconnecting May 17 19:38:06 client-5 kernel: LustreError: 167-0: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. May 17 19:38:06 client-5 kernel: LustreError: Skipped 4 previous similar messages May 17 19:38:06 client-5 kernel: LustreError: 6512:0:(lmv_obd.c:1198:lmv_statfs()) can't stat MDS #0 (lustre-MDT0000-mdc-ffff8102feea9800), error -4 May 17 19:38:06 client-5 kernel: LustreError: 6512:0:(lmv_obd.c:1198:lmv_statfs()) Skipped 4 previous similar messages May 17 19:38:06 client-5 kernel: LustreError: 6512:0:(llite_lib.c:1370:ll_statfs_internal()) md_statfs fails: rc = -4 May 17 19:38:06 client-5 kernel: LustreError: 6512:0:(llite_lib.c:1370:ll_statfs_internal()) Skipped 4 previous similar messages May 17 19:38:06 client-5 xinetd[3391]: EXIT: shell status=0 pid=6479 duration=124(sec) May 17 19:38:09 client-5 xinetd[3391]: START: shell pid=6630 from=192.168.4.5 May 17 19:38:09 client-5 rshd[6632]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 19:38:09 client-5 xinetd[3391]: EXIT: shell status=0 pid=6630 duration=0(sec) May 17 19:38:09 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 19:38:29 client-5 xinetd[3391]: START: shell pid=6846 from=192.168.4.5 May 17 19:38:29 client-5 kernel: Lustre: 16821:0:(import.c:529:import_select_connection()) MGC192.168.4.128@o2ib: tried all connections, increasing latency to 6s May 17 19:38:29 client-5 kernel: Lustre: 16820:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf659916 to 0xfff98862bf659e33 May 17 19:38:29 client-5 rshd[6849]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 19:38:50 client-5 kernel: Lustre: 16819:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369106497418286 sent from lustre-MDT0000-mdc-ffff8102feea9800 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305686297] [real_sent 1305686297] [current 1305686330] [deadline 33s] [delay 0s] req@ffff81031ecba400 x1369106497418286/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 192/192 e 0 to 1 dl 1305686330 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 19:38:50 client-5 kernel: Lustre: 16819:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 30 previous similar messages May 17 19:40:33 client-5 kernel: Lustre: 16820:0:(import.c:1190:completed_replay_interpret()) lustre-MDT0000-mdc-ffff8102feea9800: version recovery fails, reconnecting May 17 19:40:33 client-5 xinetd[3391]: EXIT: shell status=0 pid=6846 duration=124(sec) May 17 19:40:36 client-5 xinetd[3391]: START: shell pid=7000 from=192.168.4.5 May 17 19:40:36 client-5 rshd[7001]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 19:40:36 client-5 xinetd[3391]: EXIT: shell status=0 pid=7000 duration=0(sec) May 17 19:40:36 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 19:40:56 client-5 xinetd[3391]: START: shell pid=7216 from=192.168.4.5 May 17 19:40:56 client-5 rshd[7217]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 19:41:06 client-5 kernel: Lustre: 16820:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf659e33 to 0xfff98862bf65a3c7 May 17 19:43:00 client-5 kernel: Lustre: 16820:0:(import.c:1190:completed_replay_interpret()) lustre-MDT0000-mdc-ffff8102feea9800: version recovery fails, reconnecting May 17 19:43:00 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff8102feea9800: Connection restored to service lustre-MDT0000 using nid 192.168.4.128@o2ib. May 17 19:43:00 client-5 kernel: Lustre: Skipped 12 previous similar messages May 17 19:43:00 client-5 xinetd[3391]: EXIT: shell status=0 pid=7216 duration=124(sec) May 17 19:43:03 client-5 xinetd[3391]: START: shell pid=7370 from=192.168.4.5 May 17 19:43:03 client-5 rshd[7371]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 19:43:03 client-5 xinetd[3391]: EXIT: shell status=0 pid=7370 duration=0(sec) May 17 19:43:03 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 19:43:23 client-5 xinetd[3391]: START: shell pid=7585 from=192.168.4.5 May 17 19:43:23 client-5 rshd[7586]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 19:43:23 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_statfs operation failed with -107 May 17 19:43:23 client-5 kernel: LustreError: Skipped 13 previous similar messages May 17 19:43:23 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff8102feea9800: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 19:43:23 client-5 kernel: Lustre: 16820:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf65a3c7 to 0xfff98862bf65a969 May 17 19:43:23 client-5 kernel: Lustre: Skipped 9 previous similar messages May 17 19:45:27 client-5 kernel: Lustre: 16820:0:(import.c:1190:completed_replay_interpret()) lustre-MDT0000-mdc-ffff8102feea9800: version recovery fails, reconnecting May 17 19:45:27 client-5 xinetd[3391]: EXIT: shell status=0 pid=7585 duration=124(sec) May 17 19:45:30 client-5 xinetd[3391]: START: shell pid=7739 from=192.168.4.5 May 17 19:45:30 client-5 rshd[7740]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 19:45:30 client-5 xinetd[3391]: EXIT: shell status=0 pid=7739 duration=0(sec) May 17 19:45:30 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 19:45:50 client-5 xinetd[3391]: START: shell pid=7954 from=192.168.4.5 May 17 19:45:50 client-5 rshd[7955]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 19:45:50 client-5 kernel: Lustre: 16820:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf65a969 to 0xfff98862bf65aef6 May 17 19:47:54 client-5 kernel: Lustre: 16820:0:(import.c:1190:completed_replay_interpret()) lustre-MDT0000-mdc-ffff8102feea9800: version recovery fails, reconnecting May 17 19:47:54 client-5 xinetd[3391]: EXIT: shell status=0 pid=7954 duration=124(sec) May 17 19:47:56 client-5 xinetd[3391]: START: shell pid=8113 from=192.168.4.5 May 17 19:47:56 client-5 rshd[8114]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 19:47:56 client-5 xinetd[3391]: EXIT: shell status=0 pid=8113 duration=0(sec) May 17 19:47:56 client-5 xinetd[3391]: START: shell pid=8284 from=192.168.4.5 May 17 19:47:56 client-5 rshd[8286]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-vbr test 7g: rename, {lost}, create ======================================================== 19:47:56 \(1305686876\)");echo XXRETCODE:$?' May 17 19:47:56 client-5 kernel: Lustre: DEBUG MARKER: == replay-vbr test 7g: rename, {lost}, create ======================================================== 19:47:56 (1305686876) May 17 19:47:56 client-5 xinetd[3391]: EXIT: shell status=0 pid=8284 duration=0(sec) May 17 19:47:57 client-5 xinetd[3391]: START: shell pid=8370 from=192.168.4.5 May 17 19:47:57 client-5 rshd[8372]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 19:47:57 client-5 xinetd[3391]: EXIT: shell status=0 pid=8370 duration=0(sec) May 17 19:47:58 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 19:48:17 client-5 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail. May 17 19:48:17 client-5 kernel: LustreError: Skipped 4 previous similar messages May 17 19:48:17 client-5 kernel: Lustre: 16820:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf65aef6 to 0xfff98862bf65b47c May 17 19:48:17 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 19:48:17 client-5 kernel: Lustre: Skipped 4 previous similar messages May 17 19:48:17 client-5 xinetd[3391]: START: shell pid=8588 from=192.168.4.5 May 17 19:48:17 client-5 rshd[8589]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 19:49:48 client-5 xinetd[3391]: EXIT: shell status=0 pid=8588 duration=91(sec) May 17 19:49:51 client-5 xinetd[3391]: START: shell pid=8735 from=192.168.4.5 May 17 19:49:51 client-5 rshd[8736]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 19:49:51 client-5 xinetd[3391]: EXIT: shell status=0 pid=8735 duration=0(sec) May 17 19:49:51 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 19:50:11 client-5 xinetd[3391]: START: shell pid=8954 from=192.168.4.5 May 17 19:50:11 client-5 rshd[8955]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 19:50:12 client-5 kernel: Lustre: 16820:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf65b47c to 0xfff98862bf65ba1e May 17 19:50:30 client-5 kernel: Lustre: 16819:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369106497420385 sent from lustre-MDT0000-mdc-ffff8102feea9800 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305686997] [real_sent 1305686997] [current 1305687030] [deadline 33s] [delay 0s] req@ffff810317686800 x1369106497420385/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 192/192 e 0 to 1 dl 1305687030 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 19:50:30 client-5 kernel: Lustre: 16819:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 38 previous similar messages May 17 19:51:41 client-5 xinetd[3391]: EXIT: shell status=0 pid=8954 duration=90(sec) May 17 19:51:44 client-5 xinetd[3391]: START: shell pid=9104 from=192.168.4.5 May 17 19:51:44 client-5 rshd[9105]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 19:51:45 client-5 xinetd[3391]: EXIT: shell status=0 pid=9104 duration=1(sec) May 17 19:51:45 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 19:52:04 client-5 kernel: Lustre: 16821:0:(import.c:529:import_select_connection()) MGC192.168.4.128@o2ib: tried all connections, increasing latency to 6s May 17 19:52:04 client-5 kernel: Lustre: 16821:0:(import.c:529:import_select_connection()) Skipped 1 previous similar message May 17 19:52:04 client-5 kernel: Lustre: 16820:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf65ba1e to 0xfff98862bf65bfc7 May 17 19:52:04 client-5 xinetd[3391]: START: shell pid=9321 from=192.168.4.5 May 17 19:52:04 client-5 rshd[9322]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 19:53:35 client-5 kernel: Lustre: 16820:0:(client.c:2527:ptlrpc_replay_interpret()) @@@ Version mismatch during replay May 17 19:53:35 client-5 kernel: req@ffff81033f76a000 x1369106497420741/t167503724563(167503724563) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 456/408 e 0 to 0 dl 1305687248 ref 2 fl Interpret:R/ffffffff/ffffffff rc -75/-1 May 17 19:53:35 client-5 kernel: Lustre: 16820:0:(client.c:2527:ptlrpc_replay_interpret()) Skipped 4 previous similar messages May 17 19:53:35 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff810329823000: Connection restored to service lustre-MDT0000 using nid 192.168.4.128@o2ib. May 17 19:53:35 client-5 kernel: Lustre: Skipped 13 previous similar messages May 17 19:54:08 client-5 kernel: Lustre: 16820:0:(import.c:1190:completed_replay_interpret()) lustre-MDT0000-mdc-ffff8102feea9800: version recovery fails, reconnecting May 17 19:54:08 client-5 kernel: LustreError: 167-0: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. May 17 19:54:08 client-5 kernel: LustreError: Skipped 4 previous similar messages May 17 19:54:08 client-5 kernel: LustreError: 9354:0:(lmv_obd.c:1198:lmv_statfs()) can't stat MDS #0 (lustre-MDT0000-mdc-ffff8102feea9800), error -4 May 17 19:54:08 client-5 kernel: LustreError: 9354:0:(lmv_obd.c:1198:lmv_statfs()) Skipped 4 previous similar messages May 17 19:54:08 client-5 kernel: LustreError: 9354:0:(llite_lib.c:1370:ll_statfs_internal()) md_statfs fails: rc = -4 May 17 19:54:08 client-5 kernel: LustreError: 9354:0:(llite_lib.c:1370:ll_statfs_internal()) Skipped 4 previous similar messages May 17 19:54:08 client-5 xinetd[3391]: EXIT: shell status=0 pid=9321 duration=124(sec) May 17 19:54:10 client-5 xinetd[3391]: START: shell pid=9478 from=192.168.4.5 May 17 19:54:11 client-5 rshd[9479]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 19:54:11 client-5 xinetd[3391]: EXIT: shell status=0 pid=9478 duration=1(sec) May 17 19:54:11 client-5 xinetd[3391]: START: shell pid=9650 from=192.168.4.5 May 17 19:54:11 client-5 rshd[9651]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-vbr test 7h: rename, {lost}, unlink ======================================================== 19:54:11 \(1305687251\)");echo XXRETCODE:$?' May 17 19:54:11 client-5 kernel: Lustre: DEBUG MARKER: == replay-vbr test 7h: rename, {lost}, unlink ======================================================== 19:54:11 (1305687251) May 17 19:54:11 client-5 xinetd[3391]: EXIT: shell status=0 pid=9650 duration=0(sec) May 17 19:54:12 client-5 xinetd[3391]: START: shell pid=9736 from=192.168.4.5 May 17 19:54:12 client-5 rshd[9737]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 19:54:12 client-5 xinetd[3391]: EXIT: shell status=0 pid=9736 duration=0(sec) May 17 19:54:12 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 19:54:31 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The obd_ping operation failed with -107 May 17 19:54:31 client-5 kernel: LustreError: Skipped 10 previous similar messages May 17 19:54:31 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff8102feea9800: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 19:54:31 client-5 kernel: Lustre: Skipped 9 previous similar messages May 17 19:54:31 client-5 xinetd[3391]: START: shell pid=9951 from=192.168.4.5 May 17 19:54:31 client-5 rshd[9952]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 19:54:41 client-5 kernel: Lustre: 16820:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf65bfc7 to 0xfff98862bf65c59a May 17 19:56:35 client-5 kernel: Lustre: 16820:0:(import.c:1190:completed_replay_interpret()) lustre-MDT0000-mdc-ffff8102feea9800: version recovery fails, reconnecting May 17 19:56:35 client-5 xinetd[3391]: EXIT: shell status=0 pid=9951 duration=124(sec) May 17 19:56:38 client-5 xinetd[3391]: START: shell pid=10105 from=192.168.4.5 May 17 19:56:38 client-5 rshd[10106]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 19:56:38 client-5 xinetd[3391]: EXIT: shell status=0 pid=10105 duration=0(sec) May 17 19:56:38 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 19:56:58 client-5 xinetd[3391]: START: shell pid=10320 from=192.168.4.5 May 17 19:56:58 client-5 kernel: Lustre: 16820:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf65c59a to 0xfff98862bf65caa9 May 17 19:56:58 client-5 rshd[10323]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 19:59:02 client-5 kernel: Lustre: 16820:0:(import.c:1190:completed_replay_interpret()) lustre-MDT0000-mdc-ffff8102feea9800: version recovery fails, reconnecting May 17 19:59:02 client-5 xinetd[3391]: EXIT: shell status=0 pid=10320 duration=124(sec) May 17 19:59:04 client-5 xinetd[3391]: START: shell pid=10478 from=192.168.4.5 May 17 19:59:04 client-5 rshd[10480]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 19:59:05 client-5 xinetd[3391]: EXIT: shell status=0 pid=10478 duration=1(sec) May 17 19:59:05 client-5 xinetd[3391]: START: shell pid=10654 from=192.168.4.5 May 17 19:59:05 client-5 rshd[10655]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-vbr test 7i: rename, {lost}, rename ======================================================== 19:59:05 \(1305687545\)");echo XXRETCODE:$?' May 17 19:59:05 client-5 kernel: Lustre: DEBUG MARKER: == replay-vbr test 7i: rename, {lost}, rename ======================================================== 19:59:05 (1305687545) May 17 19:59:05 client-5 xinetd[3391]: EXIT: shell status=0 pid=10654 duration=0(sec) May 17 19:59:06 client-5 xinetd[3391]: START: shell pid=10740 from=192.168.4.5 May 17 19:59:06 client-5 rshd[10741]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 19:59:06 client-5 xinetd[3391]: EXIT: shell status=0 pid=10740 duration=0(sec) May 17 19:59:06 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 19:59:25 client-5 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail. May 17 19:59:25 client-5 kernel: LustreError: Skipped 4 previous similar messages May 17 19:59:25 client-5 kernel: Lustre: 16820:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf65caa9 to 0xfff98862bf65d04b May 17 19:59:25 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 19:59:25 client-5 kernel: Lustre: Skipped 4 previous similar messages May 17 19:59:25 client-5 xinetd[3391]: START: shell pid=10957 from=192.168.4.5 May 17 19:59:25 client-5 rshd[10958]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 20:01:29 client-5 kernel: Lustre: 16820:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369106497422354 sent from lustre-MDT0000-mdc-ffff8102feea9800 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305687656] [real_sent 1305687656] [current 1305687689] [deadline 33s] [delay 0s] req@ffff8102de514800 x1369106497422354/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 192/192 e 0 to 1 dl 1305687689 ref 1 fl Rpc:X/ffffffff/ffffffff rc 0/-1 May 17 20:01:29 client-5 kernel: Lustre: 16820:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 38 previous similar messages May 17 20:01:29 client-5 kernel: Lustre: 16820:0:(import.c:1190:completed_replay_interpret()) lustre-MDT0000-mdc-ffff8102feea9800: version recovery fails, reconnecting May 17 20:01:29 client-5 xinetd[3391]: EXIT: shell status=0 pid=10957 duration=124(sec) May 17 20:01:32 client-5 xinetd[3391]: START: shell pid=11115 from=192.168.4.5 May 17 20:01:32 client-5 rshd[11116]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 20:01:32 client-5 xinetd[3391]: EXIT: shell status=0 pid=11115 duration=0(sec) May 17 20:01:33 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 20:01:52 client-5 kernel: Lustre: 16820:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf65d04b to 0xfff98862bf65d602 May 17 20:01:52 client-5 xinetd[3391]: START: shell pid=11332 from=192.168.4.5 May 17 20:01:52 client-5 rshd[11333]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 20:03:56 client-5 kernel: Lustre: 16820:0:(import.c:1190:completed_replay_interpret()) lustre-MDT0000-mdc-ffff8102feea9800: version recovery fails, reconnecting May 17 20:03:56 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff8102feea9800: Connection restored to service lustre-MDT0000 using nid 192.168.4.128@o2ib. May 17 20:03:56 client-5 kernel: Lustre: Skipped 12 previous similar messages May 17 20:03:56 client-5 xinetd[3391]: EXIT: shell status=0 pid=11332 duration=124(sec) May 17 20:03:59 client-5 xinetd[3391]: START: shell pid=11484 from=192.168.4.5 May 17 20:03:59 client-5 rshd[11485]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 20:03:59 client-5 xinetd[3391]: EXIT: shell status=0 pid=11484 duration=0(sec) May 17 20:03:59 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 20:04:19 client-5 kernel: Lustre: 16821:0:(import.c:529:import_select_connection()) MGC192.168.4.128@o2ib: tried all connections, increasing latency to 6s May 17 20:04:19 client-5 kernel: Lustre: 16821:0:(import.c:529:import_select_connection()) Skipped 1 previous similar message May 17 20:04:19 client-5 kernel: Lustre: 16820:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf65d602 to 0xfff98862bf65dc14 May 17 20:04:19 client-5 xinetd[3391]: START: shell pid=11701 from=192.168.4.5 May 17 20:04:19 client-5 rshd[11702]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 20:05:29 client-5 xinetd[3391]: START: login pid=11742 from=10.10.0.1 May 17 20:05:50 client-5 kernel: Lustre: 16820:0:(client.c:2527:ptlrpc_replay_interpret()) @@@ Version mismatch during replay May 17 20:05:50 client-5 kernel: req@ffff81030ac9b400 x1369106497422942/t188978561043(188978561043) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 472/424 e 0 to 0 dl 1305687982 ref 2 fl Interpret:R/ffffffff/ffffffff rc -75/-1 May 17 20:05:50 client-5 kernel: Lustre: 16820:0:(client.c:2527:ptlrpc_replay_interpret()) Skipped 4 previous similar messages May 17 20:06:23 client-5 kernel: Lustre: 16820:0:(import.c:1190:completed_replay_interpret()) lustre-MDT0000-mdc-ffff8102feea9800: version recovery fails, reconnecting May 17 20:06:23 client-5 kernel: LustreError: 167-0: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. May 17 20:06:23 client-5 kernel: LustreError: Skipped 4 previous similar messages May 17 20:06:23 client-5 kernel: LustreError: 11734:0:(lmv_obd.c:1198:lmv_statfs()) can't stat MDS #0 (lustre-MDT0000-mdc-ffff8102feea9800), error -4 May 17 20:06:23 client-5 kernel: LustreError: 11734:0:(lmv_obd.c:1198:lmv_statfs()) Skipped 4 previous similar messages May 17 20:06:23 client-5 kernel: LustreError: 11734:0:(llite_lib.c:1370:ll_statfs_internal()) md_statfs fails: rc = -4 May 17 20:06:23 client-5 kernel: LustreError: 11734:0:(llite_lib.c:1370:ll_statfs_internal()) Skipped 4 previous similar messages May 17 20:06:23 client-5 xinetd[3391]: EXIT: shell status=0 pid=11701 duration=124(sec) May 17 20:06:26 client-5 xinetd[3391]: START: shell pid=11899 from=192.168.4.5 May 17 20:06:26 client-5 rshd[11900]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 20:06:26 client-5 xinetd[3391]: EXIT: shell status=0 pid=11899 duration=0(sec) May 17 20:06:26 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 20:06:45 client-5 xinetd[3391]: START: shell pid=12114 from=192.168.4.5 May 17 20:06:45 client-5 rshd[12115]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 20:06:46 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_statfs operation failed with -107 May 17 20:06:46 client-5 kernel: LustreError: Skipped 9 previous similar messages May 17 20:06:46 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff8102feea9800: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 20:06:46 client-5 kernel: Lustre: Skipped 9 previous similar messages May 17 20:06:56 client-5 kernel: Lustre: 16820:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf65dc14 to 0xfff98862bf65e2b9 May 17 20:08:50 client-5 kernel: Lustre: 16820:0:(import.c:1190:completed_replay_interpret()) lustre-MDT0000-mdc-ffff8102feea9800: version recovery fails, reconnecting May 17 20:08:50 client-5 xinetd[3391]: EXIT: shell status=0 pid=12114 duration=125(sec) May 17 20:08:52 client-5 xinetd[3391]: START: shell pid=12273 from=192.168.4.5 May 17 20:08:52 client-5 rshd[12274]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 20:08:52 client-5 xinetd[3391]: EXIT: shell status=0 pid=12273 duration=0(sec) May 17 20:08:52 client-5 xinetd[3391]: START: shell pid=12444 from=192.168.4.5 May 17 20:08:52 client-5 rshd[12446]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-vbr test 8a: create \| unlink, create shouldn\'t fail ======================================== 20:08:52 \(1305688132\)");echo XXRETCODE:$?' May 17 20:08:52 client-5 kernel: Lustre: DEBUG MARKER: == replay-vbr test 8a: create | unlink, create shouldn't fail ======================================== 20:08:52 (1305688132) May 17 20:08:52 client-5 xinetd[3391]: EXIT: shell status=0 pid=12444 duration=0(sec) May 17 20:08:53 client-5 xinetd[3391]: START: shell pid=12527 from=192.168.4.5 May 17 20:08:53 client-5 rshd[12528]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 20:08:53 client-5 xinetd[3391]: EXIT: shell status=0 pid=12527 duration=0(sec) May 17 20:08:53 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 20:09:07 client-5 xinetd[3391]: EXIT: login status=0 pid=10589 duration=7915(sec) May 17 20:09:13 client-5 xinetd[3391]: START: shell pid=12740 from=192.168.4.5 May 17 20:09:13 client-5 kernel: Lustre: 16820:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf65e2b9 to 0xfff98862bf65e720 May 17 20:09:13 client-5 rshd[12743]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 20:10:44 client-5 xinetd[3391]: EXIT: shell status=0 pid=12740 duration=91(sec) May 17 20:10:44 client-5 xinetd[3391]: START: shell pid=12843 from=192.168.4.5 May 17 20:10:44 client-5 rshd[12844]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 20:10:44 client-5 xinetd[3391]: EXIT: shell status=0 pid=12843 duration=0(sec) May 17 20:10:44 client-5 xinetd[3391]: START: shell pid=13015 from=192.168.4.5 May 17 20:10:44 client-5 rshd[13016]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-vbr test 8b: create \| unlink, create shouldn\'t fail ======================================== 20:10:44 \(1305688244\)");echo XXRETCODE:$?' May 17 20:10:44 client-5 kernel: Lustre: DEBUG MARKER: == replay-vbr test 8b: create | unlink, create shouldn't fail ======================================== 20:10:44 (1305688244) May 17 20:10:44 client-5 xinetd[3391]: EXIT: shell status=0 pid=13015 duration=0(sec) May 17 20:10:45 client-5 xinetd[3391]: START: shell pid=13097 from=192.168.4.5 May 17 20:10:45 client-5 rshd[13098]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 20:10:45 client-5 xinetd[3391]: EXIT: shell status=0 pid=13097 duration=0(sec) May 17 20:10:46 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 20:11:05 client-5 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail. May 17 20:11:05 client-5 kernel: LustreError: Skipped 4 previous similar messages May 17 20:11:05 client-5 kernel: Lustre: 16820:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf65e720 to 0xfff98862bf65eb17 May 17 20:11:05 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 20:11:05 client-5 kernel: Lustre: Skipped 4 previous similar messages May 17 20:11:05 client-5 xinetd[3391]: START: shell pid=13312 from=192.168.4.5 May 17 20:11:05 client-5 rshd[13313]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 20:11:31 client-5 kernel: Lustre: 16819:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369106497424167 sent from lustre-MDT0000-mdc-ffff8102feea9800 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305688258] [real_sent 1305688258] [current 1305688291] [deadline 33s] [delay 0s] req@ffff81033f776000 x1369106497424167/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 192/192 e 0 to 1 dl 1305688291 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 20:11:31 client-5 kernel: Lustre: 16819:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 37 previous similar messages May 17 20:12:36 client-5 xinetd[3391]: EXIT: shell status=0 pid=13312 duration=91(sec) May 17 20:12:36 client-5 xinetd[3391]: START: shell pid=13420 from=192.168.4.5 May 17 20:12:36 client-5 rshd[13421]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 20:12:36 client-5 xinetd[3391]: EXIT: shell status=0 pid=13420 duration=0(sec) May 17 20:12:36 client-5 xinetd[3391]: START: shell pid=13592 from=192.168.4.5 May 17 20:12:36 client-5 rshd[13593]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-vbr test 8c: create \| unlink, create shouldn\'t fail ======================================== 20:12:36 \(1305688356\)");echo XXRETCODE:$?' May 17 20:12:36 client-5 kernel: Lustre: DEBUG MARKER: == replay-vbr test 8c: create | unlink, create shouldn't fail ======================================== 20:12:36 (1305688356) May 17 20:12:36 client-5 xinetd[3391]: EXIT: shell status=0 pid=13592 duration=0(sec) May 17 20:12:37 client-5 xinetd[3391]: START: shell pid=13673 from=192.168.4.5 May 17 20:12:37 client-5 rshd[13675]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 20:12:37 client-5 xinetd[3391]: EXIT: shell status=0 pid=13673 duration=0(sec) May 17 20:12:37 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 20:12:57 client-5 xinetd[3391]: START: shell pid=13887 from=192.168.4.5 May 17 20:12:57 client-5 rshd[13888]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 20:12:57 client-5 kernel: Lustre: 16820:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf65eb17 to 0xfff98862bf65eef9 May 17 20:14:27 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff810329823000: Connection restored to service lustre-MDT0000 using nid 192.168.4.128@o2ib. May 17 20:14:27 client-5 kernel: Lustre: Skipped 13 previous similar messages May 17 20:14:27 client-5 xinetd[3391]: EXIT: shell status=0 pid=13887 duration=90(sec) May 17 20:14:28 client-5 xinetd[3391]: START: shell pid=13990 from=192.168.4.5 May 17 20:14:28 client-5 rshd[13991]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 20:14:28 client-5 xinetd[3391]: EXIT: shell status=0 pid=13990 duration=0(sec) May 17 20:14:28 client-5 xinetd[3391]: START: shell pid=14162 from=192.168.4.5 May 17 20:14:28 client-5 rshd[14163]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-vbr test 10b: 3 clients: some, none, and all reqs replayed ================================= 20:14:28 \(1305688468\)");echo XXRETCODE:$?' May 17 20:14:28 client-5 kernel: Lustre: DEBUG MARKER: == replay-vbr test 10b: 3 clients: some, none, and all reqs replayed ================================= 20:14:28 (1305688468) May 17 20:14:28 client-5 xinetd[3391]: EXIT: shell status=0 pid=14162 duration=0(sec) May 17 20:14:29 client-5 xinetd[3391]: START: shell pid=14289 from=192.168.4.5 May 17 20:14:29 client-5 rshd[14290]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 20:14:29 client-5 xinetd[3391]: EXIT: shell status=0 pid=14289 duration=0(sec) May 17 20:14:30 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 20:14:49 client-5 kernel: Lustre: 16820:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf65eef9 to 0xfff98862bf65f391 May 17 20:14:49 client-5 xinetd[3391]: START: shell pid=14523 from=192.168.4.5 May 17 20:14:50 client-5 rshd[14524]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 20:16:20 client-5 kernel: LustreError: 16820:0:(client.c:2570:ptlrpc_replay_interpret()) @@@ status -116, old was 0 req@ffff810286c35800 x1369106497424797/t210453397512(210453397512) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:23/10 lens 360/424 e 6 to 0 dl 1305688609 ref 2 fl Interpret:R/ffffffff/ffffffff rc -116/-1 May 17 20:16:20 client-5 kernel: Lustre: 16820:0:(client.c:2527:ptlrpc_replay_interpret()) @@@ Version mismatch during replay May 17 20:16:20 client-5 kernel: req@ffff810318ce9c00 x1369106497424815/t210453397522(210453397522) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 456/424 e 0 to 0 dl 1305688613 ref 2 fl Interpret:R/ffffffff/ffffffff rc -75/-1 May 17 20:16:20 client-5 kernel: Lustre: 16820:0:(client.c:2527:ptlrpc_replay_interpret()) Skipped 1 previous similar message May 17 20:16:53 client-5 kernel: Lustre: 16820:0:(import.c:1190:completed_replay_interpret()) lustre-MDT0000-mdc-ffff8102feea9800: version recovery fails, reconnecting May 17 20:16:53 client-5 kernel: LustreError: 167-0: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. May 17 20:16:53 client-5 kernel: LustreError: Skipped 1 previous similar message May 17 20:16:53 client-5 kernel: LustreError: 14556:0:(lmv_obd.c:1198:lmv_statfs()) can't stat MDS #0 (lustre-MDT0000-mdc-ffff8102feea9800), error -4 May 17 20:16:53 client-5 kernel: LustreError: 14556:0:(lmv_obd.c:1198:lmv_statfs()) Skipped 1 previous similar message May 17 20:16:53 client-5 kernel: LustreError: 14556:0:(llite_lib.c:1370:ll_statfs_internal()) md_statfs fails: rc = -4 May 17 20:16:53 client-5 kernel: LustreError: 14556:0:(llite_lib.c:1370:ll_statfs_internal()) Skipped 1 previous similar message May 17 20:16:53 client-5 xinetd[3391]: EXIT: shell status=0 pid=14523 duration=124(sec) May 17 20:16:55 client-5 xinetd[3391]: START: shell pid=14689 from=192.168.4.5 May 17 20:16:55 client-5 rshd[14690]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 20:16:55 client-5 xinetd[3391]: EXIT: shell status=0 pid=14689 duration=0(sec) May 17 20:16:56 client-5 xinetd[3391]: START: shell pid=14861 from=192.168.4.5 May 17 20:16:56 client-5 rshd[14862]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-vbr test 11a: concurrent creates don\'t affect each other =================================== 20:16:56 \(1305688616\)");echo XXRETCODE:$?' May 17 20:16:56 client-5 kernel: Lustre: DEBUG MARKER: == replay-vbr test 11a: concurrent creates don't affect each other =================================== 20:16:56 (1305688616) May 17 20:16:56 client-5 xinetd[3391]: EXIT: shell status=0 pid=14861 duration=0(sec) May 17 20:16:56 client-5 xinetd[3391]: START: shell pid=14935 from=192.168.4.5 May 17 20:16:56 client-5 rshd[14936]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 20:16:57 client-5 xinetd[3391]: EXIT: shell status=0 pid=14935 duration=1(sec) May 17 20:16:57 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 20:17:16 client-5 kernel: Lustre: 16821:0:(import.c:529:import_select_connection()) MGC192.168.4.128@o2ib: tried all connections, increasing latency to 6s May 17 20:17:16 client-5 kernel: Lustre: 16821:0:(import.c:529:import_select_connection()) Skipped 1 previous similar message May 17 20:17:16 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The obd_ping operation failed with -107 May 17 20:17:16 client-5 kernel: Lustre: 16820:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf65f391 to 0xfff98862bf660ca4 May 17 20:17:16 client-5 kernel: LustreError: Skipped 9 previous similar messages May 17 20:17:16 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff8102feea9800: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 20:17:16 client-5 kernel: Lustre: Skipped 9 previous similar messages May 17 20:17:16 client-5 xinetd[3391]: START: shell pid=15147 from=192.168.4.5 May 17 20:17:16 client-5 rshd[15148]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 20:18:47 client-5 xinetd[3391]: EXIT: shell status=0 pid=15147 duration=91(sec) May 17 20:18:47 client-5 xinetd[3391]: START: shell pid=15252 from=192.168.4.5 May 17 20:18:47 client-5 rshd[15253]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 20:18:47 client-5 xinetd[3391]: EXIT: shell status=0 pid=15252 duration=0(sec) May 17 20:18:48 client-5 xinetd[3391]: START: shell pid=15424 from=192.168.4.5 May 17 20:18:48 client-5 rshd[15425]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-vbr test 11b: concurrent creates and unlinks don\'t affect each other ======================= 20:18:48 \(1305688728\)");echo XXRETCODE:$?' May 17 20:18:48 client-5 kernel: Lustre: DEBUG MARKER: == replay-vbr test 11b: concurrent creates and unlinks don't affect each other ======================= 20:18:48 (1305688728) May 17 20:18:48 client-5 xinetd[3391]: EXIT: shell status=0 pid=15424 duration=0(sec) May 17 20:18:49 client-5 xinetd[3391]: START: shell pid=15503 from=192.168.4.5 May 17 20:18:49 client-5 rshd[15504]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "f=/mnt/lustre/fsa-\$(hostname); mcreate \$f; rm \$f");echo XXRETCODE:$?' May 17 20:18:49 client-5 xinetd[3391]: EXIT: shell status=0 pid=15503 duration=0(sec) May 17 20:18:49 client-5 kernel: Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 May 17 20:19:08 client-5 xinetd[3391]: START: shell pid=15713 from=192.168.4.5 May 17 20:19:08 client-5 rshd[15714]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 20:19:18 client-5 kernel: Lustre: 16820:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf660ca4 to 0xfff98862bf66433f May 17 20:20:39 client-5 xinetd[3391]: EXIT: shell status=0 pid=15713 duration=91(sec) May 17 20:20:39 client-5 xinetd[3391]: START: shell pid=15816 from=192.168.4.5 May 17 20:20:39 client-5 rshd[15817]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 20:20:39 client-5 xinetd[3391]: EXIT: shell status=0 pid=15816 duration=0(sec) May 17 20:20:40 client-5 xinetd[3391]: START: shell pid=15979 from=192.168.4.5 May 17 20:20:40 client-5 rshd[15980]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark SKIP: replay-vbr test_12a skipping ALWAYS excluded test 12a");echo XXRETCODE:$?' May 17 20:20:40 client-5 kernel: Lustre: DEBUG MARKER: SKIP: replay-vbr test_12a skipping ALWAYS excluded test 12a May 17 20:20:40 client-5 xinetd[3391]: EXIT: shell status=0 pid=15979 duration=0(sec) May 17 20:20:40 client-5 xinetd[3391]: START: shell pid=16029 from=192.168.4.5 May 17 20:20:40 client-5 rshd[16030]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c " running=\$(mount | grep -c /mnt/lustre' '); rc=0; if [ \$running -eq 0 ] ; then mkdir -p /mnt/lustre; mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre; rc=\$?; fi; exit \$rc");echo XXRETCODE:$?' May 17 20:20:40 client-5 xinetd[3391]: EXIT: shell status=0 pid=16029 duration=0(sec) May 17 20:20:40 client-5 xinetd[3391]: START: shell pid=16074 from=192.168.4.5 May 17 20:20:40 client-5 rshd[16075]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mount | grep -w /mnt/lustre");echo XXRETCODE:$?' May 17 20:20:40 client-5 xinetd[3391]: EXIT: shell status=0 pid=16074 duration=0(sec) May 17 20:20:40 client-5 xinetd[3391]: START: shell pid=16118 from=192.168.4.5 May 17 20:20:40 client-5 rshd[16119]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param debug=\"0x33f0406\"; lctl set_param subsystem_debug=\"0xffb7e3ff\"; lctl set_param debug_mb=32;");echo XXRETCODE:$?' May 17 20:20:40 client-5 kernel: Lustre: 16153:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 20:20:40 client-5 kernel: Lustre: 16153:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 7 previous similar messages May 17 20:20:40 client-5 xinetd[3391]: EXIT: shell status=0 pid=16118 duration=0(sec) May 17 20:20:40 client-5 xinetd[3391]: START: shell pid=16216 from=192.168.4.5 May 17 20:20:40 client-5 rshd[16217]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == replay-vbr replay-vbr.sh test complete, duration 7137 sec ========================================= 20:20:40 \(1305688840\)");echo XXRETCODE:$?' May 17 20:20:40 client-5 kernel: Lustre: DEBUG MARKER: == replay-vbr replay-vbr.sh test complete, duration 7137 sec ========================================= 20:20:40 (1305688840) May 17 20:20:40 client-5 xinetd[3391]: EXIT: shell status=0 pid=16216 duration=0(sec) May 17 20:20:58 client-5 kernel: Lustre: 16820:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf66433f to 0xfff98862bf667dd8 May 17 20:20:59 client-5 xinetd[3391]: START: shell pid=16384 from=192.168.4.5 May 17 20:20:59 client-5 rshd[16385]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 20:21:03 client-5 xinetd[3391]: EXIT: shell status=0 pid=16384 duration=4(sec) May 17 20:21:03 client-5 xinetd[3391]: START: shell pid=16428 from=192.168.4.5 May 17 20:21:03 client-5 rshd[16429]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:-f; lsof -t /mnt/lustre || need_kill=no; if [ x-f != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount -f /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre is still busy, wait one second && sleep 1; umount -f /mnt/lustre; fi fi");echo XXRETCODE:$?' May 17 20:21:03 client-5 kernel: Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request May 17 20:21:03 client-5 kernel: Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request May 17 20:21:03 client-5 kernel: Lustre: client ffff8102feea9800 umount complete May 17 20:21:03 client-5 xinetd[3391]: EXIT: shell status=0 pid=16428 duration=0(sec) May 17 20:21:03 client-5 xinetd[3391]: START: shell pid=16479 from=192.168.4.5 May 17 20:21:03 client-5 rshd[16480]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:-f; lsof -t /mnt/lustre2 || need_kill=no; if [ x-f != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre2 | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount -f /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre2 is still busy, wait one second && sleep 1; umount -f /mnt/lustre2; fi fi");echo XXRETCODE:$?' May 17 20:21:04 client-5 kernel: LustreError: 16521:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway May 17 20:21:04 client-5 kernel: LustreError: 16521:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 May 17 20:21:04 client-5 kernel: Lustre: client ffff810329823000 umount complete May 17 20:21:04 client-5 xinetd[3391]: EXIT: shell status=0 pid=16479 duration=1(sec) May 17 20:21:31 client-5 kernel: Lustre: Acceptor stopping May 17 20:21:32 client-5 kernel: Lustre: Removed LNI 10.10.4.5@tcp May 17 20:21:34 client-5 kernel: Lustre: Removed LNI 192.168.4.5@o2ib May 17 20:21:35 client-5 kernel: Lustre: 17142:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 20:21:35 client-5 kernel: Lustre: 17143:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 20:21:35 client-5 kernel: Lustre: OBD class driver, http://www.lustre.org/ May 17 20:21:35 client-5 kernel: Lustre: Lustre Version: 2.0.60 May 17 20:21:35 client-5 kernel: Lustre: Build Version: jenkins-g985ac5b-PRISTINE-2.6.18-194.17.1.el5 May 17 20:21:35 client-5 kernel: Lustre: Lustre LU module (ffffffff888eb1c0). May 17 20:21:35 client-5 kernel: Lustre: Register global MR array, MR size: 0xffffffffffffffff, array size: 1 May 17 20:21:35 client-5 kernel: Lustre: Added LNI 192.168.4.5@o2ib [8/64/0/180] May 17 20:21:35 client-5 kernel: Lustre: Added LNI 10.10.4.5@tcp [8/256/0/180] May 17 20:21:35 client-5 kernel: Lustre: Accept all, port 7988 May 17 20:21:35 client-5 kernel: Lustre: Lustre OSC module (ffffffff88b3c6c0). May 17 20:21:35 client-5 kernel: Lustre: Lustre LOV module (ffffffff88bab540). May 17 20:21:36 client-5 kernel: Lustre: Lustre client module (ffffffff88c643a0). May 17 20:21:36 client-5 xinetd[3391]: START: shell pid=17544 from=192.168.4.5 May 17 20:21:36 client-5 rshd[17545]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark -----============= acceptance-small: insanity ============----- Tue May 17 20:21:35 PDT 2011");echo XXRETCODE:$?' May 17 20:21:36 client-5 kernel: Lustre: DEBUG MARKER: -----============= acceptance-small: insanity ============----- Tue May 17 20:21:35 PDT 2011 May 17 20:21:36 client-5 xinetd[3391]: EXIT: shell status=0 pid=17544 duration=0(sec) May 17 20:21:36 client-5 xinetd[3391]: START: shell pid=17685 from=192.168.4.5 May 17 20:21:36 client-5 rshd[17688]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark excepting tests: 10");echo XXRETCODE:$?' May 17 20:21:36 client-5 kernel: Lustre: DEBUG MARKER: excepting tests: 10 May 17 20:21:36 client-5 xinetd[3391]: EXIT: shell status=0 pid=17685 duration=0(sec) May 17 20:21:36 client-5 xinetd[3391]: START: shell pid=17777 from=192.168.4.5 May 17 20:21:36 client-5 rshd[17778]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:; lsof -t /mnt/lustre || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre is still busy, wait one second && sleep 1; umount /mnt/lustre; fi fi");echo XXRETCODE:$?' May 17 20:21:36 client-5 xinetd[3391]: EXIT: shell status=0 pid=17777 duration=0(sec) May 17 20:21:36 client-5 xinetd[3391]: START: shell pid=17821 from=192.168.4.5 May 17 20:21:36 client-5 rshd[17822]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:; lsof -t /mnt/lustre2 || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre2 | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre2 is still busy, wait one second && sleep 1; umount /mnt/lustre2; fi fi");echo XXRETCODE:$?' May 17 20:21:36 client-5 xinetd[3391]: EXIT: shell status=0 pid=17821 duration=0(sec) May 17 20:21:38 client-5 kernel: Lustre: 18024:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 20:22:36 client-5 xinetd[3391]: START: shell pid=18460 from=192.168.4.5 May 17 20:22:36 client-5 rshd[18461]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); mpts=\$(mount | grep -w -c /mnt/lustre); if [ \$running -ne \$mpts ]; then echo \$(hostname) env are INSANE!; exit 1; fi");echo XXRETCODE:$?' May 17 20:22:36 client-5 xinetd[3391]: EXIT: shell status=0 pid=18460 duration=0(sec) May 17 20:22:36 client-5 xinetd[3391]: START: shell pid=18507 from=192.168.4.5 May 17 20:22:36 client-5 rshd[18508]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); mpts=\$(mount | grep -w -c /mnt/lustre2); if [ \$running -ne \$mpts ]; then echo \$(hostname) env are INSANE!; exit 1; fi");echo XXRETCODE:$?' May 17 20:22:36 client-5 xinetd[3391]: EXIT: shell status=0 pid=18507 duration=0(sec) May 17 20:22:36 client-5 kernel: Lustre: 18550:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 20:22:36 client-5 kernel: Lustre: 18550:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message May 17 20:22:40 client-5 kernel: Lustre: 18887:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 20:22:40 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 20:22:40 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 20:22:40 client-5 kernel: Lustre: 18887:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff8102f9b5f800->192.168.4.128@o2ib netid 50000: select flavor null May 17 20:22:40 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 20:22:45 client-5 kernel: Lustre: 18887:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-OST0000-osc-ffff8102f9b5f800->192.168.4.129@o2ib netid 50000: select flavor null May 17 20:22:45 client-5 kernel: Lustre: Client lustre-client has started May 17 20:22:45 client-5 kernel: Lustre: 18904:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 20:22:45 client-5 kernel: Lustre: 18904:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message May 17 20:22:45 client-5 xinetd[3391]: START: shell pid=18914 from=192.168.4.5 May 17 20:22:45 client-5 rshd[18915]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c " running=\$(mount | grep -c /mnt/lustre' '); rc=0; if [ \$running -eq 0 ] ; then mkdir -p /mnt/lustre; mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre; rc=\$?; fi; exit \$rc");echo XXRETCODE:$?' May 17 20:22:45 client-5 xinetd[3391]: EXIT: shell status=0 pid=18914 duration=0(sec) May 17 20:22:45 client-5 xinetd[3391]: START: shell pid=18959 from=192.168.4.5 May 17 20:22:45 client-5 rshd[18960]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mount | grep -w /mnt/lustre");echo XXRETCODE:$?' May 17 20:22:46 client-5 xinetd[3391]: EXIT: shell status=0 pid=18959 duration=1(sec) May 17 20:22:46 client-5 xinetd[3391]: START: shell pid=19003 from=192.168.4.5 May 17 20:22:46 client-5 rshd[19004]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param debug=\"0x33f0406\"; lctl set_param subsystem_debug=\"0xffb7e3ff\"; lctl set_param debug_mb=32;");echo XXRETCODE:$?' May 17 20:22:46 client-5 xinetd[3391]: EXIT: shell status=0 pid=19003 duration=0(sec) May 17 20:22:46 client-5 xinetd[3391]: START: shell pid=19173 from=192.168.4.5 May 17 20:22:46 client-5 rshd[19174]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark Using TIMEOUT=20");echo XXRETCODE:$?' May 17 20:22:46 client-5 kernel: Lustre: DEBUG MARKER: Using TIMEOUT=20 May 17 20:22:46 client-5 xinetd[3391]: EXIT: shell status=0 pid=19173 duration=0(sec) May 17 20:22:46 client-5 kernel: LustreError: 19227:0:(quota_ctl.c:328:client_quota_ctl()) ptlrpc_queue_wait failed, rc: -114 May 17 20:22:51 client-5 xinetd[3391]: START: shell pid=19247 from=192.168.4.5 May 17 20:22:51 client-5 rshd[19248]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "cat /proc/mounts");echo XXRETCODE:$?' May 17 20:22:51 client-5 xinetd[3391]: EXIT: shell status=0 pid=19247 duration=0(sec) May 17 20:22:52 client-5 xinetd[3391]: START: shell pid=19518 from=192.168.4.5 May 17 20:22:52 client-5 rshd[19521]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == insanity test 0: Fail all nodes, independently ==================================================== 20:22:52 \(1305688972\)");echo XXRETCODE:$?' May 17 20:22:53 client-5 kernel: Lustre: DEBUG MARKER: == insanity test 0: Fail all nodes, independently ==================================================== 20:22:52 (1305688972) May 17 20:22:53 client-5 xinetd[3391]: EXIT: shell status=0 pid=19518 duration=1(sec) May 17 20:23:02 client-5 kernel: Lustre: 17250:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369114038763556 sent from MGC192.168.4.128@o2ib to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305688975] [real_sent 1305688975] [current 1305688982] [deadline 7s] [delay 0s] req@ffff810317e2e400 x1369114038763556/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 192/192 e 0 to 1 dl 1305688982 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 20:23:02 client-5 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail. May 17 20:23:07 client-5 kernel: Lustre: 17250:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369114038763564 sent from lustre-MDT0000-mdc-ffff8102f9b5f800 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305688980] [real_sent 1305688980] [current 1305688987] [deadline 7s] [delay 0s] req@ffff8102ffaa4000 x1369114038763564/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 192/192 e 0 to 1 dl 1305688987 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 20:23:07 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff8102f9b5f800: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 20:23:08 client-5 kernel: Lustre: 17251:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369114038763571 sent from MGC192.168.4.128@o2ib to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305688982] [real_sent 1305688982] [current 1305688988] [deadline 6s] [delay 0s] req@ffff8102f9b5f400 x1369114038763571/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 368/392 e 0 to 1 dl 1305688988 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 20:23:11 client-5 xinetd[3391]: START: shell pid=19673 from=192.168.4.5 May 17 20:23:11 client-5 rshd[19674]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 20:23:12 client-5 kernel: Lustre: 17252:0:(import.c:529:import_select_connection()) MGC192.168.4.128@o2ib: tried all connections, increasing latency to 6s May 17 20:23:12 client-5 kernel: Lustre: 17251:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf667f8a to 0xfff98862bf6681d6 May 17 20:23:12 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 20:23:12 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Connection restored to service MGS using nid 192.168.4.128@o2ib. May 17 20:23:13 client-5 kernel: Lustre: 17251:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369114038763579 sent from lustre-MDT0000-mdc-ffff8102f9b5f800 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305688987] [real_sent 1305688987] [current 1305688993] [deadline 6s] [delay 0s] req@ffff8102542f2c00 x1369114038763579/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 368/392 e 0 to 1 dl 1305688993 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 20:23:17 client-5 kernel: Lustre: 17252:0:(import.c:529:import_select_connection()) lustre-MDT0000-mdc-ffff8102f9b5f800: tried all connections, increasing latency to 6s May 17 20:23:17 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff8102f9b5f800: Connection restored to service lustre-MDT0000 using nid 192.168.4.128@o2ib. May 17 20:23:17 client-5 xinetd[3391]: EXIT: shell status=0 pid=19673 duration=6(sec) May 17 20:23:27 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.129@o2ib. The obd_ping operation failed with -107 May 17 20:23:27 client-5 kernel: Lustre: lustre-OST0000-osc-ffff8102f9b5f800: Connection to service lustre-OST0000 via nid 192.168.4.129@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 20:23:31 client-5 xinetd[3391]: START: shell pid=19829 from=192.168.4.5 May 17 20:23:31 client-5 rshd[19831]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 20:23:37 client-5 kernel: Lustre: 17252:0:(import.c:529:import_select_connection()) lustre-OST0000-osc-ffff8102f9b5f800: tried all connections, increasing latency to 6s May 17 20:23:37 client-5 kernel: Lustre: lustre-OST0000-osc-ffff8102f9b5f800: Connection restored to service lustre-OST0000 using nid 192.168.4.129@o2ib. May 17 20:23:37 client-5 xinetd[3391]: EXIT: shell status=0 pid=19829 duration=6(sec) May 17 20:23:47 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.129@o2ib. The obd_ping operation failed with -107 May 17 20:23:47 client-5 kernel: LustreError: Skipped 1 previous similar message May 17 20:23:47 client-5 kernel: Lustre: lustre-OST0001-osc-ffff8102f9b5f800: Connection to service lustre-OST0001 via nid 192.168.4.129@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 20:23:51 client-5 xinetd[3391]: START: shell pid=19986 from=192.168.4.5 May 17 20:23:51 client-5 rshd[19987]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 20:23:57 client-5 kernel: Lustre: 17252:0:(import.c:529:import_select_connection()) lustre-OST0001-osc-ffff8102f9b5f800: tried all connections, increasing latency to 6s May 17 20:23:57 client-5 kernel: Lustre: lustre-OST0001-osc-ffff8102f9b5f800: Connection restored to service lustre-OST0001 using nid 192.168.4.129@o2ib. May 17 20:23:57 client-5 xinetd[3391]: EXIT: shell status=0 pid=19986 duration=6(sec) May 17 20:24:07 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.129@o2ib. The obd_ping operation failed with -107 May 17 20:24:07 client-5 kernel: LustreError: Skipped 1 previous similar message May 17 20:24:07 client-5 kernel: Lustre: lustre-OST0002-osc-ffff8102f9b5f800: Connection to service lustre-OST0002 via nid 192.168.4.129@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 20:24:11 client-5 xinetd[3391]: START: shell pid=20138 from=192.168.4.5 May 17 20:24:11 client-5 rshd[20139]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 20:24:17 client-5 kernel: Lustre: 17252:0:(import.c:529:import_select_connection()) lustre-OST0002-osc-ffff8102f9b5f800: tried all connections, increasing latency to 6s May 17 20:24:17 client-5 kernel: Lustre: lustre-OST0002-osc-ffff8102f9b5f800: Connection restored to service lustre-OST0002 using nid 192.168.4.129@o2ib. May 17 20:24:17 client-5 xinetd[3391]: EXIT: shell status=0 pid=20138 duration=6(sec) May 17 20:24:27 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.129@o2ib. The obd_ping operation failed with -107 May 17 20:24:27 client-5 kernel: LustreError: Skipped 1 previous similar message May 17 20:24:27 client-5 kernel: Lustre: lustre-OST0003-osc-ffff8102f9b5f800: Connection to service lustre-OST0003 via nid 192.168.4.129@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 20:24:34 client-5 xinetd[3391]: START: shell pid=20290 from=192.168.4.5 May 17 20:24:34 client-5 rshd[20291]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 20:24:37 client-5 kernel: Lustre: 17252:0:(import.c:529:import_select_connection()) lustre-OST0003-osc-ffff8102f9b5f800: tried all connections, increasing latency to 6s May 17 20:24:37 client-5 kernel: Lustre: lustre-OST0003-osc-ffff8102f9b5f800: Connection restored to service lustre-OST0003 using nid 192.168.4.129@o2ib. May 17 20:24:37 client-5 xinetd[3391]: EXIT: shell status=0 pid=20290 duration=3(sec) May 17 20:24:42 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.129@o2ib. The obd_ping operation failed with -107 May 17 20:24:42 client-5 kernel: LustreError: Skipped 1 previous similar message May 17 20:24:42 client-5 kernel: Lustre: lustre-OST0004-osc-ffff8102f9b5f800: Connection to service lustre-OST0004 via nid 192.168.4.129@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 20:24:51 client-5 xinetd[3391]: START: shell pid=20442 from=192.168.4.5 May 17 20:24:51 client-5 rshd[20443]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 20:24:52 client-5 kernel: Lustre: 17252:0:(import.c:529:import_select_connection()) lustre-OST0004-osc-ffff8102f9b5f800: tried all connections, increasing latency to 6s May 17 20:24:55 client-5 kernel: Lustre: lustre-OST0004-osc-ffff8102f9b5f800: Connection restored to service lustre-OST0004 using nid 192.168.4.129@o2ib. May 17 20:24:55 client-5 xinetd[3391]: EXIT: shell status=0 pid=20442 duration=4(sec) May 17 20:24:57 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.129@o2ib. The obd_ping operation failed with -107 May 17 20:24:57 client-5 kernel: LustreError: Skipped 1 previous similar message May 17 20:24:57 client-5 kernel: Lustre: lustre-OST0005-osc-ffff8102f9b5f800: Connection to service lustre-OST0005 via nid 192.168.4.129@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 20:25:09 client-5 xinetd[3391]: START: shell pid=20598 from=192.168.4.5 May 17 20:25:09 client-5 rshd[20599]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 20:25:12 client-5 kernel: Lustre: 17252:0:(import.c:529:import_select_connection()) lustre-OST0005-osc-ffff8102f9b5f800: tried all connections, increasing latency to 11s May 17 20:25:12 client-5 kernel: Lustre: 17252:0:(import.c:529:import_select_connection()) Skipped 1 previous similar message May 17 20:25:15 client-5 kernel: Lustre: lustre-OST0005-osc-ffff8102f9b5f800: Connection restored to service lustre-OST0005 using nid 192.168.4.129@o2ib. May 17 20:25:15 client-5 xinetd[3391]: EXIT: shell status=0 pid=20598 duration=6(sec) May 17 20:25:15 client-5 xinetd[3391]: START: shell pid=20686 from=192.168.4.5 May 17 20:25:15 client-5 rshd[20689]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 20:25:15 client-5 xinetd[3391]: EXIT: shell status=0 pid=20686 duration=0(sec) May 17 20:25:15 client-5 xinetd[3391]: START: shell pid=20860 from=192.168.4.5 May 17 20:25:15 client-5 rshd[20861]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == insanity test 1: MDS\/MDS failure ================================================================== 20:25:15 \(1305689115\)");echo XXRETCODE:$?' May 17 20:25:15 client-5 kernel: Lustre: DEBUG MARKER: == insanity test 1: MDS/MDS failure ================================================================== 20:25:15 (1305689115) May 17 20:25:15 client-5 xinetd[3391]: EXIT: shell status=0 pid=20860 duration=0(sec) May 17 20:25:15 client-5 xinetd[3391]: START: shell pid=20951 from=192.168.4.5 May 17 20:25:15 client-5 rshd[20952]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 20:25:15 client-5 xinetd[3391]: EXIT: shell status=0 pid=20951 duration=0(sec) May 17 20:25:16 client-5 xinetd[3391]: START: shell pid=21124 from=192.168.4.5 May 17 20:25:16 client-5 rshd[21125]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == insanity test 2: Second Failure Mode: MDS\/OST Tue May 17 20:25:15 PDT 2011 ======================== 20:25:15 \(1305689115\)");echo XXRETCODE:$?' May 17 20:25:16 client-5 kernel: Lustre: DEBUG MARKER: == insanity test 2: Second Failure Mode: MDS/OST Tue May 17 20:25:15 PDT 2011 ======================== 20:25:15 (1305689115) May 17 20:25:16 client-5 xinetd[3391]: EXIT: shell status=0 pid=21124 duration=0(sec) May 17 20:25:17 client-5 xinetd[3391]: START: shell pid=21167 from=192.168.4.5 May 17 20:25:17 client-5 rshd[21168]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 20:25:17 client-5 xinetd[3391]: EXIT: shell status=0 pid=21167 duration=0(sec) May 17 20:25:34 client-5 kernel: Lustre: 17250:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369114038763772 sent from MGC192.168.4.128@o2ib to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305689122] [real_sent 1305689122] [current 1305689134] [deadline 12s] [delay 0s] req@ffff8102fa514400 x1369114038763772/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 192/192 e 0 to 1 dl 1305689134 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 20:25:34 client-5 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail. May 17 20:25:45 client-5 xinetd[3391]: START: shell pid=21256 from=192.168.4.5 May 17 20:25:45 client-5 rshd[21257]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 20:25:45 client-5 kernel: Lustre: 17251:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369114038763796 sent from MGC192.168.4.128@o2ib to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305689134] [real_sent 1305689134] [current 1305689145] [deadline 11s] [delay 0s] req@ffff81031f564c00 x1369114038763796/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 368/392 e 0 to 1 dl 1305689145 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 20:25:49 client-5 kernel: Lustre: 17252:0:(import.c:529:import_select_connection()) MGC192.168.4.128@o2ib: tried all connections, increasing latency to 11s May 17 20:25:49 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.129@o2ib. The obd_ping operation failed with -107 May 17 20:25:49 client-5 kernel: LustreError: Skipped 2 previous similar messages May 17 20:25:49 client-5 kernel: Lustre: lustre-OST0000-osc-ffff8102f9b5f800: Connection to service lustre-OST0000 via nid 192.168.4.129@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 20:25:53 client-5 kernel: Lustre: 17250:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369114038763773 sent from lustre-MDT0000-mdc-ffff8102f9b5f800 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305689122] [real_sent 1305689122] [current 1305689153] [deadline 31s] [delay 0s] req@ffff8102fe6aec00 x1369114038763773/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 192/192 e 0 to 1 dl 1305689153 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 20:26:10 client-5 kernel: Lustre: 17250:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369114038763797 sent from lustre-MDT0000-mdc-ffff8102f9b5f800 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305689139] [real_sent 1305689139] [current 1305689170] [deadline 31s] [delay 0s] req@ffff8102f457dc00 x1369114038763797/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 192/192 e 0 to 1 dl 1305689170 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 20:26:10 client-5 kernel: Lustre: 17250:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 4 previous similar messages May 17 20:26:23 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.129@o2ib. The ost_connect operation failed with -19 May 17 20:26:23 client-5 kernel: LustreError: Skipped 6 previous similar messages May 17 20:26:44 client-5 kernel: Lustre: lustre-OST0000-osc-ffff8102f9b5f800: Connection restored to service lustre-OST0000 using nid 192.168.4.129@o2ib. May 17 20:26:49 client-5 kernel: Lustre: 17251:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369114038763859 sent from lustre-MDT0000-mdc-ffff8102f9b5f800 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305689188] [real_sent 1305689188] [current 1305689209] [deadline 21s] [delay 0s] req@ffff81031b4b0800 x1369114038763859/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 368/392 e 0 to 1 dl 1305689209 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 20:26:49 client-5 kernel: Lustre: 17251:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 4 previous similar messages May 17 20:26:53 client-5 kernel: Lustre: 17252:0:(import.c:529:import_select_connection()) lustre-MDT0000-mdc-ffff8102f9b5f800: tried all connections, increasing latency to 21s May 17 20:26:53 client-5 kernel: Lustre: 17252:0:(import.c:529:import_select_connection()) Skipped 13 previous similar messages May 17 20:26:53 client-5 xinetd[3391]: EXIT: shell status=0 pid=21256 duration=68(sec) May 17 20:26:54 client-5 xinetd[3391]: START: shell pid=21449 from=192.168.4.5 May 17 20:26:54 client-5 rshd[21450]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 20:26:55 client-5 xinetd[3391]: EXIT: shell status=0 pid=21449 duration=1(sec) May 17 20:26:55 client-5 xinetd[3391]: START: shell pid=21539 from=192.168.4.5 May 17 20:26:55 client-5 rshd[21540]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 20:26:55 client-5 xinetd[3391]: EXIT: shell status=0 pid=21539 duration=0(sec) May 17 20:26:55 client-5 xinetd[3391]: START: shell pid=21712 from=192.168.4.5 May 17 20:26:55 client-5 rshd[21713]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == insanity test 3: Thirdb Failure Mode: MDS\/CLIENT Tue May 17 20:26:55 PDT 2011 ===================== 20:26:55 \(1305689215\)");echo XXRETCODE:$?' May 17 20:26:55 client-5 kernel: Lustre: DEBUG MARKER: == insanity test 3: Thirdb Failure Mode: MDS/CLIENT Tue May 17 20:26:55 PDT 2011 ===================== 20:26:55 (1305689215) May 17 20:26:55 client-5 xinetd[3391]: EXIT: shell status=0 pid=21712 duration=0(sec) May 17 20:27:13 client-5 xinetd[3391]: START: shell pid=21867 from=192.168.4.5 May 17 20:27:13 client-5 rshd[21868]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 20:27:13 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff8102f9b5f800: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 20:27:13 client-5 kernel: Lustre: Skipped 1 previous similar message May 17 20:27:13 client-5 xinetd[3391]: EXIT: shell status=0 pid=21867 duration=0(sec) May 17 20:27:14 client-5 xinetd[3391]: START: shell pid=21907 from=192.168.4.5 May 17 20:27:15 client-5 rshd[21908]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 20:27:15 client-5 xinetd[3391]: EXIT: shell status=0 pid=21907 duration=1(sec) May 17 20:27:15 client-5 xinetd[3391]: START: shell pid=21999 from=192.168.4.5 May 17 20:27:15 client-5 rshd[22000]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark Request clients to fail: 2. Num of clients to fail: 1, already failed: 0");echo XXRETCODE:$?' May 17 20:27:15 client-5 kernel: Lustre: DEBUG MARKER: Request clients to fail: 2. Num of clients to fail: 1, already failed: 0 May 17 20:27:15 client-5 xinetd[3391]: EXIT: shell status=0 pid=21999 duration=0(sec) May 17 20:27:15 client-5 xinetd[3391]: START: shell pid=22038 from=192.168.4.5 May 17 20:27:15 client-5 rshd[22039]: root@client-5-ib.lab.whamcloud.com as root: cmd='mkdir /mnt/lustre/d0.insanity/client-5-ib && ls -l /mnt/lustre/d0.insanity/client-5-ib;echo XXRETCODE:$?' May 17 20:27:15 client-5 xinetd[3391]: EXIT: shell status=0 pid=22038 duration=0(sec) May 17 20:27:24 client-5 xinetd[3391]: START: shell pid=22095 from=192.168.4.5 May 17 20:27:24 client-5 rshd[22096]: root@client-5-ib.lab.whamcloud.com as root: cmd='rmdir /mnt/lustre/d0.insanity/client-5-ib;echo XXRETCODE:$?' May 17 20:27:25 client-5 xinetd[3391]: EXIT: shell status=0 pid=22095 duration=1(sec) May 17 20:27:25 client-5 xinetd[3391]: START: shell pid=22133 from=192.168.4.5 May 17 20:27:25 client-5 rshd[22134]: root@client-5-ib.lab.whamcloud.com as root: cmd='rmdir /mnt/lustre/d0.insanity/client-18-ib;echo XXRETCODE:$?' May 17 20:27:33 client-5 kernel: Lustre: 17251:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf6681d6 to 0xfff98862bf6685f7 May 17 20:27:33 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 20:27:36 client-5 xinetd[3391]: EXIT: shell status=0 pid=22133 duration=11(sec) May 17 20:27:37 client-5 xinetd[3391]: START: shell pid=22175 from=192.168.4.5 May 17 20:27:37 client-5 rshd[22176]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 20:27:37 client-5 xinetd[3391]: EXIT: shell status=0 pid=22175 duration=0(sec) May 17 20:27:38 client-5 xinetd[3391]: START: shell pid=22239 from=192.168.4.5 May 17 20:27:38 client-5 rshd[22240]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 20:27:38 client-5 xinetd[3391]: EXIT: shell status=0 pid=22239 duration=0(sec) May 17 20:27:40 client-5 xinetd[3391]: START: shell pid=22332 from=192.168.4.5 May 17 20:27:40 client-5 rshd[22333]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 20:27:40 client-5 xinetd[3391]: EXIT: shell status=0 pid=22332 duration=0(sec) May 17 20:27:41 client-5 xinetd[3391]: START: shell pid=22503 from=192.168.4.5 May 17 20:27:41 client-5 rshd[22506]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == insanity test 4: Fourth Failure Mode: OST\/MDS Tue May 17 20:27:41 PDT 2011 ======================== 20:27:41 \(1305689261\)");echo XXRETCODE:$?' May 17 20:27:41 client-5 kernel: Lustre: DEBUG MARKER: == insanity test 4: Fourth Failure Mode: OST/MDS Tue May 17 20:27:41 PDT 2011 ======================== 20:27:41 (1305689261) May 17 20:27:41 client-5 xinetd[3391]: EXIT: shell status=0 pid=22503 duration=0(sec) May 17 20:27:43 client-5 xinetd[3391]: START: shell pid=22579 from=192.168.4.5 May 17 20:27:43 client-5 rshd[22580]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 20:27:43 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.129@o2ib. The obd_ping operation failed with -107 May 17 20:27:43 client-5 kernel: LustreError: Skipped 3 previous similar messages May 17 20:27:54 client-5 kernel: Lustre: 17250:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369114038763920 sent from lustre-MDT0000-mdc-ffff8102f9b5f800 to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305689228] [real_sent 1305689228] [current 1305689274] [deadline 46s] [delay 0s] req@ffff8102dfa80000 x1369114038763920/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.128@o2ib:12/10 lens 192/192 e 0 to 1 dl 1305689274 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 20:27:54 client-5 kernel: Lustre: 17250:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 3 previous similar messages May 17 20:28:15 client-5 xinetd[3391]: START: shell pid=22668 from=192.168.4.5 May 17 20:28:15 client-5 rshd[22669]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 20:28:15 client-5 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail. May 17 20:28:39 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff8102f9b5f800: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 20:28:39 client-5 kernel: Lustre: Skipped 1 previous similar message May 17 20:29:04 client-5 kernel: Lustre: 17252:0:(import.c:529:import_select_connection()) lustre-OST0000-osc-ffff8102f9b5f800: tried all connections, increasing latency to 21s May 17 20:29:04 client-5 kernel: Lustre: 17252:0:(import.c:529:import_select_connection()) Skipped 16 previous similar messages May 17 20:29:39 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff8102f9b5f800: Connection restored to service lustre-MDT0000 using nid 192.168.4.128@o2ib. May 17 20:29:39 client-5 kernel: Lustre: Skipped 3 previous similar messages May 17 20:29:44 client-5 kernel: Lustre: 17251:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf6685f7 to 0xfff98862bf668771 May 17 20:29:44 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 20:30:48 client-5 xinetd[3391]: EXIT: shell status=0 pid=22668 duration=153(sec) May 17 20:30:48 client-5 xinetd[3391]: EXIT: shell status=0 pid=22579 duration=185(sec) May 17 20:30:49 client-5 xinetd[3391]: START: shell pid=22839 from=192.168.4.5 May 17 20:30:49 client-5 rshd[22840]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 20:30:49 client-5 xinetd[3391]: EXIT: shell status=0 pid=22839 duration=0(sec) May 17 20:30:49 client-5 xinetd[3391]: START: shell pid=22929 from=192.168.4.5 May 17 20:30:49 client-5 rshd[22930]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 20:30:49 client-5 xinetd[3391]: EXIT: shell status=0 pid=22929 duration=0(sec) May 17 20:30:50 client-5 xinetd[3391]: START: shell pid=23102 from=192.168.4.5 May 17 20:30:50 client-5 rshd[23103]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == insanity test 5: Fifth Failure Mode: OST\/OST Tue May 17 20:30:50 PDT 2011 ========================= 20:30:50 \(1305689450\)");echo XXRETCODE:$?' May 17 20:30:50 client-5 kernel: Lustre: DEBUG MARKER: == insanity test 5: Fifth Failure Mode: OST/OST Tue May 17 20:30:50 PDT 2011 ========================= 20:30:50 (1305689450) May 17 20:30:50 client-5 xinetd[3391]: EXIT: shell status=0 pid=23102 duration=0(sec) May 17 20:30:51 client-5 xinetd[3391]: START: shell pid=23146 from=192.168.4.5 May 17 20:30:51 client-5 rshd[23147]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 20:30:51 client-5 xinetd[3391]: EXIT: shell status=0 pid=23146 duration=0(sec) May 17 20:30:59 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.129@o2ib. The obd_ping operation failed with -107 May 17 20:30:59 client-5 kernel: LustreError: Skipped 17 previous similar messages May 17 20:30:59 client-5 kernel: Lustre: lustre-OST0000-osc-ffff8102f9b5f800: Connection to service lustre-OST0000 via nid 192.168.4.129@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 20:31:03 client-5 xinetd[3391]: START: shell pid=23219 from=192.168.4.5 May 17 20:31:03 client-5 rshd[23220]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 20:31:20 client-5 xinetd[3391]: START: shell pid=23296 from=192.168.4.5 May 17 20:31:20 client-5 rshd[23297]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 20:31:29 client-5 xinetd[3391]: EXIT: shell status=0 pid=23219 duration=26(sec) May 17 20:31:29 client-5 xinetd[3391]: EXIT: shell status=0 pid=23296 duration=9(sec) May 17 20:31:47 client-5 xinetd[3391]: START: shell pid=23455 from=192.168.4.5 May 17 20:31:47 client-5 rshd[23456]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 20:31:47 client-5 xinetd[3391]: EXIT: shell status=0 pid=23455 duration=0(sec) May 17 20:31:47 client-5 xinetd[3391]: START: shell pid=23545 from=192.168.4.5 May 17 20:31:47 client-5 rshd[23546]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 20:31:47 client-5 xinetd[3391]: EXIT: shell status=0 pid=23545 duration=0(sec) May 17 20:31:47 client-5 xinetd[3391]: START: shell pid=23718 from=192.168.4.5 May 17 20:31:47 client-5 rshd[23719]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == insanity test 6: Sixth Failure Mode: OST\/CLIENT Tue May 17 20:31:47 PDT 2011 ====================== 20:31:47 \(1305689507\)");echo XXRETCODE:$?' May 17 20:31:47 client-5 kernel: Lustre: DEBUG MARKER: == insanity test 6: Sixth Failure Mode: OST/CLIENT Tue May 17 20:31:47 PDT 2011 ====================== 20:31:47 (1305689507) May 17 20:31:47 client-5 xinetd[3391]: EXIT: shell status=0 pid=23718 duration=0(sec) May 17 20:31:48 client-5 xinetd[3391]: START: shell pid=23762 from=192.168.4.5 May 17 20:31:48 client-5 rshd[23763]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 20:31:48 client-5 xinetd[3391]: EXIT: shell status=0 pid=23762 duration=0(sec) May 17 20:31:48 client-5 xinetd[3391]: START: shell pid=23802 from=192.168.4.5 May 17 20:31:48 client-5 rshd[23803]: root@client-5-ib.lab.whamcloud.com as root: cmd='touch /mnt/lustre/d0.insanity/client-5-ib_testfile;echo XXRETCODE:$?' May 17 20:31:49 client-5 xinetd[3391]: EXIT: shell status=0 pid=23802 duration=1(sec) May 17 20:32:04 client-5 xinetd[3391]: START: shell pid=23881 from=192.168.4.5 May 17 20:32:04 client-5 rshd[23882]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 20:32:08 client-5 xinetd[3391]: START: shell pid=23973 from=192.168.4.5 May 17 20:32:08 client-5 rshd[23974]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark Request clients to fail: . Num of clients to fail: 1, already failed: 0");echo XXRETCODE:$?' May 17 20:32:08 client-5 kernel: Lustre: DEBUG MARKER: Request clients to fail: . Num of clients to fail: 1, already failed: 0 May 17 20:32:08 client-5 xinetd[3391]: START: shell pid=24012 from=192.168.4.5 May 17 20:32:08 client-5 xinetd[3391]: EXIT: shell status=0 pid=23973 duration=0(sec) May 17 20:32:08 client-5 rshd[24013]: root@client-5-ib.lab.whamcloud.com as root: cmd='mkdir /mnt/lustre/d0.insanity/client-5-ib && ls -l /mnt/lustre/d0.insanity/client-5-ib;echo XXRETCODE:$?' May 17 20:32:08 client-5 xinetd[3391]: EXIT: shell status=0 pid=24012 duration=0(sec) May 17 20:32:09 client-5 xinetd[3391]: START: shell pid=24065 from=192.168.4.5 May 17 20:32:09 client-5 rshd[24066]: root@client-5-ib.lab.whamcloud.com as root: cmd='rmdir /mnt/lustre/d0.insanity/client-5-ib;echo XXRETCODE:$?' May 17 20:32:09 client-5 xinetd[3391]: START: shell pid=24103 from=192.168.4.5 May 17 20:32:09 client-5 xinetd[3391]: EXIT: shell status=0 pid=24065 duration=0(sec) May 17 20:32:09 client-5 rshd[24104]: root@client-5-ib.lab.whamcloud.com as root: cmd='rmdir /mnt/lustre/d0.insanity/client-18-ib;echo XXRETCODE:$?' May 17 20:32:20 client-5 xinetd[3391]: EXIT: shell status=0 pid=24103 duration=11(sec) May 17 20:32:21 client-5 xinetd[3391]: START: shell pid=24144 from=192.168.4.5 May 17 20:32:21 client-5 rshd[24146]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 20:33:56 client-5 kernel: Lustre: lustre-OST0000-osc-ffff8102f9b5f800: Connection restored to service lustre-OST0000 using nid 192.168.4.129@o2ib. May 17 20:33:56 client-5 kernel: Lustre: Skipped 4 previous similar messages May 17 20:33:56 client-5 xinetd[3391]: EXIT: shell status=0 pid=24144 duration=95(sec) May 17 20:33:56 client-5 xinetd[3391]: EXIT: shell status=0 pid=23881 duration=112(sec) May 17 20:35:02 client-5 xinetd[3391]: START: shell pid=24308 from=192.168.4.5 May 17 20:35:02 client-5 rshd[24309]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 20:35:02 client-5 xinetd[3391]: EXIT: shell status=0 pid=24308 duration=0(sec) May 17 20:35:02 client-5 xinetd[3391]: START: shell pid=24397 from=192.168.4.5 May 17 20:35:02 client-5 rshd[24399]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 20:35:02 client-5 xinetd[3391]: EXIT: shell status=0 pid=24397 duration=0(sec) May 17 20:35:03 client-5 xinetd[3391]: START: shell pid=24571 from=192.168.4.5 May 17 20:35:03 client-5 rshd[24572]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == insanity test 7: Seventh Failure Mode: CLIENT\/MDS Tue May 17 20:35:03 PDT 2011 ==================== 20:35:03 \(1305689703\)");echo XXRETCODE:$?' May 17 20:35:03 client-5 kernel: Lustre: DEBUG MARKER: == insanity test 7: Seventh Failure Mode: CLIENT/MDS Tue May 17 20:35:03 PDT 2011 ==================== 20:35:03 (1305689703) May 17 20:35:03 client-5 xinetd[3391]: EXIT: shell status=0 pid=24571 duration=0(sec) May 17 20:35:04 client-5 xinetd[3391]: START: shell pid=24615 from=192.168.4.5 May 17 20:35:04 client-5 rshd[24616]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 20:35:04 client-5 xinetd[3391]: EXIT: shell status=0 pid=24615 duration=0(sec) May 17 20:35:04 client-5 xinetd[3391]: START: shell pid=24655 from=192.168.4.5 May 17 20:35:04 client-5 rshd[24656]: root@client-5-ib.lab.whamcloud.com as root: cmd='touch /mnt/lustre/d0.insanity/client-5-ib_testfile;echo XXRETCODE:$?' May 17 20:35:04 client-5 xinetd[3391]: EXIT: shell status=0 pid=24655 duration=0(sec) May 17 20:35:04 client-5 xinetd[3391]: START: shell pid=24752 from=192.168.4.5 May 17 20:35:04 client-5 rshd[24754]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark Request clients to fail: 2. Num of clients to fail: 1, already failed: 0");echo XXRETCODE:$?' May 17 20:35:04 client-5 kernel: Lustre: DEBUG MARKER: Request clients to fail: 2. Num of clients to fail: 1, already failed: 0 May 17 20:35:04 client-5 xinetd[3391]: EXIT: shell status=0 pid=24752 duration=0(sec) May 17 20:35:04 client-5 xinetd[3391]: START: shell pid=24792 from=192.168.4.5 May 17 20:35:04 client-5 rshd[24793]: root@client-5-ib.lab.whamcloud.com as root: cmd='mkdir /mnt/lustre/d0.insanity/client-5-ib && ls -l /mnt/lustre/d0.insanity/client-5-ib;echo XXRETCODE:$?' May 17 20:35:04 client-5 xinetd[3391]: EXIT: shell status=0 pid=24792 duration=0(sec) May 17 20:35:05 client-5 xinetd[3391]: START: shell pid=24845 from=192.168.4.5 May 17 20:35:05 client-5 rshd[24846]: root@client-5-ib.lab.whamcloud.com as root: cmd='rmdir /mnt/lustre/d0.insanity/client-5-ib;echo XXRETCODE:$?' May 17 20:35:05 client-5 xinetd[3391]: START: shell pid=24883 from=192.168.4.5 May 17 20:35:05 client-5 xinetd[3391]: EXIT: shell status=0 pid=24845 duration=0(sec) May 17 20:35:05 client-5 rshd[24884]: root@client-5-ib.lab.whamcloud.com as root: cmd='rmdir /mnt/lustre/d0.insanity/client-18-ib;echo XXRETCODE:$?' May 17 20:35:16 client-5 xinetd[3391]: EXIT: shell status=0 pid=24883 duration=11(sec) May 17 20:35:17 client-5 xinetd[3391]: START: shell pid=24922 from=192.168.4.5 May 17 20:35:17 client-5 rshd[24924]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 20:35:17 client-5 xinetd[3391]: EXIT: shell status=0 pid=24922 duration=0(sec) May 17 20:35:17 client-5 xinetd[3391]: START: shell pid=24961 from=192.168.4.5 May 17 20:35:17 client-5 rshd[24962]: root@client-5-ib.lab.whamcloud.com as root: cmd='ls -l /mnt/lustre/d0.insanity;echo XXRETCODE:$?' May 17 20:35:17 client-5 xinetd[3391]: EXIT: shell status=0 pid=24961 duration=0(sec) May 17 20:35:17 client-5 xinetd[3391]: START: shell pid=25000 from=192.168.4.5 May 17 20:35:17 client-5 rshd[25001]: root@client-5-ib.lab.whamcloud.com as root: cmd='rm -f /mnt/lustre/d0.insanity/*_testfile;echo XXRETCODE:$?' May 17 20:35:17 client-5 xinetd[3391]: EXIT: shell status=0 pid=25000 duration=0(sec) May 17 20:36:18 client-5 xinetd[3391]: START: shell pid=25047 from=192.168.4.5 May 17 20:36:18 client-5 rshd[25048]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 20:36:19 client-5 xinetd[3391]: EXIT: shell status=0 pid=25047 duration=1(sec) May 17 20:36:19 client-5 xinetd[3391]: START: shell pid=25085 from=192.168.4.5 May 17 20:36:19 client-5 rshd[25086]: root@client-5-ib.lab.whamcloud.com as root: cmd='rm /mnt/lustre/d0.insanity/client-5-ib_testfile;echo XXRETCODE:$?' May 17 20:36:19 client-5 xinetd[3391]: EXIT: shell status=0 pid=25085 duration=0(sec) May 17 20:36:26 client-5 kernel: Lustre: 17250:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369114038764854 sent from MGC192.168.4.128@o2ib to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305689779] [real_sent 1305689779] [current 1305689786] [deadline 7s] [delay 0s] req@ffff8102f52d9400 x1369114038764854/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 192/192 e 0 to 1 dl 1305689786 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 20:36:26 client-5 kernel: Lustre: 17250:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 16 previous similar messages May 17 20:36:26 client-5 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail. May 17 20:36:37 client-5 kernel: Lustre: 17252:0:(import.c:529:import_select_connection()) MGC192.168.4.128@o2ib: tried all connections, increasing latency to 6s May 17 20:36:37 client-5 kernel: Lustre: 17252:0:(import.c:529:import_select_connection()) Skipped 23 previous similar messages May 17 20:36:37 client-5 kernel: Lustre: 17251:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0xfff98862bf668771 to 0xfff98862bf668ca3 May 17 20:36:37 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The obd_ping operation failed with -107 May 17 20:36:37 client-5 kernel: LustreError: Skipped 18 previous similar messages May 17 20:36:37 client-5 kernel: Lustre: lustre-MDT0000-mdc-ffff8102f9b5f800: Connection to service lustre-MDT0000 via nid 192.168.4.128@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 20:36:37 client-5 kernel: Lustre: Skipped 2 previous similar messages May 17 20:36:37 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 20:36:37 client-5 xinetd[3391]: START: shell pid=25242 from=192.168.4.5 May 17 20:36:37 client-5 rshd[25244]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 20:36:37 client-5 xinetd[3391]: START: shell pid=25281 from=192.168.4.5 May 17 20:36:37 client-5 xinetd[3391]: EXIT: shell status=0 pid=25242 duration=0(sec) May 17 20:36:37 client-5 rshd[25282]: root@client-5-ib.lab.whamcloud.com as root: cmd='ls -l /mnt/lustre/d0.insanity;echo XXRETCODE:$?' May 17 20:36:38 client-5 xinetd[3391]: START: shell pid=25319 from=192.168.4.5 May 17 20:36:38 client-5 xinetd[3391]: EXIT: shell status=0 pid=25281 duration=1(sec) May 17 20:36:38 client-5 rshd[25320]: root@client-5-ib.lab.whamcloud.com as root: cmd='rm -f /mnt/lustre/d0.insanity/*_testfile;echo XXRETCODE:$?' May 17 20:36:38 client-5 xinetd[3391]: EXIT: shell status=0 pid=25319 duration=0(sec) May 17 20:36:39 client-5 xinetd[3391]: START: shell pid=25383 from=192.168.4.5 May 17 20:36:39 client-5 rshd[25384]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 20:36:39 client-5 xinetd[3391]: EXIT: shell status=0 pid=25383 duration=0(sec) May 17 20:37:39 client-5 xinetd[3391]: START: shell pid=25480 from=192.168.4.5 May 17 20:37:39 client-5 rshd[25481]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 20:37:39 client-5 xinetd[3391]: EXIT: shell status=0 pid=25480 duration=0(sec) May 17 20:37:40 client-5 xinetd[3391]: START: shell pid=25653 from=192.168.4.5 May 17 20:37:40 client-5 rshd[25654]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == insanity test 8: Eighth Failure Mode: CLIENT\/OST Tue May 17 20:37:40 PDT 2011 ===================== 20:37:40 \(1305689860\)");echo XXRETCODE:$?' May 17 20:37:40 client-5 kernel: Lustre: DEBUG MARKER: == insanity test 8: Eighth Failure Mode: CLIENT/OST Tue May 17 20:37:40 PDT 2011 ===================== 20:37:40 (1305689860) May 17 20:37:40 client-5 xinetd[3391]: EXIT: shell status=0 pid=25653 duration=0(sec) May 17 20:37:41 client-5 xinetd[3391]: START: shell pid=25696 from=192.168.4.5 May 17 20:37:41 client-5 rshd[25698]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 20:37:41 client-5 xinetd[3391]: EXIT: shell status=0 pid=25696 duration=0(sec) May 17 20:37:41 client-5 xinetd[3391]: START: shell pid=25737 from=192.168.4.5 May 17 20:37:41 client-5 rshd[25738]: root@client-5-ib.lab.whamcloud.com as root: cmd='touch /mnt/lustre/d0.insanity/client-5-ib_testfile;echo XXRETCODE:$?' May 17 20:37:41 client-5 xinetd[3391]: EXIT: shell status=0 pid=25737 duration=0(sec) May 17 20:37:41 client-5 xinetd[3391]: START: shell pid=25835 from=192.168.4.5 May 17 20:37:41 client-5 rshd[25836]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark Request clients to fail: 2. Num of clients to fail: 1, already failed: 0");echo XXRETCODE:$?' May 17 20:37:41 client-5 kernel: Lustre: DEBUG MARKER: Request clients to fail: 2. Num of clients to fail: 1, already failed: 0 May 17 20:37:41 client-5 xinetd[3391]: EXIT: shell status=0 pid=25835 duration=0(sec) May 17 20:37:41 client-5 xinetd[3391]: START: shell pid=25874 from=192.168.4.5 May 17 20:37:41 client-5 rshd[25875]: root@client-5-ib.lab.whamcloud.com as root: cmd='mkdir /mnt/lustre/d0.insanity/client-5-ib && ls -l /mnt/lustre/d0.insanity/client-5-ib;echo XXRETCODE:$?' May 17 20:37:41 client-5 xinetd[3391]: EXIT: shell status=0 pid=25874 duration=0(sec) May 17 20:37:42 client-5 xinetd[3391]: START: shell pid=25927 from=192.168.4.5 May 17 20:37:42 client-5 rshd[25928]: root@client-5-ib.lab.whamcloud.com as root: cmd='rmdir /mnt/lustre/d0.insanity/client-5-ib;echo XXRETCODE:$?' May 17 20:37:42 client-5 xinetd[3391]: START: shell pid=25965 from=192.168.4.5 May 17 20:37:42 client-5 xinetd[3391]: EXIT: shell status=0 pid=25927 duration=0(sec) May 17 20:37:42 client-5 rshd[25966]: root@client-5-ib.lab.whamcloud.com as root: cmd='rmdir /mnt/lustre/d0.insanity/client-18-ib;echo XXRETCODE:$?' May 17 20:37:53 client-5 xinetd[3391]: EXIT: shell status=0 pid=25965 duration=11(sec) May 17 20:37:54 client-5 xinetd[3391]: START: shell pid=26005 from=192.168.4.5 May 17 20:37:54 client-5 rshd[26006]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 20:37:54 client-5 xinetd[3391]: START: shell pid=26043 from=192.168.4.5 May 17 20:37:54 client-5 xinetd[3391]: EXIT: shell status=0 pid=26005 duration=0(sec) May 17 20:37:54 client-5 rshd[26044]: root@client-5-ib.lab.whamcloud.com as root: cmd='ls -l /mnt/lustre/d0.insanity;echo XXRETCODE:$?' May 17 20:37:54 client-5 xinetd[3391]: START: shell pid=26082 from=192.168.4.5 May 17 20:37:54 client-5 xinetd[3391]: EXIT: shell status=0 pid=26043 duration=0(sec) May 17 20:37:54 client-5 rshd[26083]: root@client-5-ib.lab.whamcloud.com as root: cmd='rm -f /mnt/lustre/d0.insanity/*_testfile;echo XXRETCODE:$?' May 17 20:37:54 client-5 xinetd[3391]: EXIT: shell status=0 pid=26082 duration=0(sec) May 17 20:38:55 client-5 xinetd[3391]: START: shell pid=26129 from=192.168.4.5 May 17 20:38:55 client-5 rshd[26130]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 20:38:55 client-5 xinetd[3391]: EXIT: shell status=0 pid=26129 duration=0(sec) May 17 20:38:55 client-5 xinetd[3391]: START: shell pid=26169 from=192.168.4.5 May 17 20:38:55 client-5 rshd[26170]: root@client-5-ib.lab.whamcloud.com as root: cmd='touch /mnt/lustre/d0.insanity/client-5-ib_testfile;echo XXRETCODE:$?' May 17 20:38:56 client-5 xinetd[3391]: EXIT: shell status=0 pid=26169 duration=1(sec) May 17 20:39:09 client-5 xinetd[3391]: START: shell pid=26244 from=192.168.4.5 May 17 20:39:09 client-5 rshd[26245]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 20:39:21 client-5 kernel: Lustre: lustre-OST0000-osc-ffff8102f9b5f800: Connection restored to service lustre-OST0000 using nid 192.168.4.129@o2ib. May 17 20:39:21 client-5 kernel: Lustre: Skipped 2 previous similar messages May 17 20:39:21 client-5 xinetd[3391]: EXIT: shell status=0 pid=26244 duration=12(sec) May 17 20:39:22 client-5 xinetd[3391]: START: shell pid=26367 from=192.168.4.5 May 17 20:39:22 client-5 rshd[26368]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 20:39:22 client-5 xinetd[3391]: EXIT: shell status=0 pid=26367 duration=0(sec) May 17 20:39:23 client-5 xinetd[3391]: START: shell pid=26407 from=192.168.4.5 May 17 20:39:23 client-5 rshd[26408]: root@client-5-ib.lab.whamcloud.com as root: cmd='touch /mnt/lustre/d0.insanity/client-5-ib_testfile2;echo XXRETCODE:$?' May 17 20:39:23 client-5 xinetd[3391]: EXIT: shell status=0 pid=26407 duration=0(sec) May 17 20:40:23 client-5 xinetd[3391]: START: shell pid=26506 from=192.168.4.5 May 17 20:40:23 client-5 rshd[26509]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 20:40:24 client-5 xinetd[3391]: EXIT: shell status=0 pid=26506 duration=1(sec) May 17 20:40:24 client-5 xinetd[3391]: START: shell pid=26681 from=192.168.4.5 May 17 20:40:24 client-5 rshd[26682]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == insanity test 9: Ninth Failure Mode: CLIENT\/CLIENT Tue May 17 20:40:24 PDT 2011 =================== 20:40:24 \(1305690024\)");echo XXRETCODE:$?' May 17 20:40:24 client-5 kernel: Lustre: DEBUG MARKER: == insanity test 9: Ninth Failure Mode: CLIENT/CLIENT Tue May 17 20:40:24 PDT 2011 =================== 20:40:24 (1305690024) May 17 20:40:24 client-5 xinetd[3391]: EXIT: shell status=0 pid=26681 duration=0(sec) May 17 20:40:25 client-5 xinetd[3391]: START: shell pid=26723 from=192.168.4.5 May 17 20:40:25 client-5 rshd[26725]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 20:40:25 client-5 xinetd[3391]: EXIT: shell status=0 pid=26723 duration=0(sec) May 17 20:40:25 client-5 xinetd[3391]: START: shell pid=26764 from=192.168.4.5 May 17 20:40:25 client-5 rshd[26765]: root@client-5-ib.lab.whamcloud.com as root: cmd='touch /mnt/lustre/d0.insanity/client-5-ib_testfile;echo XXRETCODE:$?' May 17 20:40:25 client-5 xinetd[3391]: EXIT: shell status=0 pid=26764 duration=0(sec) May 17 20:40:25 client-5 xinetd[3391]: START: shell pid=26860 from=192.168.4.5 May 17 20:40:25 client-5 rshd[26863]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark Request clients to fail: 2. Num of clients to fail: 1, already failed: 0");echo XXRETCODE:$?' May 17 20:40:25 client-5 kernel: Lustre: DEBUG MARKER: Request clients to fail: 2. Num of clients to fail: 1, already failed: 0 May 17 20:40:25 client-5 xinetd[3391]: EXIT: shell status=0 pid=26860 duration=0(sec) May 17 20:40:25 client-5 xinetd[3391]: START: shell pid=26901 from=192.168.4.5 May 17 20:40:25 client-5 rshd[26902]: root@client-5-ib.lab.whamcloud.com as root: cmd='mkdir /mnt/lustre/d0.insanity/client-5-ib && ls -l /mnt/lustre/d0.insanity/client-5-ib;echo XXRETCODE:$?' May 17 20:40:25 client-5 xinetd[3391]: EXIT: shell status=0 pid=26901 duration=0(sec) May 17 20:40:26 client-5 xinetd[3391]: START: shell pid=26954 from=192.168.4.5 May 17 20:40:26 client-5 rshd[26955]: root@client-5-ib.lab.whamcloud.com as root: cmd='rmdir /mnt/lustre/d0.insanity/client-5-ib;echo XXRETCODE:$?' May 17 20:40:26 client-5 xinetd[3391]: START: shell pid=26992 from=192.168.4.5 May 17 20:40:26 client-5 xinetd[3391]: EXIT: shell status=0 pid=26954 duration=0(sec) May 17 20:40:26 client-5 rshd[26993]: root@client-5-ib.lab.whamcloud.com as root: cmd='rmdir /mnt/lustre/d0.insanity/client-18-ib;echo XXRETCODE:$?' May 17 20:40:37 client-5 xinetd[3391]: EXIT: shell status=0 pid=26992 duration=11(sec) May 17 20:40:38 client-5 xinetd[3391]: START: shell pid=27031 from=192.168.4.5 May 17 20:40:38 client-5 rshd[27033]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 20:40:38 client-5 xinetd[3391]: EXIT: shell status=0 pid=27031 duration=0(sec) May 17 20:40:38 client-5 xinetd[3391]: START: shell pid=27070 from=192.168.4.5 May 17 20:40:38 client-5 rshd[27071]: root@client-5-ib.lab.whamcloud.com as root: cmd='ls -l /mnt/lustre/d0.insanity;echo XXRETCODE:$?' May 17 20:40:38 client-5 xinetd[3391]: START: shell pid=27110 from=192.168.4.5 May 17 20:40:38 client-5 xinetd[3391]: EXIT: shell status=0 pid=27070 duration=0(sec) May 17 20:40:38 client-5 rshd[27111]: root@client-5-ib.lab.whamcloud.com as root: cmd='rm -f /mnt/lustre/d0.insanity/*_testfile;echo XXRETCODE:$?' May 17 20:40:38 client-5 xinetd[3391]: EXIT: shell status=0 pid=27110 duration=0(sec) May 17 20:41:39 client-5 xinetd[3391]: START: shell pid=27154 from=192.168.4.5 May 17 20:41:39 client-5 rshd[27155]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 20:41:40 client-5 xinetd[3391]: EXIT: shell status=0 pid=27154 duration=1(sec) May 17 20:41:40 client-5 xinetd[3391]: START: shell pid=27194 from=192.168.4.5 May 17 20:41:40 client-5 rshd[27195]: root@client-5-ib.lab.whamcloud.com as root: cmd='touch /mnt/lustre/d0.insanity/client-5-ib_testfile;echo XXRETCODE:$?' May 17 20:41:40 client-5 xinetd[3391]: EXIT: shell status=0 pid=27194 duration=0(sec) May 17 20:41:40 client-5 xinetd[3391]: START: shell pid=27287 from=192.168.4.5 May 17 20:41:40 client-5 rshd[27289]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark Request clients to fail: 2. Num of clients to fail: 1, already failed: 1");echo XXRETCODE:$?' May 17 20:41:40 client-5 kernel: Lustre: DEBUG MARKER: Request clients to fail: 2. Num of clients to fail: 1, already failed: 1 May 17 20:41:40 client-5 xinetd[3391]: EXIT: shell status=0 pid=27287 duration=0(sec) May 17 20:41:40 client-5 xinetd[3391]: START: shell pid=27381 from=192.168.4.5 May 17 20:41:40 client-5 rshd[27382]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark No clients failed!");echo XXRETCODE:$?' May 17 20:41:40 client-5 kernel: Lustre: DEBUG MARKER: No clients failed! May 17 20:41:40 client-5 xinetd[3391]: EXIT: shell status=0 pid=27381 duration=0(sec) May 17 20:41:41 client-5 xinetd[3391]: START: shell pid=27422 from=192.168.4.5 May 17 20:41:41 client-5 rshd[27423]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 20:41:41 client-5 xinetd[3391]: EXIT: shell status=0 pid=27422 duration=0(sec) May 17 20:41:41 client-5 xinetd[3391]: START: shell pid=27460 from=192.168.4.5 May 17 20:41:41 client-5 rshd[27461]: root@client-5-ib.lab.whamcloud.com as root: cmd='ls -l /mnt/lustre/d0.insanity;echo XXRETCODE:$?' May 17 20:41:41 client-5 xinetd[3391]: START: shell pid=27499 from=192.168.4.5 May 17 20:41:41 client-5 xinetd[3391]: EXIT: shell status=0 pid=27460 duration=0(sec) May 17 20:41:41 client-5 rshd[27500]: root@client-5-ib.lab.whamcloud.com as root: cmd='rm -f /mnt/lustre/d0.insanity/*_testfile;echo XXRETCODE:$?' May 17 20:41:41 client-5 xinetd[3391]: EXIT: shell status=0 pid=27499 duration=0(sec) May 17 20:41:43 client-5 xinetd[3391]: START: shell pid=27563 from=192.168.4.5 May 17 20:41:43 client-5 rshd[27564]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 20:41:43 client-5 xinetd[3391]: EXIT: shell status=0 pid=27563 duration=0(sec) May 17 20:42:43 client-5 xinetd[3391]: START: shell pid=27658 from=192.168.4.5 May 17 20:42:43 client-5 rshd[27659]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 20:42:43 client-5 xinetd[3391]: EXIT: shell status=0 pid=27658 duration=0(sec) May 17 20:42:43 client-5 xinetd[3391]: START: shell pid=27821 from=192.168.4.5 May 17 20:42:43 client-5 rshd[27822]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark SKIP: insanity test_10 skipping ALWAYS excluded test 10");echo XXRETCODE:$?' May 17 20:42:43 client-5 kernel: Lustre: DEBUG MARKER: SKIP: insanity test_10 skipping ALWAYS excluded test 10 May 17 20:42:43 client-5 xinetd[3391]: EXIT: shell status=0 pid=27821 duration=0(sec) May 17 20:42:43 client-5 xinetd[3391]: START: shell pid=27919 from=192.168.4.5 May 17 20:42:43 client-5 rshd[27920]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == insanity insanity.sh test complete, duration 1267 sec ============================================= 20:42:43 \(1305690163\)");echo XXRETCODE:$?' May 17 20:42:43 client-5 kernel: Lustre: DEBUG MARKER: == insanity insanity.sh test complete, duration 1267 sec ============================================= 20:42:43 (1305690163) May 17 20:42:43 client-5 xinetd[3391]: EXIT: shell status=0 pid=27919 duration=0(sec) May 17 20:42:59 client-5 kernel: Lustre: 17250:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369114038765594 sent from MGC192.168.4.128@o2ib to NID 192.168.4.128@o2ib has timed out for slow reply: [sent 1305690166] [real_sent 1305690166] [current 1305690178] [deadline 12s] [delay 0s] req@ffff81031f8f0000 x1369114038765594/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 192/192 e 0 to 1 dl 1305690178 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 May 17 20:42:59 client-5 kernel: Lustre: 17250:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 3 previous similar messages May 17 20:42:59 client-5 kernel: LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail. May 17 20:43:02 client-5 xinetd[3391]: START: shell pid=28080 from=192.168.4.5 May 17 20:43:02 client-5 rshd[28081]: root@client-5-ib.lab.whamcloud.com as root: cmd='stat -f /mnt/lustre;echo XXRETCODE:$?' May 17 20:43:02 client-5 xinetd[3391]: EXIT: shell status=0 pid=28080 duration=0(sec) May 17 20:43:02 client-5 xinetd[3391]: START: shell pid=28121 from=192.168.4.5 May 17 20:43:02 client-5 rshd[28122]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:-f; lsof -t /mnt/lustre || need_kill=no; if [ x-f != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount -f /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre is still busy, wait one second && sleep 1; umount -f /mnt/lustre; fi fi");echo XXRETCODE:$?' May 17 20:43:02 client-5 kernel: Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request May 17 20:43:02 client-5 kernel: Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request May 17 20:43:10 client-5 kernel: Lustre: client ffff8102f9b5f800 umount complete May 17 20:43:10 client-5 xinetd[3391]: EXIT: shell status=0 pid=28121 duration=8(sec) May 17 20:43:10 client-5 xinetd[3391]: START: shell pid=28172 from=192.168.4.5 May 17 20:43:10 client-5 rshd[28173]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:-f; lsof -t /mnt/lustre2 || need_kill=no; if [ x-f != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre2 | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount -f /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre2 is still busy, wait one second && sleep 1; umount -f /mnt/lustre2; fi fi");echo XXRETCODE:$?' May 17 20:43:10 client-5 xinetd[3391]: EXIT: shell status=0 pid=28172 duration=0(sec) May 17 20:43:36 client-5 kernel: Lustre: Acceptor stopping May 17 20:43:37 client-5 kernel: Lustre: Removed LNI 10.10.4.5@tcp May 17 20:43:39 client-5 kernel: Lustre: Removed LNI 192.168.4.5@o2ib May 17 20:43:40 client-5 kernel: Lustre: 28797:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 20:43:40 client-5 kernel: Lustre: 28798:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 20:43:40 client-5 kernel: Lustre: OBD class driver, http://www.lustre.org/ May 17 20:43:40 client-5 kernel: Lustre: Lustre Version: 2.0.60 May 17 20:43:40 client-5 kernel: Lustre: Build Version: jenkins-g985ac5b-PRISTINE-2.6.18-194.17.1.el5 May 17 20:43:40 client-5 kernel: Lustre: Lustre LU module (ffffffff888eb1c0). May 17 20:43:41 client-5 kernel: Lustre: Register global MR array, MR size: 0xffffffffffffffff, array size: 1 May 17 20:43:41 client-5 kernel: Lustre: Added LNI 192.168.4.5@o2ib [8/64/0/180] May 17 20:43:41 client-5 kernel: Lustre: Added LNI 10.10.4.5@tcp [8/256/0/180] May 17 20:43:41 client-5 kernel: Lustre: Accept all, port 7988 May 17 20:43:41 client-5 kernel: Lustre: Lustre OSC module (ffffffff88b3c6c0). May 17 20:43:41 client-5 kernel: Lustre: Lustre LOV module (ffffffff88bab540). May 17 20:43:41 client-5 kernel: Lustre: Lustre client module (ffffffff88c643a0). May 17 20:43:41 client-5 xinetd[3391]: START: shell pid=29203 from=192.168.4.5 May 17 20:43:41 client-5 rshd[29205]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark -----============= acceptance-small: sanity-quota ============----- Tue May 17 20:43:40 PDT 2011");echo XXRETCODE:$?' May 17 20:43:41 client-5 kernel: Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-quota ============----- Tue May 17 20:43:40 PDT 2011 May 17 20:43:41 client-5 xinetd[3391]: EXIT: shell status=0 pid=29203 duration=0(sec) May 17 20:43:41 client-5 xinetd[3391]: START: shell pid=29338 from=192.168.4.5 May 17 20:43:41 client-5 rshd[29339]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre opts:; lsof -t /mnt/lustre || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount /mnt/lustre 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre is still busy, wait one second && sleep 1; umount /mnt/lustre; fi fi");echo XXRETCODE:$?' May 17 20:43:41 client-5 xinetd[3391]: EXIT: shell status=0 pid=29338 duration=0(sec) May 17 20:43:41 client-5 xinetd[3391]: START: shell pid=29382 from=192.168.4.5 May 17 20:43:41 client-5 rshd[29383]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:; lsof -t /mnt/lustre2 || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre2 | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; busy=\$(umount /mnt/lustre2 2>&1 | grep -c busy); if [ \$busy -ne 0 ] ; then echo /mnt/lustre2 is still busy, wait one second && sleep 1; umount /mnt/lustre2; fi fi");echo XXRETCODE:$?' May 17 20:43:41 client-5 xinetd[3391]: EXIT: shell status=0 pid=29382 duration=0(sec) May 17 20:43:43 client-5 kernel: Lustre: 29585:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 20:44:41 client-5 xinetd[3391]: START: shell pid=30017 from=192.168.4.5 May 17 20:44:41 client-5 rshd[30018]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre' ' /proc/mounts); mpts=\$(mount | grep -w -c /mnt/lustre); if [ \$running -ne \$mpts ]; then echo \$(hostname) env are INSANE!; exit 1; fi");echo XXRETCODE:$?' May 17 20:44:41 client-5 xinetd[3391]: EXIT: shell status=0 pid=30017 duration=0(sec) May 17 20:44:41 client-5 xinetd[3391]: START: shell pid=30064 from=192.168.4.5 May 17 20:44:41 client-5 rshd[30065]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "running=\$(grep -c /mnt/lustre2' ' /proc/mounts); mpts=\$(mount | grep -w -c /mnt/lustre2); if [ \$running -ne \$mpts ]; then echo \$(hostname) env are INSANE!; exit 1; fi");echo XXRETCODE:$?' May 17 20:44:41 client-5 xinetd[3391]: EXIT: shell status=0 pid=30064 duration=0(sec) May 17 20:44:41 client-5 kernel: Lustre: 30107:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 20:44:41 client-5 kernel: Lustre: 30107:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message May 17 20:44:45 client-5 kernel: Lustre: 30444:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 50000: select flavor null May 17 20:44:45 client-5 kernel: LustreError: 152-6: Ignoring deprecated mount option 'acl'. May 17 20:44:45 client-5 kernel: Lustre: MGC192.168.4.128@o2ib: Reactivating import May 17 20:44:45 client-5 kernel: Lustre: 30444:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-mdc-ffff81031957b000->192.168.4.128@o2ib netid 50000: select flavor null May 17 20:44:45 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The mds_connect operation failed with -11 May 17 20:44:50 client-5 kernel: Lustre: 30444:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-OST0000-osc-ffff81031957b000->192.168.4.129@o2ib netid 50000: select flavor null May 17 20:44:50 client-5 kernel: Lustre: Client lustre-client has started May 17 20:44:50 client-5 kernel: Lustre: 30461:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. May 17 20:44:50 client-5 kernel: Lustre: 30461:0:(debug.c:320:libcfs_debug_str2mask()) Skipped 1 previous similar message May 17 20:44:50 client-5 xinetd[3391]: START: shell pid=30470 from=192.168.4.5 May 17 20:44:50 client-5 rshd[30472]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c " running=\$(mount | grep -c /mnt/lustre' '); rc=0; if [ \$running -eq 0 ] ; then mkdir -p /mnt/lustre; mount -t lustre -o flock,user_xattr,acl fat-intel-1-ib@o2ib:/lustre /mnt/lustre; rc=\$?; fi; exit \$rc");echo XXRETCODE:$?' May 17 20:44:51 client-5 xinetd[3391]: EXIT: shell status=0 pid=30470 duration=1(sec) May 17 20:44:51 client-5 xinetd[3391]: START: shell pid=30516 from=192.168.4.5 May 17 20:44:51 client-5 rshd[30517]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mount | grep -w /mnt/lustre");echo XXRETCODE:$?' May 17 20:44:51 client-5 xinetd[3391]: EXIT: shell status=0 pid=30516 duration=0(sec) May 17 20:44:51 client-5 xinetd[3391]: START: shell pid=30560 from=192.168.4.5 May 17 20:44:51 client-5 rshd[30561]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param debug=\"0x33f0406\"; lctl set_param subsystem_debug=\"0xffb7e3ff\"; lctl set_param debug_mb=32;");echo XXRETCODE:$?' May 17 20:44:51 client-5 xinetd[3391]: EXIT: shell status=0 pid=30560 duration=0(sec) May 17 20:44:51 client-5 xinetd[3391]: START: shell pid=30730 from=192.168.4.5 May 17 20:44:51 client-5 rshd[30731]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark Using TIMEOUT=20");echo XXRETCODE:$?' May 17 20:44:51 client-5 kernel: Lustre: DEBUG MARKER: Using TIMEOUT=20 May 17 20:44:51 client-5 xinetd[3391]: EXIT: shell status=0 pid=30730 duration=0(sec) May 17 20:44:56 client-5 xinetd[3391]: START: shell pid=30807 from=192.168.4.5 May 17 20:44:56 client-5 rshd[30808]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "cat /proc/mounts");echo XXRETCODE:$?' May 17 20:44:57 client-5 xinetd[3391]: EXIT: shell status=0 pid=30807 duration=1(sec) May 17 20:44:57 client-5 xinetd[3391]: START: shell pid=31075 from=192.168.4.5 May 17 20:44:58 client-5 rshd[31076]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark excepting tests: 10 12");echo XXRETCODE:$?' May 17 20:44:58 client-5 kernel: Lustre: DEBUG MARKER: excepting tests: 10 12 May 17 20:44:58 client-5 xinetd[3391]: EXIT: shell status=0 pid=31075 duration=1(sec) May 17 20:44:58 client-5 xinetd[3391]: START: shell pid=31176 from=192.168.4.5 May 17 20:44:58 client-5 rshd[31178]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param debug=+quota");echo XXRETCODE:$?' May 17 20:44:58 client-5 xinetd[3391]: EXIT: shell status=0 pid=31176 duration=0(sec) May 17 20:44:58 client-5 xinetd[3391]: START: shell pid=31270 from=192.168.4.5 May 17 20:44:58 client-5 rshd[31271]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark do the quotacheck ...");echo XXRETCODE:$?' May 17 20:44:58 client-5 kernel: Lustre: DEBUG MARKER: do the quotacheck ... May 17 20:44:58 client-5 xinetd[3391]: EXIT: shell status=0 pid=31270 duration=0(sec) May 17 20:45:01 client-5 xinetd[3391]: START: shell pid=31471 from=192.168.4.5 May 17 20:45:01 client-5 rshd[31472]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity-quota test 0: Test basic quota performance ================================================= 20:45:01 \(1305690301\)");echo XXRETCODE:$?' May 17 20:45:01 client-5 kernel: Lustre: DEBUG MARKER: == sanity-quota test 0: Test basic quota performance ================================================= 20:45:01 (1305690301) May 17 20:45:01 client-5 xinetd[3391]: EXIT: shell status=0 pid=31471 duration=0(sec) May 17 20:45:03 client-5 xinetd[3391]: START: shell pid=31593 from=192.168.4.5 May 17 20:45:03 client-5 rshd[31594]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "lctl set_param -n fail_loc=0 2>/dev/null || true");echo XXRETCODE:$?' May 17 20:45:03 client-5 xinetd[3391]: EXIT: shell status=0 pid=31593 duration=0(sec) May 17 20:45:05 client-5 xinetd[3391]: START: shell pid=31879 from=192.168.4.5 May 17 20:45:05 client-5 rshd[31880]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark == sanity-quota test 1: Block hard limit \(normal use and out of quota\) =============================== 20:45:05 \(1305690305\)");echo XXRETCODE:$?' May 17 20:45:05 client-5 kernel: Lustre: DEBUG MARKER: == sanity-quota test 1: Block hard limit (normal use and out of quota) =============================== 20:45:05 (1305690305) May 17 20:45:05 client-5 xinetd[3391]: EXIT: shell status=0 pid=31879 duration=0(sec) May 17 20:45:07 client-5 xinetd[3391]: START: shell pid=32069 from=192.168.4.5 May 17 20:45:07 client-5 rshd[32071]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark User quota \(limit: 115253 kbytes\)");echo XXRETCODE:$?' May 17 20:45:07 client-5 kernel: Lustre: DEBUG MARKER: User quota (limit: 115253 kbytes) May 17 20:45:07 client-5 xinetd[3391]: EXIT: shell status=0 pid=32069 duration=0(sec) May 17 20:45:10 client-5 xinetd[3391]: START: shell pid=32174 from=192.168.4.5 May 17 20:45:10 client-5 rshd[32176]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark Write ...");echo XXRETCODE:$?' May 17 20:45:10 client-5 kernel: Lustre: DEBUG MARKER: Write ... May 17 20:45:10 client-5 xinetd[3391]: EXIT: shell status=0 pid=32174 duration=0(sec) May 17 20:45:16 client-5 xinetd[3391]: START: shell pid=32269 from=192.168.4.5 May 17 20:45:16 client-5 rshd[32270]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark Done");echo XXRETCODE:$?' May 17 20:45:16 client-5 kernel: Lustre: DEBUG MARKER: Done May 17 20:45:16 client-5 xinetd[3391]: EXIT: shell status=0 pid=32269 duration=0(sec) May 17 20:45:16 client-5 xinetd[3391]: START: shell pid=32361 from=192.168.4.5 May 17 20:45:16 client-5 rshd[32363]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark Write out of block quota ...");echo XXRETCODE:$?' May 17 20:45:16 client-5 kernel: Lustre: DEBUG MARKER: Write out of block quota ... May 17 20:45:16 client-5 xinetd[3391]: EXIT: shell status=0 pid=32361 duration=0(sec) May 17 20:45:22 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.129@o2ib. The ost_write operation failed with -122 May 17 20:45:22 client-5 kernel: LustreError: 32397:0:(vvp_io.c:987:vvp_io_commit_write()) Write page 27435 of inode ffff8102e6e0e1d0 failed -122 May 17 20:45:22 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc start May 17 20:45:22 client-5 kernel: Lustre: DEBUG MARKER: cancel_lru_locks osc stop May 17 20:45:22 client-5 kernel: LustreError: 32412:0:(vvp_io.c:987:vvp_io_commit_write()) Write page 28813 of inode ffff8102e6e0e1d0 failed -122 May 17 20:45:23 client-5 xinetd[3391]: START: shell pid=32512 from=192.168.4.5 May 17 20:45:23 client-5 rshd[32513]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark --------------------------------------");echo XXRETCODE:$?' May 17 20:45:23 client-5 kernel: Lustre: DEBUG MARKER: -------------------------------------- May 17 20:45:23 client-5 xinetd[3391]: EXIT: shell status=0 pid=32512 duration=0(sec) May 17 20:45:23 client-5 xinetd[3391]: START: shell pid=32605 from=192.168.4.5 May 17 20:45:23 client-5 rshd[32606]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark Group quota \(limit: 115253 kbytes\)");echo XXRETCODE:$?' May 17 20:45:23 client-5 kernel: Lustre: DEBUG MARKER: Group quota (limit: 115253 kbytes) May 17 20:45:23 client-5 xinetd[3391]: EXIT: shell status=0 pid=32605 duration=0(sec) May 17 20:45:23 client-5 kernel: LustreError: 32640:0:(quota_ctl.c:328:client_quota_ctl()) ptlrpc_queue_wait failed, rc: -5 May 17 20:45:26 client-5 xinetd[3391]: START: shell pid=32710 from=192.168.4.5 May 17 20:45:27 client-5 rshd[32711]: root@client-5-ib.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "/usr/sbin/lctl mark Write ...");echo XXRETCODE:$?' May 17 20:45:27 client-5 kernel: Lustre: DEBUG MARKER: Write ... May 17 20:45:27 client-5 xinetd[3391]: EXIT: shell status=0 pid=32710 duration=1(sec) May 17 20:48:29 client-5 kernel: INFO: task pdflush:395 blocked for more than 120 seconds. May 17 20:48:29 client-5 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. May 17 20:48:29 client-5 kernel: pdflush D ffff810001004420 0 395 123 396 394 (L-TLB) May 17 20:48:29 client-5 kernel: ffff81033e977b30 0000000000000046 ffff8102c2ec44a8 0000000000000008 May 17 20:48:29 client-5 kernel: ffff81032217d440 000000000000000a ffff81033fc0c0c0 ffffffff80308b60 May 17 20:48:29 client-5 kernel: 0000401e1441fdc5 0000000000002389 ffff81033fc0c2a8 00000000020bdfef May 17 20:48:29 client-5 kernel: Call Trace: May 17 20:48:29 client-5 kernel: [] do_gettimeofday+0x40/0x90 May 17 20:48:29 client-5 kernel: [] sync_page+0x0/0x43 May 17 20:48:29 client-5 kernel: [] io_schedule+0x3f/0x67 May 17 20:48:29 client-5 kernel: [] sync_page+0x3e/0x43 May 17 20:48:29 client-5 kernel: [] __wait_on_bit_lock+0x36/0x66 May 17 20:48:29 client-5 kernel: [] __lock_page+0x5e/0x64 May 17 20:48:29 client-5 kernel: [] wake_bit_function+0x0/0x23 May 17 20:48:29 client-5 kernel: [] pagevec_lookup_tag+0x1a/0x21 May 17 20:48:29 client-5 kernel: [] mpage_writepages+0x14f/0x37d May 17 20:48:29 client-5 kernel: [] :lustre:ll_writepage+0x0/0x450 May 17 20:48:29 client-5 kernel: [] __next_cpu+0x19/0x28 May 17 20:48:29 client-5 kernel: [] find_busiest_group+0x20d/0x621 May 17 20:48:29 client-5 kernel: [] do_writepages+0x20/0x2f May 17 20:48:29 client-5 kernel: [] __writeback_single_inode+0x1ae/0x328 May 17 20:48:29 client-5 kernel: [] sync_sb_inodes+0x1b5/0x26f May 17 20:48:29 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 20:48:29 client-5 kernel: [] writeback_inodes+0x82/0xd8 May 17 20:48:29 client-5 kernel: [] wb_kupdate+0xd4/0x14e May 17 20:48:29 client-5 kernel: [] pdflush+0x0/0x1fb May 17 20:48:29 client-5 kernel: [] pdflush+0x151/0x1fb May 17 20:48:29 client-5 kernel: [] wb_kupdate+0x0/0x14e May 17 20:48:29 client-5 kernel: [] kthread+0xfe/0x132 May 17 20:48:29 client-5 kernel: [] child_rip+0xa/0x11 May 17 20:48:29 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 20:48:29 client-5 kernel: [] kthread+0x0/0x132 May 17 20:48:29 client-5 kernel: [] child_rip+0x0/0x11 May 17 20:48:29 client-5 kernel: May 17 20:50:29 client-5 kernel: INFO: task pdflush:395 blocked for more than 120 seconds. May 17 20:50:29 client-5 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. May 17 20:50:29 client-5 kernel: pdflush D ffff810001004420 0 395 123 396 394 (L-TLB) May 17 20:50:29 client-5 kernel: ffff81033e977b30 0000000000000046 ffff8102c2ec44a8 0000000000000008 May 17 20:50:29 client-5 kernel: ffff81032217d440 000000000000000a ffff81033fc0c0c0 ffffffff80308b60 May 17 20:50:29 client-5 kernel: 0000401e1441fdc5 0000000000002389 ffff81033fc0c2a8 00000000020bdfef May 17 20:50:29 client-5 kernel: Call Trace: May 17 20:50:29 client-5 kernel: [] do_gettimeofday+0x40/0x90 May 17 20:50:29 client-5 kernel: [] sync_page+0x0/0x43 May 17 20:50:29 client-5 kernel: [] io_schedule+0x3f/0x67 May 17 20:50:29 client-5 kernel: [] sync_page+0x3e/0x43 May 17 20:50:29 client-5 kernel: [] __wait_on_bit_lock+0x36/0x66 May 17 20:50:29 client-5 kernel: [] __lock_page+0x5e/0x64 May 17 20:50:29 client-5 kernel: [] wake_bit_function+0x0/0x23 May 17 20:50:29 client-5 kernel: [] pagevec_lookup_tag+0x1a/0x21 May 17 20:50:29 client-5 kernel: [] mpage_writepages+0x14f/0x37d May 17 20:50:29 client-5 kernel: [] :lustre:ll_writepage+0x0/0x450 May 17 20:50:29 client-5 kernel: [] __next_cpu+0x19/0x28 May 17 20:50:29 client-5 kernel: [] find_busiest_group+0x20d/0x621 May 17 20:50:29 client-5 kernel: [] do_writepages+0x20/0x2f May 17 20:50:29 client-5 kernel: [] __writeback_single_inode+0x1ae/0x328 May 17 20:50:29 client-5 kernel: [] sync_sb_inodes+0x1b5/0x26f May 17 20:50:29 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 20:50:29 client-5 kernel: [] writeback_inodes+0x82/0xd8 May 17 20:50:29 client-5 kernel: [] wb_kupdate+0xd4/0x14e May 17 20:50:29 client-5 kernel: [] pdflush+0x0/0x1fb May 17 20:50:29 client-5 kernel: [] pdflush+0x151/0x1fb May 17 20:50:29 client-5 kernel: [] wb_kupdate+0x0/0x14e May 17 20:50:29 client-5 kernel: [] kthread+0xfe/0x132 May 17 20:50:29 client-5 kernel: [] child_rip+0xa/0x11 May 17 20:50:29 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 20:50:29 client-5 kernel: [] kthread+0x0/0x132 May 17 20:50:29 client-5 kernel: [] child_rip+0x0/0x11 May 17 20:50:29 client-5 kernel: May 17 20:52:29 client-5 kernel: INFO: task pdflush:395 blocked for more than 120 seconds. May 17 20:52:29 client-5 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. May 17 20:52:29 client-5 kernel: pdflush D ffff810001004420 0 395 123 396 394 (L-TLB) May 17 20:52:29 client-5 kernel: ffff81033e977b30 0000000000000046 ffff8102c2ec44a8 0000000000000008 May 17 20:52:29 client-5 kernel: ffff81032217d440 000000000000000a ffff81033fc0c0c0 ffffffff80308b60 May 17 20:52:29 client-5 kernel: 0000401e1441fdc5 0000000000002389 ffff81033fc0c2a8 00000000020bdfef May 17 20:52:29 client-5 kernel: Call Trace: May 17 20:52:29 client-5 kernel: [] do_gettimeofday+0x40/0x90 May 17 20:52:29 client-5 kernel: [] sync_page+0x0/0x43 May 17 20:52:29 client-5 kernel: [] io_schedule+0x3f/0x67 May 17 20:52:29 client-5 kernel: [] sync_page+0x3e/0x43 May 17 20:52:29 client-5 kernel: [] __wait_on_bit_lock+0x36/0x66 May 17 20:52:29 client-5 kernel: [] __lock_page+0x5e/0x64 May 17 20:52:29 client-5 kernel: [] wake_bit_function+0x0/0x23 May 17 20:52:29 client-5 kernel: [] pagevec_lookup_tag+0x1a/0x21 May 17 20:52:29 client-5 kernel: [] mpage_writepages+0x14f/0x37d May 17 20:52:29 client-5 kernel: [] :lustre:ll_writepage+0x0/0x450 May 17 20:52:29 client-5 kernel: [] __next_cpu+0x19/0x28 May 17 20:52:29 client-5 kernel: [] find_busiest_group+0x20d/0x621 May 17 20:52:29 client-5 kernel: [] do_writepages+0x20/0x2f May 17 20:52:29 client-5 kernel: [] __writeback_single_inode+0x1ae/0x328 May 17 20:52:29 client-5 kernel: [] sync_sb_inodes+0x1b5/0x26f May 17 20:52:29 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 20:52:29 client-5 kernel: [] writeback_inodes+0x82/0xd8 May 17 20:52:29 client-5 kernel: [] wb_kupdate+0xd4/0x14e May 17 20:52:29 client-5 kernel: [] pdflush+0x0/0x1fb May 17 20:52:29 client-5 kernel: [] pdflush+0x151/0x1fb May 17 20:52:29 client-5 kernel: [] wb_kupdate+0x0/0x14e May 17 20:52:29 client-5 kernel: [] kthread+0xfe/0x132 May 17 20:52:29 client-5 kernel: [] child_rip+0xa/0x11 May 17 20:52:29 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 20:52:29 client-5 kernel: [] kthread+0x0/0x132 May 17 20:52:29 client-5 kernel: [] child_rip+0x0/0x11 May 17 20:52:29 client-5 kernel: May 17 20:54:29 client-5 kernel: INFO: task pdflush:395 blocked for more than 120 seconds. May 17 20:54:29 client-5 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. May 17 20:54:29 client-5 kernel: pdflush D ffff810001004420 0 395 123 396 394 (L-TLB) May 17 20:54:29 client-5 kernel: ffff81033e977b30 0000000000000046 ffff8102c2ec44a8 0000000000000008 May 17 20:54:29 client-5 kernel: ffff81032217d440 000000000000000a ffff81033fc0c0c0 ffffffff80308b60 May 17 20:54:29 client-5 kernel: 0000401e1441fdc5 0000000000002389 ffff81033fc0c2a8 00000000020bdfef May 17 20:54:29 client-5 kernel: Call Trace: May 17 20:54:29 client-5 kernel: [] do_gettimeofday+0x40/0x90 May 17 20:54:29 client-5 kernel: [] sync_page+0x0/0x43 May 17 20:54:29 client-5 kernel: [] io_schedule+0x3f/0x67 May 17 20:54:29 client-5 kernel: [] sync_page+0x3e/0x43 May 17 20:54:29 client-5 kernel: [] __wait_on_bit_lock+0x36/0x66 May 17 20:54:29 client-5 kernel: [] __lock_page+0x5e/0x64 May 17 20:54:29 client-5 kernel: [] wake_bit_function+0x0/0x23 May 17 20:54:29 client-5 kernel: [] pagevec_lookup_tag+0x1a/0x21 May 17 20:54:29 client-5 kernel: [] mpage_writepages+0x14f/0x37d May 17 20:54:29 client-5 kernel: [] :lustre:ll_writepage+0x0/0x450 May 17 20:54:29 client-5 kernel: [] __next_cpu+0x19/0x28 May 17 20:54:29 client-5 kernel: [] find_busiest_group+0x20d/0x621 May 17 20:54:29 client-5 kernel: [] do_writepages+0x20/0x2f May 17 20:54:29 client-5 kernel: [] __writeback_single_inode+0x1ae/0x328 May 17 20:54:29 client-5 kernel: [] sync_sb_inodes+0x1b5/0x26f May 17 20:54:29 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 20:54:29 client-5 kernel: [] writeback_inodes+0x82/0xd8 May 17 20:54:29 client-5 kernel: [] wb_kupdate+0xd4/0x14e May 17 20:54:29 client-5 kernel: [] pdflush+0x0/0x1fb May 17 20:54:29 client-5 kernel: [] pdflush+0x151/0x1fb May 17 20:54:29 client-5 kernel: [] wb_kupdate+0x0/0x14e May 17 20:54:29 client-5 kernel: [] kthread+0xfe/0x132 May 17 20:54:29 client-5 kernel: [] child_rip+0xa/0x11 May 17 20:54:29 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 20:54:29 client-5 kernel: [] kthread+0x0/0x132 May 17 20:54:29 client-5 kernel: [] child_rip+0x0/0x11 May 17 20:54:29 client-5 kernel: May 17 20:56:29 client-5 kernel: INFO: task pdflush:395 blocked for more than 120 seconds. May 17 20:56:29 client-5 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. May 17 20:56:29 client-5 kernel: pdflush D ffff810001004420 0 395 123 396 394 (L-TLB) May 17 20:56:30 client-5 kernel: ffff81033e977b30 0000000000000046 ffff8102c2ec44a8 0000000000000008 May 17 20:56:30 client-5 kernel: ffff81032217d440 000000000000000a ffff81033fc0c0c0 ffffffff80308b60 May 17 20:56:30 client-5 kernel: 0000401e1441fdc5 0000000000002389 ffff81033fc0c2a8 00000000020bdfef May 17 20:56:30 client-5 kernel: Call Trace: May 17 20:56:30 client-5 kernel: [] do_gettimeofday+0x40/0x90 May 17 20:56:30 client-5 kernel: [] sync_page+0x0/0x43 May 17 20:56:30 client-5 kernel: [] io_schedule+0x3f/0x67 May 17 20:56:30 client-5 kernel: [] sync_page+0x3e/0x43 May 17 20:56:30 client-5 kernel: [] __wait_on_bit_lock+0x36/0x66 May 17 20:56:30 client-5 kernel: [] __lock_page+0x5e/0x64 May 17 20:56:30 client-5 kernel: [] wake_bit_function+0x0/0x23 May 17 20:56:30 client-5 kernel: [] pagevec_lookup_tag+0x1a/0x21 May 17 20:56:30 client-5 kernel: [] mpage_writepages+0x14f/0x37d May 17 20:56:30 client-5 kernel: [] :lustre:ll_writepage+0x0/0x450 May 17 20:56:30 client-5 kernel: [] __next_cpu+0x19/0x28 May 17 20:56:30 client-5 kernel: [] find_busiest_group+0x20d/0x621 May 17 20:56:30 client-5 kernel: [] do_writepages+0x20/0x2f May 17 20:56:30 client-5 kernel: [] __writeback_single_inode+0x1ae/0x328 May 17 20:56:30 client-5 kernel: [] sync_sb_inodes+0x1b5/0x26f May 17 20:56:30 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 20:56:30 client-5 kernel: [] writeback_inodes+0x82/0xd8 May 17 20:56:30 client-5 kernel: [] wb_kupdate+0xd4/0x14e May 17 20:56:30 client-5 kernel: [] pdflush+0x0/0x1fb May 17 20:56:30 client-5 kernel: [] pdflush+0x151/0x1fb May 17 20:56:30 client-5 kernel: [] wb_kupdate+0x0/0x14e May 17 20:56:30 client-5 kernel: [] kthread+0xfe/0x132 May 17 20:56:30 client-5 kernel: [] child_rip+0xa/0x11 May 17 20:56:30 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 20:56:30 client-5 kernel: [] kthread+0x0/0x132 May 17 20:56:30 client-5 kernel: [] child_rip+0x0/0x11 May 17 20:56:30 client-5 kernel: May 17 20:58:30 client-5 kernel: INFO: task pdflush:395 blocked for more than 120 seconds. May 17 20:58:30 client-5 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. May 17 20:58:30 client-5 kernel: pdflush D ffff810001004420 0 395 123 396 394 (L-TLB) May 17 20:58:30 client-5 kernel: ffff81033e977b30 0000000000000046 ffff8102c2ec44a8 0000000000000008 May 17 20:58:30 client-5 kernel: ffff81032217d440 000000000000000a ffff81033fc0c0c0 ffffffff80308b60 May 17 20:58:30 client-5 kernel: 0000401e1441fdc5 0000000000002389 ffff81033fc0c2a8 00000000020bdfef May 17 20:58:30 client-5 kernel: Call Trace: May 17 20:58:30 client-5 kernel: [] do_gettimeofday+0x40/0x90 May 17 20:58:30 client-5 kernel: [] sync_page+0x0/0x43 May 17 20:58:30 client-5 kernel: [] io_schedule+0x3f/0x67 May 17 20:58:30 client-5 kernel: [] sync_page+0x3e/0x43 May 17 20:58:30 client-5 kernel: [] __wait_on_bit_lock+0x36/0x66 May 17 20:58:30 client-5 kernel: [] __lock_page+0x5e/0x64 May 17 20:58:30 client-5 kernel: [] wake_bit_function+0x0/0x23 May 17 20:58:30 client-5 kernel: [] pagevec_lookup_tag+0x1a/0x21 May 17 20:58:30 client-5 kernel: [] mpage_writepages+0x14f/0x37d May 17 20:58:30 client-5 kernel: [] :lustre:ll_writepage+0x0/0x450 May 17 20:58:30 client-5 kernel: [] __next_cpu+0x19/0x28 May 17 20:58:30 client-5 kernel: [] find_busiest_group+0x20d/0x621 May 17 20:58:30 client-5 kernel: [] do_writepages+0x20/0x2f May 17 20:58:30 client-5 kernel: [] __writeback_single_inode+0x1ae/0x328 May 17 20:58:30 client-5 kernel: [] sync_sb_inodes+0x1b5/0x26f May 17 20:58:30 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 20:58:30 client-5 kernel: [] writeback_inodes+0x82/0xd8 May 17 20:58:30 client-5 kernel: [] wb_kupdate+0xd4/0x14e May 17 20:58:30 client-5 kernel: [] pdflush+0x0/0x1fb May 17 20:58:30 client-5 kernel: [] pdflush+0x151/0x1fb May 17 20:58:30 client-5 kernel: [] wb_kupdate+0x0/0x14e May 17 20:58:30 client-5 kernel: [] kthread+0xfe/0x132 May 17 20:58:30 client-5 kernel: [] child_rip+0xa/0x11 May 17 20:58:30 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 20:58:30 client-5 kernel: [] kthread+0x0/0x132 May 17 20:58:30 client-5 kernel: [] child_rip+0x0/0x11 May 17 20:58:30 client-5 kernel: May 17 21:00:30 client-5 kernel: INFO: task pdflush:395 blocked for more than 120 seconds. May 17 21:00:30 client-5 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. May 17 21:00:30 client-5 kernel: pdflush D ffff810001004420 0 395 123 396 394 (L-TLB) May 17 21:00:30 client-5 kernel: ffff81033e977b30 0000000000000046 ffff8102c2ec44a8 0000000000000008 May 17 21:00:30 client-5 kernel: ffff81032217d440 000000000000000a ffff81033fc0c0c0 ffffffff80308b60 May 17 21:00:30 client-5 kernel: 0000401e1441fdc5 0000000000002389 ffff81033fc0c2a8 00000000020bdfef May 17 21:00:30 client-5 kernel: Call Trace: May 17 21:00:30 client-5 kernel: [] do_gettimeofday+0x40/0x90 May 17 21:00:30 client-5 kernel: [] sync_page+0x0/0x43 May 17 21:00:30 client-5 kernel: [] io_schedule+0x3f/0x67 May 17 21:00:30 client-5 kernel: [] sync_page+0x3e/0x43 May 17 21:00:30 client-5 kernel: [] __wait_on_bit_lock+0x36/0x66 May 17 21:00:30 client-5 kernel: [] __lock_page+0x5e/0x64 May 17 21:00:30 client-5 kernel: [] wake_bit_function+0x0/0x23 May 17 21:00:30 client-5 kernel: [] pagevec_lookup_tag+0x1a/0x21 May 17 21:00:30 client-5 kernel: [] mpage_writepages+0x14f/0x37d May 17 21:00:30 client-5 kernel: [] :lustre:ll_writepage+0x0/0x450 May 17 21:00:30 client-5 kernel: [] __next_cpu+0x19/0x28 May 17 21:00:30 client-5 kernel: [] find_busiest_group+0x20d/0x621 May 17 21:00:30 client-5 kernel: [] do_writepages+0x20/0x2f May 17 21:00:30 client-5 kernel: [] __writeback_single_inode+0x1ae/0x328 May 17 21:00:30 client-5 kernel: [] sync_sb_inodes+0x1b5/0x26f May 17 21:00:30 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:00:30 client-5 kernel: [] writeback_inodes+0x82/0xd8 May 17 21:00:30 client-5 kernel: [] wb_kupdate+0xd4/0x14e May 17 21:00:30 client-5 kernel: [] pdflush+0x0/0x1fb May 17 21:00:30 client-5 kernel: [] pdflush+0x151/0x1fb May 17 21:00:30 client-5 kernel: [] wb_kupdate+0x0/0x14e May 17 21:00:30 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:00:30 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:00:30 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:00:30 client-5 kernel: [] kthread+0x0/0x132 May 17 21:00:30 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:00:30 client-5 kernel: May 17 21:00:50 client-5 kernel: Lustre: 28904:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369115429175799 sent from lustre-OST0001-osc-ffff81031957b000 to NID 192.168.4.129@o2ib has timed out for slow reply: [sent 1305690327] [real_sent 1305690327] [current 1305691250] [deadline 923s] [delay 0s] req@ffff8102eb50ec00 x1369115429175799/t0(0) o-1->lustre-OST0001_UUID@192.168.4.129@o2ib:6/4 lens 456/416 e 5 to 1 dl 1305691250 ref 2 fl Rpc:X/ffffffff/ffffffff rc 0/-1 May 17 21:00:50 client-5 kernel: Lustre: lustre-OST0001-osc-ffff81031957b000: Connection to service lustre-OST0001 via nid 192.168.4.129@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 21:00:50 client-5 kernel: Lustre: 28904:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369115429175800 sent from lustre-OST0001-osc-ffff81031957b000 to NID 192.168.4.129@o2ib has timed out for slow reply: [sent 1305690327] [real_sent 1305690327] [current 1305691250] [deadline 923s] [delay 0s] req@ffff8102eb50e800 x1369115429175800/t0(0) o-1->lustre-OST0001_UUID@192.168.4.129@o2ib:6/4 lens 456/416 e 5 to 1 dl 1305691250 ref 2 fl Rpc:X/ffffffff/ffffffff rc 0/-1 May 17 21:00:50 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.129@o2ib. The ost_connect operation failed with -16 May 17 21:00:50 client-5 kernel: LustreError: Skipped 1 previous similar message May 17 21:00:55 client-5 kernel: Lustre: lustre-OST0001-osc-ffff81031957b000: Connection restored to service lustre-OST0001 using nid 192.168.4.129@o2ib. May 17 21:02:30 client-5 kernel: INFO: task pdflush:395 blocked for more than 120 seconds. May 17 21:02:30 client-5 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. May 17 21:02:30 client-5 kernel: pdflush D ffff810001004420 0 395 123 396 394 (L-TLB) May 17 21:02:30 client-5 kernel: ffff81033e977b30 0000000000000046 ffff8102c2ec44a8 0000000000000008 May 17 21:02:30 client-5 kernel: ffff81032217d440 000000000000000a ffff81033fc0c0c0 ffffffff80308b60 May 17 21:02:30 client-5 kernel: 0000401e1441fdc5 0000000000002389 ffff81033fc0c2a8 00000000020bdfef May 17 21:02:30 client-5 kernel: Call Trace: May 17 21:02:30 client-5 kernel: [] do_gettimeofday+0x40/0x90 May 17 21:02:30 client-5 kernel: [] sync_page+0x0/0x43 May 17 21:02:30 client-5 kernel: [] io_schedule+0x3f/0x67 May 17 21:02:30 client-5 kernel: [] sync_page+0x3e/0x43 May 17 21:02:30 client-5 kernel: [] __wait_on_bit_lock+0x36/0x66 May 17 21:02:30 client-5 kernel: [] __lock_page+0x5e/0x64 May 17 21:02:30 client-5 kernel: [] wake_bit_function+0x0/0x23 May 17 21:02:30 client-5 kernel: [] pagevec_lookup_tag+0x1a/0x21 May 17 21:02:30 client-5 kernel: [] mpage_writepages+0x14f/0x37d May 17 21:02:30 client-5 kernel: [] :lustre:ll_writepage+0x0/0x450 May 17 21:02:30 client-5 kernel: [] __next_cpu+0x19/0x28 May 17 21:02:30 client-5 kernel: [] find_busiest_group+0x20d/0x621 May 17 21:02:30 client-5 kernel: [] do_writepages+0x20/0x2f May 17 21:02:30 client-5 kernel: [] __writeback_single_inode+0x1ae/0x328 May 17 21:02:30 client-5 kernel: [] sync_sb_inodes+0x1b5/0x26f May 17 21:02:30 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:02:30 client-5 kernel: [] writeback_inodes+0x82/0xd8 May 17 21:02:30 client-5 kernel: [] wb_kupdate+0xd4/0x14e May 17 21:02:30 client-5 kernel: [] pdflush+0x0/0x1fb May 17 21:02:30 client-5 kernel: [] pdflush+0x151/0x1fb May 17 21:02:30 client-5 kernel: [] wb_kupdate+0x0/0x14e May 17 21:02:30 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:02:30 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:02:30 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:02:30 client-5 kernel: [] kthread+0x0/0x132 May 17 21:02:30 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:02:30 client-5 kernel: May 17 21:04:30 client-5 kernel: INFO: task pdflush:395 blocked for more than 120 seconds. May 17 21:04:30 client-5 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. May 17 21:04:30 client-5 kernel: pdflush D ffff810001004420 0 395 123 396 394 (L-TLB) May 17 21:04:30 client-5 kernel: ffff81033e977b30 0000000000000046 ffff8102c2ec44a8 0000000000000008 May 17 21:04:30 client-5 kernel: ffff81032217d440 000000000000000a ffff81033fc0c0c0 ffffffff80308b60 May 17 21:04:30 client-5 kernel: 0000401e1441fdc5 0000000000002389 ffff81033fc0c2a8 00000000020bdfef May 17 21:04:30 client-5 kernel: Call Trace: May 17 21:04:30 client-5 kernel: [] do_gettimeofday+0x40/0x90 May 17 21:04:30 client-5 kernel: [] sync_page+0x0/0x43 May 17 21:04:30 client-5 kernel: [] io_schedule+0x3f/0x67 May 17 21:04:30 client-5 kernel: [] sync_page+0x3e/0x43 May 17 21:04:30 client-5 kernel: [] __wait_on_bit_lock+0x36/0x66 May 17 21:04:30 client-5 kernel: [] __lock_page+0x5e/0x64 May 17 21:04:30 client-5 kernel: [] wake_bit_function+0x0/0x23 May 17 21:04:30 client-5 kernel: [] pagevec_lookup_tag+0x1a/0x21 May 17 21:04:30 client-5 kernel: [] mpage_writepages+0x14f/0x37d May 17 21:04:30 client-5 kernel: [] :lustre:ll_writepage+0x0/0x450 May 17 21:04:30 client-5 kernel: [] __next_cpu+0x19/0x28 May 17 21:04:30 client-5 kernel: [] find_busiest_group+0x20d/0x621 May 17 21:04:30 client-5 kernel: [] do_writepages+0x20/0x2f May 17 21:04:30 client-5 kernel: [] __writeback_single_inode+0x1ae/0x328 May 17 21:04:30 client-5 kernel: [] sync_sb_inodes+0x1b5/0x26f May 17 21:04:30 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:04:30 client-5 kernel: [] writeback_inodes+0x82/0xd8 May 17 21:04:30 client-5 kernel: [] wb_kupdate+0xd4/0x14e May 17 21:04:30 client-5 kernel: [] pdflush+0x0/0x1fb May 17 21:04:30 client-5 kernel: [] pdflush+0x151/0x1fb May 17 21:04:30 client-5 kernel: [] wb_kupdate+0x0/0x14e May 17 21:04:30 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:04:30 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:04:30 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:04:30 client-5 kernel: [] kthread+0x0/0x132 May 17 21:04:30 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:04:30 client-5 kernel: May 17 21:06:30 client-5 kernel: INFO: task pdflush:395 blocked for more than 120 seconds. May 17 21:06:30 client-5 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. May 17 21:06:30 client-5 kernel: pdflush D ffff810001004420 0 395 123 396 394 (L-TLB) May 17 21:06:30 client-5 kernel: ffff81033e977b30 0000000000000046 ffff8102c2ec44a8 0000000000000008 May 17 21:06:30 client-5 kernel: ffff81032217d440 000000000000000a ffff81033fc0c0c0 ffffffff80308b60 May 17 21:06:30 client-5 kernel: 0000401e1441fdc5 0000000000002389 ffff81033fc0c2a8 00000000020bdfef May 17 21:06:30 client-5 kernel: Call Trace: May 17 21:06:30 client-5 kernel: [] do_gettimeofday+0x40/0x90 May 17 21:06:30 client-5 kernel: [] sync_page+0x0/0x43 May 17 21:06:30 client-5 kernel: [] io_schedule+0x3f/0x67 May 17 21:06:30 client-5 kernel: [] sync_page+0x3e/0x43 May 17 21:06:30 client-5 kernel: [] __wait_on_bit_lock+0x36/0x66 May 17 21:06:30 client-5 kernel: [] __lock_page+0x5e/0x64 May 17 21:06:30 client-5 kernel: [] wake_bit_function+0x0/0x23 May 17 21:06:30 client-5 kernel: [] pagevec_lookup_tag+0x1a/0x21 May 17 21:06:30 client-5 kernel: [] mpage_writepages+0x14f/0x37d May 17 21:06:30 client-5 kernel: [] :lustre:ll_writepage+0x0/0x450 May 17 21:06:30 client-5 kernel: [] __next_cpu+0x19/0x28 May 17 21:06:30 client-5 kernel: [] find_busiest_group+0x20d/0x621 May 17 21:06:30 client-5 kernel: [] do_writepages+0x20/0x2f May 17 21:06:30 client-5 kernel: [] __writeback_single_inode+0x1ae/0x328 May 17 21:06:30 client-5 kernel: [] sync_sb_inodes+0x1b5/0x26f May 17 21:06:30 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:06:30 client-5 kernel: [] writeback_inodes+0x82/0xd8 May 17 21:06:30 client-5 kernel: [] wb_kupdate+0xd4/0x14e May 17 21:06:30 client-5 kernel: [] pdflush+0x0/0x1fb May 17 21:06:30 client-5 kernel: [] pdflush+0x151/0x1fb May 17 21:06:30 client-5 kernel: [] wb_kupdate+0x0/0x14e May 17 21:06:30 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:06:30 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:06:30 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:06:30 client-5 kernel: [] kthread+0x0/0x132 May 17 21:06:30 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:06:30 client-5 kernel: May 17 21:19:41 client-5 kernel: ffffffff8003290a>] kthread+0xfe/0x132 May 17 21:19:41 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] kthread+0x0/0x132 May 17 21:19:41 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:19:41 client-5 kernel: May 17 21:19:41 client-5 kernel: kmpathd/2 S ffff810001015120 0 1887 123 1888 1886 (L-TLB) May 17 21:19:41 client-5 kernel: ffff810333e07e70 0000000000000046 0000001c39efe867 0000000000000003 May 17 21:19:41 client-5 kernel: ffff81033b90e080 0000000000000006 ffff81033b90e080 ffff81010b764080 May 17 21:19:41 client-5 kernel: 0000001c39f10293 00000000000008a1 ffff81033b90e268 000000023faccbc0 May 17 21:19:41 client-5 kernel: Call Trace: May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:19:41 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:19:41 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:19:41 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] kthread+0x0/0x132 May 17 21:19:41 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:19:41 client-5 kernel: May 17 21:19:41 client-5 kernel: kmpathd/3 S ffff81000101d7a0 0 1888 123 1889 1887 (L-TLB) May 17 21:19:41 client-5 kernel: ffff810333e09e70 0000000000000046 0000001c39f0fd94 0000000000000003 May 17 21:19:41 client-5 kernel: ffff81033fbb4820 0000000000000005 ffff81033fbb4820 ffff81010b770100 May 17 21:19:41 client-5 kernel: 0000001c39f111ec 00000000000003b0 ffff81033fbb4a08 000000033eac60c0 May 17 21:19:41 client-5 kernel: Call Trace: May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:19:41 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:19:41 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:19:41 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] kthread+0x0/0x132 May 17 21:19:41 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:19:41 client-5 kernel: May 17 21:19:41 client-5 kernel: kmpath_handle S ffff81000101d7a0 0 1889 123 2114 1888 (L-TLB) May 17 21:19:41 client-5 kernel: ffff810333e0be70 0000000000000046 0000001c39f1373a 0000000000000003 May 17 21:19:41 client-5 kernel: ffff81033ab037e0 0000000000000004 ffff81033ab037e0 ffff81010b770100 May 17 21:19:41 client-5 kernel: 0000001c39f14a96 000000000000033a ffff81033ab039c8 000000033eac60c0 May 17 21:19:41 client-5 kernel: Call Trace: May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:19:41 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:19:41 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:19:41 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] kthread+0x0/0x132 May 17 21:19:41 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:19:41 client-5 kernel: May 17 21:19:41 client-5 kernel: mthcacatas S ffff81000101d7a0 0 2114 123 2132 1889 (L-TLB) May 17 21:19:41 client-5 kernel: ffff810332403e70 0000000000000046 0000001c8b232caf 0000000000000003 May 17 21:19:41 client-5 kernel: ffff81033bb43820 0000000000000009 ffff81033bb43820 ffff81010b770100 May 17 21:19:41 client-5 kernel: 0000001c8b241d5f 0000000000000551 ffff81033bb43a08 000000033b464880 May 17 21:19:41 client-5 kernel: Call Trace: May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:19:41 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:19:41 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:19:41 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] kthread+0x0/0x132 May 17 21:19:41 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:19:41 client-5 kernel: May 17 21:19:41 client-5 kernel: mlx4_en S ffff81000101d7a0 0 2132 123 2176 2114 (L-TLB) May 17 21:19:41 client-5 kernel: ffff810332757e70 0000000000000046 0000001c8f01669c 0000000000000003 May 17 21:19:41 client-5 kernel: ffff81033b1580c0 0000000000000009 ffff81033b1580c0 ffff81010b770100 May 17 21:19:41 client-5 kernel: 0000001c8f0196a3 0000000000000424 ffff81033b1582a8 000000033be7bbc0 May 17 21:19:41 client-5 kernel: Call Trace: May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:19:41 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:19:41 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:19:41 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] kthread+0x0/0x132 May 17 21:19:41 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:19:41 client-5 kernel: May 17 21:19:41 client-5 kernel: iw_cxgb3 S 0000000000000282 0 2176 123 2185 2132 (L-TLB) May 17 21:19:41 client-5 kernel: ffff81033f677e70 0000000000000046 0000001c97d61818 0000000000000003 May 17 21:19:41 client-5 kernel: ffff81033fbad100 0000000000000008 ffff81033fbad100 ffff81033fb787e0 May 17 21:19:41 client-5 kernel: 0000001c97d62416 00000000000002c7 ffff81033fbad2e8 0000000300000000 May 17 21:19:41 client-5 kernel: Call Trace: May 17 21:19:41 client-5 kernel: [] thread_return+0x62/0xfe May 17 21:19:41 client-5 kernel: [] recalc_sigpending_and_wake+0x9/0x1a May 17 21:19:41 client-5 kernel: [] do_sigaction+0x14d/0x199 May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:19:41 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:19:41 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:19:41 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] kthread+0x0/0x132 May 17 21:19:41 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:19:41 client-5 kernel: May 17 21:19:41 client-5 kernel: nesewq S ffff81000101d7a0 0 2185 123 2186 2176 (L-TLB) May 17 21:19:41 client-5 kernel: ffff8103316ade70 0000000000000046 0000001c9a6256b4 0000000000000003 May 17 21:19:41 client-5 kernel: ffff81033b9567a0 0000000000000006 ffff81033b9567a0 ffff81010b770100 May 17 21:19:41 client-5 kernel: 0000001c9a629af2 000000000000054b ffff81033b956988 000000033a993b80 May 17 21:19:41 client-5 kernel: Call Trace: May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:19:41 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:19:41 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:19:41 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] kthread+0x0/0x132 May 17 21:19:41 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:19:41 client-5 kernel: May 17 21:19:41 client-5 kernel: nesdwq S ffff81000101d7a0 0 2186 123 2188 2185 (L-TLB) May 17 21:19:41 client-5 kernel: ffff8103316afe70 0000000000000046 0000001c9a62d44f 0000000000000003 May 17 21:19:41 client-5 kernel: ffff81033ba9a100 0000000000000006 ffff81033ba9a100 ffff81010b770100 May 17 21:19:41 client-5 kernel: 0000001c9a6323ed 000000000000035d ffff81033ba9a2e8 000000033a993b80 May 17 21:19:41 client-5 kernel: Call Trace: May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:19:41 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:19:41 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:19:41 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] kthread+0x0/0x132 May 17 21:19:41 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:19:41 client-5 kernel: May 17 21:19:41 client-5 kernel: nes_ud S ffff81000101d7a0 0 2188 123 2244 2186 (L-TLB) May 17 21:19:41 client-5 kernel: ffff81033118fe70 0000000000000046 0000001c9a680d59 0000000000000003 May 17 21:19:41 client-5 kernel: ffff81033fb5f100 0000000000000005 ffff81033fb5f100 ffff81010b770100 May 17 21:19:41 client-5 kernel: 0000001c9a6822a2 00000000000002f8 ffff81033fb5f2e8 000000033a993b80 May 17 21:19:41 client-5 kernel: Call Trace: May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:19:41 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:19:41 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:19:41 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] kthread+0x0/0x132 May 17 21:19:41 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:19:41 client-5 kernel: May 17 21:19:41 client-5 kernel: ib_mcast S 0000000000000282 0 2244 123 2245 2188 (L-TLB) May 17 21:19:41 client-5 kernel: ffff810331209e70 0000000000000046 ffff81033cfc5a00 ffff81033cfc5a60 May 17 21:19:41 client-5 kernel: 0000000300000000 000000000000000a ffff81033bd96820 ffff81033b90e7e0 May 17 21:19:41 client-5 kernel: 00000020516862a2 0000000000003813 ffff81033bd96a08 000000008002e23f May 17 21:19:41 client-5 kernel: Call Trace: May 17 21:19:41 client-5 kernel: [] :ib_sa:mcast_work_handler+0x0/0x4e5 May 17 21:19:41 client-5 kernel: [] run_workqueue+0xba/0xe4 May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:19:41 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:19:41 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:19:41 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] kthread+0x0/0x132 May 17 21:19:41 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:19:41 client-5 kernel: May 17 21:19:41 client-5 kernel: ib_inform S 0000000000000282 0 2245 123 2246 2244 (L-TLB) May 17 21:19:41 client-5 kernel: ffff81033120be70 0000000000000046 0000001ca52755ce 0000000000000003 May 17 21:19:41 client-5 kernel: ffff81033ab1b100 0000000000000006 ffff81033ab1b100 ffff81033bf037a0 May 17 21:19:41 client-5 kernel: 0000001ca5275d8d 000000000000018e ffff81033ab1b2e8 0000000300000000 May 17 21:19:41 client-5 kernel: Call Trace: May 17 21:19:41 client-5 kernel: [] thread_return+0x62/0xfe May 17 21:19:41 client-5 kernel: [] recalc_sigpending_and_wake+0x9/0x1a May 17 21:19:41 client-5 kernel: [] do_sigaction+0x14d/0x199 May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:19:41 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:19:41 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:19:41 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] kthread+0x0/0x132 May 17 21:19:41 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:19:41 client-5 kernel: May 17 21:19:41 client-5 kernel: local_sa S ffff81000101d7a0 0 2246 123 2252 2245 (L-TLB) May 17 21:19:41 client-5 kernel: ffff810331da1e70 0000000000000046 0000001ca5278521 0000000000000003 May 17 21:19:41 client-5 kernel: 0000000300000000 000000000000000a ffff81033fc127a0 ffff81010b770100 May 17 21:19:41 client-5 kernel: 0000001fab5ac812 00000000000013c7 ffff81033fc12988 000000038002e23f May 17 21:19:41 client-5 kernel: Call Trace: May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:19:41 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:19:41 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:19:41 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] kthread+0x0/0x132 May 17 21:19:41 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:19:41 client-5 kernel: May 17 21:19:41 client-5 kernel: ib_cm/0 S ffff81033ae7b7e0 0 2252 123 2253 2246 (L-TLB) May 17 21:19:41 client-5 kernel: ffff81033132be70 0000000000000046 ffff8103113205c0 ffffffff8850104a May 17 21:19:41 client-5 kernel: 0000000300000000 000000000000000a ffff81033aba40c0 ffff81033ae7b7e0 May 17 21:19:41 client-5 kernel: 000033688079addc 0000000000004d8b ffff81033aba42a8 000000008002e23f May 17 21:19:41 client-5 kernel: Call Trace: May 17 21:19:41 client-5 kernel: [] :ib_cm:cm_rep_handler+0x3ac/0x3ec May 17 21:19:41 client-5 kernel: [] :ib_cm:cm_work_handler+0x0/0xb0f May 17 21:19:41 client-5 kernel: [] run_workqueue+0xba/0xe4 May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:19:41 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:19:41 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:19:41 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] kthread+0x0/0x132 May 17 21:19:41 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:19:41 client-5 kernel: May 17 21:19:41 client-5 kernel: ib_cm/1 S ffff81033ae7b7e0 0 2253 123 2254 2252 (L-TLB) May 17 21:19:41 client-5 kernel: ffff81033132de70 0000000000000046 ffff81030fa07d80 ffffffff8850104a May 17 21:19:41 client-5 kernel: 0000000300000000 0000000000000009 ffff81033fbaf7a0 ffff81033ae7b7e0 May 17 21:19:41 client-5 kernel: 0000400e244fe122 0000000000006048 ffff81033fbaf988 000000018002e23f May 17 21:19:41 client-5 kernel: Call Trace: May 17 21:19:41 client-5 kernel: [] :ib_cm:cm_rep_handler+0x3ac/0x3ec May 17 21:19:41 client-5 kernel: [] :ib_cm:cm_work_handler+0x0/0xb0f May 17 21:19:41 client-5 kernel: [] run_workqueue+0xba/0xe4 May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:19:41 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:19:41 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:19:41 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] kthread+0x0/0x132 May 17 21:19:41 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:19:41 client-5 kernel: May 17 21:19:41 client-5 kernel: ib_cm/2 S ffff810001015120 0 2254 123 2255 2253 (L-TLB) May 17 21:19:41 client-5 kernel: ffff81033132fe70 0000000000000046 ffff81032349ab80 ffffffff8850104a May 17 21:19:41 client-5 kernel: 0000000300000000 000000000000000a ffff81033bb84860 ffff81010b764080 May 17 21:19:41 client-5 kernel: 0000300fdf89d3f9 00000000000046d3 ffff81033bb84a48 000000028002e23f May 17 21:19:41 client-5 kernel: Call Trace: May 17 21:19:41 client-5 kernel: [] :ib_cm:cm_rep_handler+0x3ac/0x3ec May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:19:41 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:19:41 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:19:41 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] kthread+0x0/0x132 May 17 21:19:41 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:19:41 client-5 kernel: May 17 21:19:41 client-5 kernel: ib_cm/3 S ffff81033ae7b7e0 0 2255 123 2272 2254 (L-TLB) May 17 21:19:41 client-5 kernel: ffff810331331e70 0000000000000046 ffff81031e3278c0 ffffffff8850104a May 17 21:19:41 client-5 kernel: 0000000300000000 000000000000000a ffff81033b0407e0 ffff81033ae7b7e0 May 17 21:19:41 client-5 kernel: 0000385250d68cb5 000000000000626e ffff81033b0409c8 000000038002e23f May 17 21:19:41 client-5 kernel: Call Trace: May 17 21:19:41 client-5 kernel: [] :ib_cm:cm_rep_handler+0x3ac/0x3ec May 17 21:19:41 client-5 kernel: [] :ib_cm:cm_work_handler+0x0/0xb0f May 17 21:19:41 client-5 kernel: [] run_workqueue+0xba/0xe4 May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:19:41 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:19:41 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:19:41 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] kthread+0x0/0x132 May 17 21:19:41 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:19:41 client-5 kernel: May 17 21:19:41 client-5 kernel: ipoib S ffff81000100caa0 0 2272 123 2382 2255 (L-TLB) May 17 21:19:41 client-5 kernel: ffff8103308f1e70 0000000000000046 ffff8103308f2f68 0000000000000286 May 17 21:19:41 client-5 kernel: 0000000300000000 000000000000000a ffff81033bb84100 ffff81010b70e100 May 17 21:19:41 client-5 kernel: 000041f1c4b3c73d 0000000000000aba ffff81033bb842e8 000000018002e23f May 17 21:19:41 client-5 kernel: Call Trace: May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:19:41 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:19:41 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:19:41 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] kthread+0x0/0x132 May 17 21:19:41 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:19:41 client-5 kernel: May 17 21:19:41 client-5 kernel: ib_addr S ffff81000100caa0 0 2382 123 2388 2272 (L-TLB) May 17 21:19:41 client-5 kernel: ffff81032a949e70 0000000000000046 0000000000000000 0000000000000000 May 17 21:19:41 client-5 kernel: 0000000300000000 000000000000000a ffff81033b304860 ffff81010b70e100 May 17 21:19:41 client-5 kernel: 000041eeed7daf1a 0000000000000897 ffff81033b304a48 000000018002e23f May 17 21:19:41 client-5 kernel: Call Trace: May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:19:41 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:19:41 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:19:41 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] kthread+0x0/0x132 May 17 21:19:41 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:19:41 client-5 kernel: May 17 21:19:41 client-5 kernel: iw_cm_wq S ffff81000101d7a0 0 2388 123 2393 2382 (L-TLB) May 17 21:19:41 client-5 kernel: ffff81032a931e70 0000000000000046 0000001de92c58d9 0000000000000003 May 17 21:19:41 client-5 kernel: ffff81033aca80c0 0000000000000008 ffff81033aca80c0 ffff81010b770100 May 17 21:19:41 client-5 kernel: 0000001de92c7102 00000000000004d8 ffff81033aca82a8 000000033bb63400 May 17 21:19:41 client-5 kernel: Call Trace: May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:19:41 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:19:41 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:19:41 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] kthread+0x0/0x132 May 17 21:19:41 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:19:41 client-5 kernel: May 17 21:19:41 client-5 kernel: rdma_cm S ffff810001004420 0 2393 123 2400 2388 (L-TLB) May 17 21:19:41 client-5 kernel: ffff81032aaf5e70 0000000000000046 0000001de99294a5 0000000000000003 May 17 21:19:41 client-5 kernel: 0000000300000000 000000000000000a ffff81033fb320c0 ffffffff80308b60 May 17 21:19:41 client-5 kernel: 0000400e24324b35 0000000000005d86 ffff81033fb322a8 000000008002e23f May 17 21:19:41 client-5 kernel: Call Trace: May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:19:41 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:19:41 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:19:41 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] kthread+0x0/0x132 May 17 21:19:41 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:19:41 client-5 kernel: May 17 21:19:41 client-5 kernel: rx_comp_wq/0 S 0000000000000282 0 2400 123 2401 2393 (L-TLB) May 17 21:19:41 client-5 kernel: ffff81032a24fe70 0000000000000046 0000001deb854d91 0000000000000003 May 17 21:19:41 client-5 kernel: ffff81033aa460c0 0000000000000008 ffff81033aa460c0 ffff81033c5467e0 May 17 21:19:41 client-5 kernel: 0000001deb85db3a 0000000000000554 ffff81033aa462a8 0000000000000000 May 17 21:19:41 client-5 kernel: Call Trace: May 17 21:19:41 client-5 kernel: [] thread_return+0x62/0xfe May 17 21:19:41 client-5 kernel: [] recalc_sigpending_and_wake+0x9/0x1a May 17 21:19:41 client-5 kernel: [] do_sigaction+0x14d/0x199 May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:19:41 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:19:41 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:19:41 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] kthread+0x0/0x132 May 17 21:19:41 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:19:41 client-5 kernel: May 17 21:19:41 client-5 kernel: rx_comp_wq/1 S ffff81000100caa0 0 2401 123 2402 2400 (L-TLB) May 17 21:19:41 client-5 kernel: ffff81032a151e70 0000000000000046 0000001deb85f0d8 0000000000000003 May 17 21:19:41 client-5 kernel: ffff81033b6f6080 0000000000000006 ffff81033b6f6080 ffff81010b70e100 May 17 21:19:41 client-5 kernel: 0000001deb86cd6a 000000000000082e ffff81033b6f6268 000000013eac60c0 May 17 21:19:41 client-5 kernel: Call Trace: May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:19:41 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:19:41 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:19:41 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] kthread+0x0/0x132 May 17 21:19:41 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:19:41 client-5 kernel: May 17 21:19:41 client-5 kernel: rx_comp_wq/2 S ffff810001015120 0 2402 123 2403 2401 (L-TLB) May 17 21:19:41 client-5 kernel: ffff81032a153e70 0000000000000046 0000001deb86cf2f 0000000000000003 May 17 21:19:41 client-5 kernel: ffff81033ad4f100 0000000000000006 ffff81033ad4f100 ffff81010b764080 May 17 21:19:41 client-5 kernel: 0000001deb87ed98 0000000000000a5c ffff81033ad4f2e8 000000023faccbc0 May 17 21:19:41 client-5 kernel: Call Trace: May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:19:41 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:19:41 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:19:41 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] kthread+0x0/0x132 May 17 21:19:41 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:19:41 client-5 kernel: May 17 21:19:41 client-5 kernel: rx_comp_wq/3 S ffff81000101d7a0 0 2403 123 2404 2402 (L-TLB) May 17 21:19:41 client-5 kernel: ffff81032a155e70 0000000000000046 0000001deb87f978 0000000000000003 May 17 21:19:41 client-5 kernel: ffff81033fb39040 0000000000000007 ffff81033fb39040 ffff81010b770100 May 17 21:19:41 client-5 kernel: 0000001deb8813e3 000000000000049f ffff81033fb39228 000000033bb63400 May 17 21:19:41 client-5 kernel: Call Trace: May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:19:41 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:19:41 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:19:41 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] kthread+0x0/0x132 May 17 21:19:41 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:19:41 client-5 kernel: May 17 21:19:41 client-5 kernel: sdp_wq S ffff81000101d7a0 0 2404 123 2405 2403 (L-TLB) May 17 21:19:41 client-5 kernel: ffff81032a157e70 0000000000000046 0000001deb885379 0000000000000003 May 17 21:19:41 client-5 kernel: ffff81033fbad860 0000000000000007 ffff81033fbad860 ffff81010b770100 May 17 21:19:41 client-5 kernel: 0000001deb88667c 0000000000000323 ffff81033fbada48 000000033bb63400 May 17 21:19:41 client-5 kernel: Call Trace: May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:19:41 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:19:41 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:19:41 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] kthread+0x0/0x132 May 17 21:19:41 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:19:41 client-5 kernel: May 17 21:19:41 client-5 kernel: ib_fmr(mlx4_0 S ffff81000101d7a0 0 2405 123 2412 2404 (L-TLB) May 17 21:19:41 client-5 kernel: ffff81032a141ed0 0000000000000046 ffff81032a141ee0 ffffffff80062ff8 May 17 21:19:41 client-5 kernel: ffff81033c46a100 0000000000000007 ffff81033c46a100 ffff81010b770100 May 17 21:19:41 client-5 kernel: 0000001deb8a82f5 000000000000040f ffff81033c46a2e8 00000003000003a2 May 17 21:19:41 client-5 kernel: Call Trace: May 17 21:19:41 client-5 kernel: [] thread_return+0x62/0xfe May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] :ib_core:ib_fmr_cleanup_thread+0x0/0x87 May 17 21:19:41 client-5 kernel: [] :ib_core:ib_fmr_cleanup_thread+0x6a/0x87 May 17 21:19:41 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:19:41 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] kthread+0x0/0x132 May 17 21:19:41 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:19:41 client-5 kernel: May 17 21:19:41 client-5 kernel: qlgc_vnic_vip S 0000000000000282 0 2412 123 2413 2405 (L-TLB) May 17 21:19:41 client-5 kernel: ffff81032a2afe70 0000000000000046 ffff81010b7e4068 000000000000003d May 17 21:19:41 client-5 kernel: ffff81032a2ae000 0000000000000007 ffff81033fbb1080 ffff81033c7d90c0 May 17 21:19:41 client-5 kernel: 0000001ded911df2 0000000000000d87 ffff81033fbb1268 000000033f521100 May 17 21:19:41 client-5 kernel: Call Trace: May 17 21:19:41 client-5 kernel: [] thread_return+0x62/0xfe May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] :qlgc_vnic:viport_statemachine+0x0/0x138a May 17 21:19:41 client-5 kernel: [] :qlgc_vnic:viport_statemachine+0xc3/0x138a May 17 21:19:41 client-5 kernel: [] autoremove_wake_function+0x0/0x2e May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] :qlgc_vnic:viport_statemachine+0x0/0x138a May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:19:41 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] kthread+0x0/0x132 May 17 21:19:41 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:19:41 client-5 kernel: May 17 21:19:41 client-5 kernel: qlgc_vnic_npe S ffff81000101d7a0 0 2413 123 2471 2412 (L-TLB) May 17 21:19:41 client-5 kernel: ffff810329c0de80 0000000000000046 ffff81033c7d90c0 ffffffff8008c787 May 17 21:19:41 client-5 kernel: ffff81033c7d90c0 0000000000000007 ffff81033c7d90c0 ffff81010b770100 May 17 21:19:41 client-5 kernel: 0000001ded918289 0000000000000361 ffff81033c7d92a8 000000033f521100 May 17 21:19:41 client-5 kernel: Call Trace: May 17 21:19:41 client-5 kernel: [] dequeue_task+0x18/0x37 May 17 21:19:41 client-5 kernel: [] thread_return+0x62/0xfe May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] :qlgc_vnic:vnic_npevent_statemachine+0x0/0xa85 May 17 21:19:41 client-5 kernel: [] :qlgc_vnic:vnic_npevent_statemachine+0x9b/0xa85 May 17 21:19:41 client-5 kernel: [] autoremove_wake_function+0x0/0x2e May 17 21:19:41 client-5 kernel: [] :qlgc_vnic:vnic_npevent_statemachine+0x0/0xa85 May 17 21:19:41 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:19:41 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] kthread+0x0/0x132 May 17 21:19:41 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:19:41 client-5 kernel: May 17 21:19:41 client-5 kernel: iscsi_eh S ffff81000101d7a0 0 2471 123 3068 2413 (L-TLB) May 17 21:19:41 client-5 kernel: ffff81032a475e70 0000000000000046 0000001df2588751 0000000000000003 May 17 21:19:41 client-5 kernel: ffff81033fbb8100 0000000000000007 ffff81033fbb8100 ffff81010b770100 May 17 21:19:41 client-5 kernel: 0000001df258da9c 000000000000064e ffff81033fbb82e8 000000033c4fd8c0 May 17 21:19:41 client-5 kernel: Call Trace: May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:19:41 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:19:41 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:19:41 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] kthread+0x0/0x132 May 17 21:19:41 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:19:41 client-5 kernel: May 17 21:19:41 client-5 kernel: brcm_iscsiuio S ffff81033eb71098 0 2526 1 2527 692 (NOTLB) May 17 21:19:41 client-5 kernel: ffff81032a2a3a98 0000000000000086 0000000000000000 ffff8103338adf70 May 17 21:19:41 client-5 kernel: ffff810331b406c0 0000000000000001 ffff81033ab03080 ffff81033b49b820 May 17 21:19:41 client-5 kernel: 0000001df98a3de9 000000000004a2e1 ffff81033ab03268 0000000080019c08 May 17 21:19:41 client-5 kernel: Call Trace: May 17 21:19:41 client-5 kernel: [] schedule_timeout+0x1e/0xad May 17 21:19:41 client-5 kernel: [] prepare_to_wait_exclusive+0x38/0x65 May 17 21:19:41 client-5 kernel: [] __skb_recv_datagram+0x193/0x215 May 17 21:19:41 client-5 kernel: [] autoremove_wake_function+0x0/0x2e May 17 21:19:41 client-5 kernel: [] :jbd:journal_stop+0x1f3/0x1ff May 17 21:19:41 client-5 kernel: [] skb_recv_datagram+0x1c/0x21 May 17 21:19:41 client-5 kernel: [] :ext3:ext3_ordered_write_end+0xd7/0x116 May 17 21:19:41 client-5 kernel: [] netlink_recvmsg+0x47/0x290 May 17 21:19:41 client-5 kernel: [] generic_file_buffered_write+0x20b/0x675 May 17 21:19:41 client-5 kernel: [] zone_statistics+0x3e/0x6d May 17 21:19:41 client-5 kernel: [] sock_recvmsg+0x107/0x15f May 17 21:19:41 client-5 kernel: [] autoremove_wake_function+0x0/0x2e May 17 21:19:41 client-5 kernel: [] do_wp_page+0x87c/0x902 May 17 21:19:41 client-5 kernel: [] find_get_page+0x21/0x51 May 17 21:19:41 client-5 kernel: [] sys_recvmsg+0x15c/0x24c May 17 21:19:41 client-5 kernel: [] __up_read+0x19/0x7f May 17 21:19:41 client-5 kernel: [] do_page_fault+0x4fe/0x874 May 17 21:19:41 client-5 kernel: [] sys_newstat+0x28/0x31 May 17 21:19:41 client-5 kernel: [] system_call+0x7e/0x83 May 17 21:19:41 client-5 kernel: May 17 21:19:41 client-5 kernel: brcm_iscsiuio S 0000000000000000 0 2527 1 2528 2526 (NOTLB) May 17 21:19:41 client-5 kernel: ffff81032a537e58 0000000000000086 ffff81032a537e48 ffffffff8000eaaf May 17 21:19:41 client-5 kernel: ffff81033968d000 0000000000000001 ffff81033a4dd860 ffff81033fb397a0 May 17 21:19:41 client-5 kernel: 0000001df98b14a1 0000000000004214 ffff81033a4dda48 000000003968d000 May 17 21:19:41 client-5 kernel: Call Trace: May 17 21:19:41 client-5 kernel: [] link_path_walk+0xa6/0xb2 May 17 21:19:41 client-5 kernel: [] _atomic_dec_and_lock+0x39/0x57 May 17 21:19:41 client-5 kernel: [] mntput_no_expire+0x19/0x89 May 17 21:19:41 client-5 kernel: [] schedule_timeout+0x1e/0xad May 17 21:19:41 client-5 kernel: [] recalc_sigpending+0xe/0x25 May 17 21:19:41 client-5 kernel: [] dequeue_signal+0x47/0xcd May 17 21:19:41 client-5 kernel: [] sys_rt_sigtimedwait+0x1ce/0x2b4 May 17 21:19:41 client-5 kernel: [] sys_newstat+0x28/0x31 May 17 21:19:41 client-5 kernel: [] system_call+0x7e/0x83 May 17 21:19:41 client-5 kernel: May 17 21:19:41 client-5 kernel: brcm_iscsiuio S ffff810001004420 0 2528 1 2531 2527 (NOTLB) May 17 21:19:41 client-5 kernel: ffff810329c2fd78 0000000000000086 00000000ffffffff 0000000000000000 May 17 21:19:41 client-5 kernel: ffff8103a9c2fe47 0000000000000001 ffff81033fb397a0 ffffffff80308b60 May 17 21:19:41 client-5 kernel: 0000001df98b4127 0000000000002c86 ffff81033fb39988 00000000802b278f May 17 21:19:41 client-5 kernel: Call Trace: May 17 21:19:41 client-5 kernel: [] schedule_timeout+0x1e/0xad May 17 21:19:41 client-5 kernel: [] prepare_to_wait_exclusive+0x38/0x65 May 17 21:19:41 client-5 kernel: [] __skb_recv_datagram+0x193/0x215 May 17 21:19:41 client-5 kernel: [] autoremove_wake_function+0x0/0x2e May 17 21:19:41 client-5 kernel: [] inotify_d_instantiate+0x12/0x3c May 17 21:19:41 client-5 kernel: [] skb_recv_datagram+0x1c/0x21 May 17 21:19:41 client-5 kernel: [] unix_accept+0x54/0x103 May 17 21:19:41 client-5 kernel: [] sys_accept+0x11c/0x1ea May 17 21:19:41 client-5 kernel: [] mntput_no_expire+0x19/0x89 May 17 21:19:41 client-5 kernel: [] filp_close+0x5c/0x64 May 17 21:19:41 client-5 kernel: [] recalc_sigpending+0xe/0x25 May 17 21:19:41 client-5 kernel: [] sigprocmask+0xb7/0xdb May 17 21:19:41 client-5 kernel: [] system_call+0x7e/0x83 May 17 21:19:41 client-5 kernel: May 17 21:19:41 client-5 kernel: iscsid S ffff81000101d7a0 0 2531 1 2532 2528 (NOTLB) May 17 21:19:41 client-5 kernel: ffff81032a2ffeb8 0000000000000086 ffff81032a2ffe58 ffff81000101f880 May 17 21:19:41 client-5 kernel: 000000003b9aca00 0000000000000007 ffff81033ba9a860 ffff81010b770100 May 17 21:19:41 client-5 kernel: 000041f0631f89b4 00000000000013af ffff81033ba9aa48 00000003d38531df May 17 21:19:41 client-5 kernel: Call Trace: May 17 21:19:41 client-5 kernel: [] enqueue_hrtimer+0x55/0x70 May 17 21:19:41 client-5 kernel: [] do_nanosleep+0x3f/0x70 May 17 21:19:41 client-5 kernel: [] hrtimer_nanosleep+0x58/0x118 May 17 21:19:41 client-5 kernel: [] hrtimer_wakeup+0x0/0x22 May 17 21:19:41 client-5 kernel: [] sigprocmask+0xb7/0xdb May 17 21:19:41 client-5 kernel: [] sys_nanosleep+0x4c/0x62 May 17 21:19:41 client-5 kernel: [] system_call+0x7e/0x83 May 17 21:19:41 client-5 kernel: May 17 21:19:41 client-5 kernel: iscsid S ffff81000100caa0 0 2532 1 2995 2531 (NOTLB) May 17 21:19:41 client-5 kernel: ffff81032a35db38 0000000000000086 0000000000582657 ffff81032a35db48 May 17 21:19:41 client-5 kernel: ffff81032a977830 000000000000000a ffff81033b49b820 ffff81010b70e100 May 17 21:19:41 client-5 kernel: 000041f211a4769a 000000000000149d ffff81033b49ba08 000000012a35de34 May 17 21:19:41 client-5 kernel: Call Trace: May 17 21:19:41 client-5 kernel: [] unix_poll+0x0/0x99 May 17 21:19:41 client-5 kernel: [] schedule_timeout+0x8a/0xad May 17 21:19:41 client-5 kernel: [] process_timeout+0x0/0x5 May 17 21:19:41 client-5 kernel: [] do_sys_poll+0x277/0x360 May 17 21:19:41 client-5 kernel: [] __pollwait+0x0/0xe2 May 17 21:19:41 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:41 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:41 client-5 kernel: [] __rmqueue+0x91/0xc6 May 17 21:19:41 client-5 kernel: [] vsnprintf+0x5e7/0x62f May 17 21:19:41 client-5 kernel: [] snprintf+0x44/0x4c May 17 21:19:41 client-5 kernel: [] d_lookup+0x1e/0x42 May 17 21:19:41 client-5 kernel: [] proc_flush_task+0x4e/0x1f6 May 17 21:19:41 client-5 kernel: [] sched_exit+0xa9/0xb5 May 17 21:19:41 client-5 kernel: [] release_task+0x3a5/0x3cb May 17 21:19:41 client-5 kernel: [] remove_wait_queue+0x1c/0x2c May 17 21:19:41 client-5 kernel: [] do_wait+0xa3b/0xad7 May 17 21:19:41 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:41 client-5 kernel: [] sys_poll+0x2d/0x34 May 17 21:19:41 client-5 kernel: [] system_call+0x7e/0x83 May 17 21:19:41 client-5 kernel: May 17 21:19:41 client-5 kernel: auditd S ffff81000101d7a0 0 2995 1 2997 2996 2532 (NOTLB) May 17 21:19:41 client-5 kernel: ffff8103317d7e78 0000000000000082 ffff810300000000 ffff8103317d7c28 May 17 21:19:41 client-5 kernel: ffff8103317d7ee8 000000000000000a ffff81033aba4820 ffff81010b770100 May 17 21:19:41 client-5 kernel: 000041e9c2469db7 0000000000002c45 ffff81033aba4a08 00000003317d7e60 May 17 21:19:41 client-5 kernel: Call Trace: May 17 21:19:41 client-5 kernel: [] schedule_timeout+0x8a/0xad May 17 21:19:41 client-5 kernel: [] process_timeout+0x0/0x5 May 17 21:19:41 client-5 kernel: [] sys_epoll_wait+0x184/0x3f9 May 17 21:19:41 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:41 client-5 kernel: [] ktime_get_ts+0x1a/0x4e May 17 21:19:41 client-5 kernel: [] system_call+0x7e/0x83 May 17 21:19:41 client-5 kernel: May 17 21:19:41 client-5 kernel: auditd S ffff81000100caa0 0 2996 1 3017 2995 (NOTLB) May 17 21:19:41 client-5 kernel: ffff81032a0cfd28 0000000000000082 0000000000000000 0000000000000000 May 17 21:19:41 client-5 kernel: ffff81032a0cfd98 000000000000000a ffff81033a646820 ffff81010b70e100 May 17 21:19:41 client-5 kernel: 000040efcc157d35 0000000000007c4d ffff81033a646a08 00000001000000b6 May 17 21:19:41 client-5 kernel: Call Trace: May 17 21:19:41 client-5 kernel: [] schedule_timeout+0x1e/0xad May 17 21:19:41 client-5 kernel: [] wait_on_page_writeback_range+0xd6/0x12e May 17 21:19:41 client-5 kernel: [] add_wait_queue+0x24/0x34 May 17 21:19:41 client-5 kernel: [] do_futex+0x25e/0xcf3 May 17 21:19:41 client-5 kernel: [] __next_cpu+0x19/0x28 May 17 21:19:41 client-5 kernel: [] autoremove_wake_function+0x0/0x2e May 17 21:19:41 client-5 kernel: [] :ext3:ext3_statfs+0xf6/0x113 May 17 21:19:41 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:41 client-5 kernel: [] sys_fstatfs+0x57/0x68 May 17 21:19:41 client-5 kernel: [] sys_futex+0x10a/0x12b May 17 21:19:41 client-5 kernel: [] system_call+0x7e/0x83 May 17 21:19:41 client-5 kernel: May 17 21:19:41 client-5 kernel: audispd S ffff810001004420 0 2997 2995 2998 (NOTLB) May 17 21:19:41 client-5 kernel: ffff810329ad9d28 0000000000000082 ffffffff800cc235 ffff810329ad9d28 May 17 21:19:41 client-5 kernel: 0000000000000206 000000000000000a ffff81033ad91100 ffffffff80308b60 May 17 21:19:41 client-5 kernel: 000040efcc139a6c 0000000000001e5c ffff81033ad912e8 0000000000000bb5 May 17 21:19:41 client-5 kernel: Call Trace: May 17 21:19:41 client-5 kernel: [] zone_statistics+0x3e/0x6d May 17 21:19:41 client-5 kernel: [] schedule_timeout+0x1e/0xad May 17 21:19:41 client-5 kernel: [] add_wait_queue+0x24/0x34 May 17 21:19:41 client-5 kernel: [] do_futex+0x25e/0xcf3 May 17 21:19:41 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:41 client-5 kernel: [] __sched_text_start+0xf6/0xbd6 May 17 21:19:41 client-5 kernel: [] sys_futex+0x10a/0x12b May 17 21:19:41 client-5 kernel: [] system_call+0x7e/0x83 May 17 21:19:41 client-5 kernel: May 17 21:19:41 client-5 kernel: audispd S ffff810001004420 0 2998 2995 2997 (NOTLB) May 17 21:19:41 client-5 kernel: ffff810332333b38 0000000000000082 ffff8102fc0f2400 000000000000000a May 17 21:19:41 client-5 kernel: ffff81033fc017e0 000000000000000a ffff81033fc017e0 ffffffff80308b60 May 17 21:19:41 client-5 kernel: 000041ef7ba5e32d 0000000000001ddc ffff81033fc019c8 000000002c352e34 May 17 21:19:41 client-5 kernel: Call Trace: May 17 21:19:41 client-5 kernel: [] unix_poll+0x0/0x99 May 17 21:19:41 client-5 kernel: [] schedule_timeout+0x8a/0xad May 17 21:19:41 client-5 kernel: [] process_timeout+0x0/0x5 May 17 21:19:41 client-5 kernel: [] do_sys_poll+0x277/0x360 May 17 21:19:41 client-5 kernel: [] __pollwait+0x0/0xe2 May 17 21:19:41 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:41 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:41 client-5 kernel: [] try_to_wake_up+0x472/0x484 May 17 21:19:41 client-5 kernel: [] sock_readv+0xb7/0xd1 May 17 21:19:41 client-5 kernel: [] zone_statistics+0x3e/0x6d May 17 21:19:41 client-5 kernel: [] __wake_up_common+0x3e/0x68 May 17 21:19:41 client-5 kernel: [] __wake_up+0x38/0x4f May 17 21:19:41 client-5 kernel: [] wake_futex+0x39/0x43 May 17 21:19:41 client-5 kernel: [] do_futex+0x7b7/0xcf3 May 17 21:19:41 client-5 kernel: [] do_readv_writev+0x26e/0x291 May 17 21:19:41 client-5 kernel: [] sys_futex+0x10a/0x12b May 17 21:19:41 client-5 kernel: [] sys_poll+0x2d/0x34 May 17 21:19:41 client-5 kernel: [] system_call+0x7e/0x83 May 17 21:19:41 client-5 kernel: May 17 21:19:41 client-5 kernel: syslogd S ffff81000101d7a0 0 3017 1 3020 2996 (NOTLB) May 17 21:19:41 client-5 kernel: ffff810329cdda18 0000000000000086 ffff810329e64910 ffffffff880312e8 May 17 21:19:41 client-5 kernel: ffff810329e64910 000000000000000a ffff81033c46a860 ffff81010b770100 May 17 21:19:41 client-5 kernel: 000041eaabe46efb 0000000000004d4f ffff81033c46aa48 0000000300000000 May 17 21:19:41 client-5 kernel: Call Trace: May 17 21:19:41 client-5 kernel: [] :jbd:__journal_file_buffer+0x13e/0x243 May 17 21:19:41 client-5 kernel: [] schedule_timeout+0x1e/0xad May 17 21:19:41 client-5 kernel: [] add_wait_queue+0x24/0x34 May 17 21:19:41 client-5 kernel: [] do_select+0x3fe/0x462 May 17 21:19:41 client-5 kernel: [] __pollwait+0x0/0xe2 May 17 21:19:41 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:41 client-5 kernel: [] cfq_insert_request+0x3d/0x4fc May 17 21:19:41 client-5 kernel: [] elv_insert+0xfe/0x1c0 May 17 21:19:41 client-5 kernel: [] __next_cpu+0x19/0x28 May 17 21:19:41 client-5 kernel: [] find_busiest_group+0x20d/0x621 May 17 21:19:41 client-5 kernel: [] thread_return+0x62/0xfe May 17 21:19:41 client-5 kernel: [] find_get_pages_tag+0x34/0x89 May 17 21:19:41 client-5 kernel: [] core_sys_select+0x1bc/0x265 May 17 21:19:41 client-5 kernel: [] math_state_restore+0x23/0x4c May 17 21:19:41 client-5 kernel: [] error_exit+0x0/0x84 May 17 21:19:41 client-5 kernel: [] sys_rt_sigreturn+0x283/0x356 May 17 21:19:41 client-5 kernel: [] sys_rt_sigreturn+0x323/0x356 May 17 21:19:41 client-5 kernel: [] sys_select+0xc1/0x17c May 17 21:19:41 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:19:41 client-5 kernel: May 17 21:19:41 client-5 kernel: klogd R running task 0 3020 1 3083 3017 (NOTLB) May 17 21:19:41 client-5 kernel: ffff81033127be68 0000000000000086 ffff810300000000 ffff81033127bdd8 May 17 21:19:41 client-5 kernel: 0000000000000001 000000000000000a ffff81033aabe040 ffff81033c46a860 May 17 21:19:41 client-5 kernel: 0000413c552298e7 000000000000150a ffff81033aabe228 0000000300003000 May 17 21:19:41 client-5 kernel: Call Trace: May 17 21:19:41 client-5 kernel: [] do_sync_write+0xc7/0x104 May 17 21:19:41 client-5 kernel: [] do_syslog+0x11d/0x3aa May 17 21:19:41 client-5 kernel: [] autoremove_wake_function+0x0/0x2e May 17 21:19:41 client-5 kernel: [] kmsg_read+0x3a/0x44 May 17 21:19:41 client-5 kernel: [] vfs_read+0xcb/0x171 May 17 21:19:41 client-5 kernel: [] sys_read+0x45/0x6e May 17 21:19:41 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:19:41 client-5 kernel: May 17 21:19:41 client-5 kernel: kondemand/0 S ffff810001004420 0 3068 123 3070 2471 (L-TLB) May 17 21:19:41 client-5 kernel: ffff810329bb7e70 0000000000000046 ffff810001007ea0 0000000000000000 May 17 21:19:41 client-5 kernel: 0000000300000000 000000000000000a ffff81033add9040 ffffffff80308b60 May 17 21:19:41 client-5 kernel: 000041f235f0ac54 0000000000000b75 ffff81033add9228 000000008002e23f May 17 21:19:41 client-5 kernel: Call Trace: May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:19:41 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:19:41 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:19:41 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] kthread+0x0/0x132 May 17 21:19:41 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:19:41 client-5 kernel: May 17 21:19:41 client-5 kernel: kondemand/1 S ffff81000100caa0 0 3070 123 3071 3068 (L-TLB) May 17 21:19:41 client-5 kernel: ffff810329bbbe70 0000000000000046 ffff810001010520 0000000000000000 May 17 21:19:41 client-5 kernel: 0000000300000000 000000000000000a ffff81033b27c080 ffff81010b70e100 May 17 21:19:41 client-5 kernel: 000041f23ab39721 0000000000000700 ffff81033b27c268 000000018002e23f May 17 21:19:41 client-5 kernel: Call Trace: May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:19:41 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:19:41 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:19:41 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] kthread+0x0/0x132 May 17 21:19:41 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:19:41 client-5 kernel: May 17 21:19:41 client-5 kernel: kondemand/2 S ffff810001015120 0 3071 123 3072 3070 (L-TLB) May 17 21:19:41 client-5 kernel: ffff810329bbde70 0000000000000046 ffff810001018ba0 0000000000000000 May 17 21:19:41 client-5 kernel: 0000000300000000 000000000000000a ffff81033b0aa7e0 ffff81010b764080 May 17 21:19:41 client-5 kernel: 000041f23e457528 0000000000000d12 ffff810329bbde30 000000028002e23f May 17 21:19:41 client-5 kernel: Call Trace: May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:19:41 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:19:41 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:19:41 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] kthread+0x0/0x132 May 17 21:19:41 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:19:41 client-5 kernel: May 17 21:19:41 client-5 kernel: kondemand/3 S ffff81000101d7a0 0 3072 123 3128 3071 (L-TLB) May 17 21:19:41 client-5 kernel: ffff810329bbfe70 0000000000000046 ffff810001021220 0000000000000000 May 17 21:19:41 client-5 kernel: 0000000300000000 000000000000000a ffff81033f5c1100 ffff81010b770100 May 17 21:19:41 client-5 kernel: 000041f06cf11e4e 00000000000018eb ffff81033f5c12e8 000000038002e23f May 17 21:19:41 client-5 kernel: Call Trace: May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:19:41 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:19:41 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:19:41 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] kthread+0x0/0x132 May 17 21:19:41 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:19:41 client-5 kernel: May 17 21:19:41 client-5 kernel: irqbalance S ffff81000100caa0 0 3083 1 3101 3020 (NOTLB) May 17 21:19:41 client-5 kernel: ffff810329bb5eb8 0000000000000086 ffff810329bb5e58 ffff81000100eb80 May 17 21:19:41 client-5 kernel: 0000000253e7e528 0000000000000007 ffff81033a3bf860 ffff81010b70e100 May 17 21:19:41 client-5 kernel: 000041f1e7298dea 0000000000204643 ffff81033a3bfa48 0000000170800a57 May 17 21:19:41 client-5 kernel: Call Trace: May 17 21:19:41 client-5 kernel: [] enqueue_hrtimer+0x55/0x70 May 17 21:19:41 client-5 kernel: [] do_nanosleep+0x3f/0x70 May 17 21:19:41 client-5 kernel: [] hrtimer_nanosleep+0x58/0x118 May 17 21:19:41 client-5 kernel: [] hrtimer_wakeup+0x0/0x22 May 17 21:19:41 client-5 kernel: [] sys_nanosleep+0x4c/0x62 May 17 21:19:41 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:19:41 client-5 kernel: May 17 21:19:41 client-5 kernel: portmap S ffff810001004420 0 3101 1 3141 3083 (NOTLB) May 17 21:19:41 client-5 kernel: ffff810329bc7b38 0000000000000082 ffff810327d997e0 ffff810329fb60c0 May 17 21:19:41 client-5 kernel: ffff810327d997c0 000000000000000a ffff81033b49b0c0 ffffffff80308b60 May 17 21:19:41 client-5 kernel: 000041db426c1dda 00000000000032bc ffff81033b49b2a8 000000008024db4e May 17 21:19:41 client-5 kernel: Call Trace: May 17 21:19:41 client-5 kernel: [] schedule_timeout+0x1e/0xad May 17 21:19:41 client-5 kernel: [] add_wait_queue+0x24/0x34 May 17 21:19:41 client-5 kernel: [] tcp_poll+0x1c/0x132 May 17 21:19:41 client-5 kernel: [] do_sys_poll+0x277/0x360 May 17 21:19:41 client-5 kernel: [] __pollwait+0x0/0xe2 May 17 21:19:41 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:41 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:41 client-5 kernel: [] autoremove_wake_function+0x0/0x2e May 17 21:19:41 client-5 kernel: [] zone_statistics+0x3e/0x6d May 17 21:19:41 client-5 kernel: [] vsnprintf+0x5e7/0x62f May 17 21:19:41 client-5 kernel: [] sys_sendmsg+0x217/0x28a May 17 21:19:41 client-5 kernel: [] audit_syscall_entry+0x180/0x1b3 May 17 21:19:41 client-5 kernel: [] sys_poll+0x2d/0x34 May 17 21:19:41 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:19:41 client-5 kernel: May 17 21:19:41 client-5 kernel: rpciod/0 S 0000000000000282 0 3128 123 3129 3072 (L-TLB) May 17 21:19:41 client-5 kernel: ffff8103295e5e70 0000000000000046 0000001f8250ec9a 0000000000000003 May 17 21:19:41 client-5 kernel: ffff81033fc12040 0000000000000006 ffff81033fc12040 ffff81033fbbb040 May 17 21:19:41 client-5 kernel: 0000001f8251052f 00000000000004d1 ffff81033fc12228 0000000000000000 May 17 21:19:41 client-5 kernel: Call Trace: May 17 21:19:41 client-5 kernel: [] thread_return+0x62/0xfe May 17 21:19:41 client-5 kernel: [] recalc_sigpending_and_wake+0x9/0x1a May 17 21:19:41 client-5 kernel: [] do_sigaction+0x14d/0x199 May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:19:41 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:19:41 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:19:41 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] kthread+0x0/0x132 May 17 21:19:41 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:19:41 client-5 kernel: May 17 21:19:41 client-5 kernel: rpciod/1 S ffff810319a7b100 0 3129 123 3130 3128 (L-TLB) May 17 21:19:41 client-5 kernel: ffff8103295e7e70 0000000000000046 ffff810191bda530 ffff810191bda528 May 17 21:19:41 client-5 kernel: 0000000300000000 000000000000000a ffff81033fbc1860 ffff810319a7b100 May 17 21:19:41 client-5 kernel: 000041db426bd3b1 00000000000009b8 ffff81033fbc1a48 000000018002e23f May 17 21:19:41 client-5 kernel: Call Trace: May 17 21:19:41 client-5 kernel: [] :sunrpc:xs_udp_connect_worker+0x0/0x188 May 17 21:19:41 client-5 kernel: [] run_workqueue+0xba/0xe4 May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:19:41 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:19:41 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:19:41 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] kthread+0x0/0x132 May 17 21:19:41 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:19:41 client-5 kernel: May 17 21:19:41 client-5 kernel: rpciod/2 S ffff81031f9dc7e0 0 3130 123 3131 3129 (L-TLB) May 17 21:19:41 client-5 kernel: ffff8103295e9e70 0000000000000046 ffff810315d6e530 ffff810315d6e528 May 17 21:19:41 client-5 kernel: 0000000300000000 000000000000000a ffff81033c1a3040 ffff81031f9dc7e0 May 17 21:19:41 client-5 kernel: 00002b7d34bd07a9 0000000000001084 ffff81033c1a3228 000000028002e23f May 17 21:19:41 client-5 kernel: Call Trace: May 17 21:19:41 client-5 kernel: [] :sunrpc:xs_udp_connect_worker+0x0/0x188 May 17 21:19:41 client-5 kernel: [] run_workqueue+0xba/0xe4 May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:19:41 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:19:41 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:19:41 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] kthread+0x0/0x132 May 17 21:19:41 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:19:41 client-5 kernel: May 17 21:19:41 client-5 kernel: rpciod/3 S ffff81000101d7a0 0 3131 123 932 3130 (L-TLB) May 17 21:19:41 client-5 kernel: ffff8103295ebe70 0000000000000046 ffff8102f96f87d8 ffff8102f96f8580 May 17 21:19:41 client-5 kernel: 0000000300000000 000000000000000a ffff81033c54a7a0 ffff81010b770100 May 17 21:19:41 client-5 kernel: 00002b79f6371f67 0000000000006218 ffff81033c54a988 000000038002e23f May 17 21:19:41 client-5 kernel: Call Trace: May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:19:41 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:19:41 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:19:41 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:19:41 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:41 client-5 kernel: [] kthread+0x0/0x132 May 17 21:19:41 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:19:41 client-5 kernel: May 17 21:19:41 client-5 kernel: rpc.statd S ffff810001015120 0 3141 1 3163 3101 (NOTLB) May 17 21:19:41 client-5 kernel: ffff8103295eda18 0000000000000086 0000000000000001 ffffffff80150790 May 17 21:19:41 client-5 kernel: 00000000000280d2 0000000000000008 ffff81033ae7b080 ffff81010b764080 May 17 21:19:41 client-5 kernel: 0000001f864d63ed 0000000000002d35 ffff81033ae7b268 000000028008bae0 May 17 21:19:41 client-5 kernel: Call Trace: May 17 21:19:41 client-5 kernel: [] __next_cpu+0x19/0x28 May 17 21:19:41 client-5 kernel: [] schedule_timeout+0x1e/0xad May 17 21:19:41 client-5 kernel: [] add_wait_queue+0x24/0x34 May 17 21:19:41 client-5 kernel: [] tcp_poll+0x1c/0x132 May 17 21:19:41 client-5 kernel: [] do_select+0x3fe/0x462 May 17 21:19:41 client-5 kernel: [] __pollwait+0x0/0xe2 May 17 21:19:41 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:41 client-5 last message repeated 2 times May 17 21:19:41 client-5 kernel: [] autoremove_wake_function+0x0/0x2e May 17 21:19:42 client-5 kernel: [] skb_queue_tail+0x17/0x3e May 17 21:19:42 client-5 kernel: [] sock_def_readable+0x10/0x62 May 17 21:19:42 client-5 kernel: [] unix_dgram_sendmsg+0x44e/0x4e0 May 17 21:19:42 client-5 kernel: [] sock_sendmsg+0xf8/0x14a May 17 21:19:42 client-5 kernel: [] dput+0x2c/0x114 May 17 21:19:42 client-5 kernel: [] autoremove_wake_function+0x0/0x2e May 17 21:19:42 client-5 kernel: [] core_sys_select+0x1bc/0x265 May 17 21:19:42 client-5 kernel: [] _atomic_dec_and_lock+0x39/0x57 May 17 21:19:42 client-5 kernel: [] mntput_no_expire+0x19/0x89 May 17 21:19:42 client-5 kernel: [] sys_sendto+0x11c/0x14f May 17 21:19:42 client-5 kernel: [] sys_select+0xc1/0x17c May 17 21:19:42 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:19:42 client-5 kernel: May 17 21:19:42 client-5 kernel: rpc.idmapd S ffff810001004420 0 3163 1 3176 3141 (NOTLB) May 17 21:19:42 client-5 kernel: ffff81032946fe78 0000000000000082 ffff810329558380 ffff81032946fc38 May 17 21:19:42 client-5 kernel: ffff81032946fee8 0000000000000007 ffff81033b686040 ffffffff80308b60 May 17 21:19:42 client-5 kernel: 000041db426dac61 0000000000011024 ffff81033b686228 0000000000000246 May 17 21:19:42 client-5 kernel: Call Trace: May 17 21:19:42 client-5 kernel: [] schedule_timeout+0x1e/0xad May 17 21:19:42 client-5 kernel: [] sys_epoll_wait+0x184/0x3f9 May 17 21:19:42 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:42 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:19:42 client-5 kernel: May 17 21:19:42 client-5 kernel: dbus-daemon S ffff810001004420 0 3176 1 3184 3163 (NOTLB) May 17 21:19:42 client-5 kernel: ffff810329651b38 0000000000000082 0000000000000000 0000000000000000 May 17 21:19:42 client-5 kernel: 0000000000000008 000000000000000a ffff81033fbbb040 ffffffff80308b60 May 17 21:19:42 client-5 kernel: 00004181495d616e 0000000000000c8a ffff81033fbbb228 0000000000000001 May 17 21:19:42 client-5 kernel: Call Trace: May 17 21:19:42 client-5 kernel: [] schedule_timeout+0x1e/0xad May 17 21:19:42 client-5 kernel: [] add_wait_queue+0x24/0x34 May 17 21:19:42 client-5 kernel: [] do_sys_poll+0x277/0x360 May 17 21:19:42 client-5 kernel: [] __pollwait+0x0/0xe2 May 17 21:19:42 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:42 client-5 last message repeated 4 times May 17 21:19:42 client-5 kernel: [] autoremove_wake_function+0x0/0x2e May 17 21:19:42 client-5 kernel: [] sock_aio_read+0x4f/0x5e May 17 21:19:42 client-5 kernel: [] mntput_no_expire+0x19/0x89 May 17 21:19:42 client-5 kernel: [] do_sync_read+0xc7/0x104 May 17 21:19:42 client-5 kernel: [] do_readv_writev+0x26e/0x291 May 17 21:19:42 client-5 kernel: [] audit_syscall_entry+0x180/0x1b3 May 17 21:19:42 client-5 kernel: [] sys_poll+0x2d/0x34 May 17 21:19:42 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:19:42 client-5 kernel: May 17 21:19:42 client-5 kernel: hcid S ffff81033fbbb040 0 3184 1 3190 3176 (NOTLB) May 17 21:19:42 client-5 kernel: ffff8103296b1b38 0000000000000086 0000000000000000 0000000000000000 May 17 21:19:42 client-5 kernel: ffff81033c46a860 0000000000000001 ffff81033b304100 ffff81033fbbb040 May 17 21:19:42 client-5 kernel: 0000001f966db063 0000000000024fd5 ffff81033b3042e8 0000000300000000 May 17 21:19:42 client-5 kernel: Call Trace: May 17 21:19:42 client-5 kernel: [] try_to_wake_up+0x472/0x484 May 17 21:19:42 client-5 kernel: [] schedule_timeout+0x1e/0xad May 17 21:19:42 client-5 kernel: [] add_wait_queue+0x24/0x34 May 17 21:19:42 client-5 kernel: [] do_sys_poll+0x277/0x360 May 17 21:19:42 client-5 kernel: [] __pollwait+0x0/0xe2 May 17 21:19:42 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:42 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:42 client-5 kernel: [] sock_sendmsg+0xf8/0x14a May 17 21:19:42 client-5 kernel: [] dput+0x2c/0x114 May 17 21:19:42 client-5 kernel: [] find_get_page+0x21/0x51 May 17 21:19:42 client-5 kernel: [] filemap_nopage+0x193/0x360 May 17 21:19:42 client-5 kernel: [] __handle_mm_fault+0x5f3/0x1039 May 17 21:19:42 client-5 kernel: [] _atomic_dec_and_lock+0x39/0x57 May 17 21:19:42 client-5 kernel: [] do_page_fault+0x4fe/0x874 May 17 21:19:42 client-5 kernel: [] sock_ioctl+0x1c1/0x1e5 May 17 21:19:42 client-5 kernel: [] do_ioctl+0x21/0x6b May 17 21:19:42 client-5 kernel: [] vfs_ioctl+0x457/0x4b9 May 17 21:19:42 client-5 kernel: [] audit_syscall_entry+0x180/0x1b3 May 17 21:19:42 client-5 kernel: [] sys_poll+0x2d/0x34 May 17 21:19:42 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:19:42 client-5 kernel: May 17 21:19:42 client-5 kernel: sdpd S ffff810001015120 0 3190 1 3221 3184 (NOTLB) May 17 21:19:42 client-5 kernel: ffff8103296db9c8 0000000000000082 ffffffff800cc235 ffff81010b70b0c0 May 17 21:19:42 client-5 kernel: 0000000000000286 0000000000000001 ffff81033b0aa080 ffff81010b764080 May 17 21:19:42 client-5 kernel: 0000001f96f336d4 0000000000003a31 ffff81033b0aa268 0000000200000001 May 17 21:19:42 client-5 kernel: Call Trace: May 17 21:19:42 client-5 kernel: [] zone_statistics+0x3e/0x6d May 17 21:19:42 client-5 kernel: [] schedule_timeout+0x8a/0xad May 17 21:19:42 client-5 kernel: [] process_timeout+0x0/0x5 May 17 21:19:42 client-5 kernel: [] do_select+0x3fe/0x462 May 17 21:19:42 client-5 kernel: [] __pollwait+0x0/0xe2 May 17 21:19:42 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:42 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:42 client-5 kernel: [] __wake_up_common+0x3e/0x68 May 17 21:19:42 client-5 kernel: [] __wake_up+0x38/0x4f May 17 21:19:42 client-5 kernel: [] sock_def_readable+0x37/0x62 May 17 21:19:42 client-5 kernel: [] unix_dgram_sendmsg+0x44e/0x4e0 May 17 21:19:42 client-5 kernel: [] touch_atime+0x67/0xaa May 17 21:19:42 client-5 kernel: [] sock_sendmsg+0xf8/0x14a May 17 21:19:42 client-5 kernel: [] core_sys_select+0x1bc/0x265 May 17 21:19:42 client-5 kernel: [] __handle_mm_fault+0x5f3/0x1039 May 17 21:19:42 client-5 kernel: [] _atomic_dec_and_lock+0x39/0x57 May 17 21:19:42 client-5 kernel: [] __up_read+0x19/0x7f May 17 21:19:42 client-5 kernel: [] do_page_fault+0x4fe/0x874 May 17 21:19:42 client-5 kernel: [] sys_pselect7+0x114/0x210 May 17 21:19:42 client-5 kernel: [] audit_syscall_entry+0x180/0x1b3 May 17 21:19:42 client-5 kernel: [] sys_pselect6+0x54/0x61 May 17 21:19:42 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:19:42 client-5 kernel: May 17 21:19:42 client-5 kernel: krfcommd S ffff810001004420 0 3221 1 3256 3190 (L-TLB) May 17 21:19:42 client-5 kernel: ffff8103297e7e40 0000000000000046 ffff810329cf6340 0000000000000000 May 17 21:19:42 client-5 kernel: 00000000ffffffea 0000000000000006 ffff81033b657100 ffffffff80308b60 May 17 21:19:42 client-5 kernel: 0000001f975fed7a 0000000000000f4d ffff81033b6572e8 00000000800335e5 May 17 21:19:42 client-5 kernel: Call Trace: May 17 21:19:42 client-5 kernel: [] :rfcomm:rfcomm_run+0x1a0/0x1207 May 17 21:19:42 client-5 kernel: [] audit_syscall_exit+0x336/0x362 May 17 21:19:42 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:19:42 client-5 kernel: [] :rfcomm:rfcomm_run+0x0/0x1207 May 17 21:19:42 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:19:42 client-5 kernel: May 17 21:19:42 client-5 kernel: pcscd S 0000000000000020 0 3256 1 3265 3221 (NOTLB) May 17 21:19:42 client-5 kernel: ffff810329bd5a18 0000000000000082 ffff81033e52e288 ffff81032a9f9d20 May 17 21:19:42 client-5 kernel: ffff81033e548000 0000000000000001 ffff81033c5167a0 ffff81033aca8820 May 17 21:19:42 client-5 kernel: 0000001fa70bab0b 0000000000057933 ffff81033c516988 000000003e52e288 May 17 21:19:42 client-5 kernel: Call Trace: May 17 21:19:42 client-5 kernel: [] elv_insert+0xfe/0x1c0 May 17 21:19:42 client-5 kernel: [] __make_request+0x3c5/0x414 May 17 21:19:42 client-5 kernel: [] schedule_timeout+0x1e/0xad May 17 21:19:42 client-5 kernel: [] add_wait_queue+0x24/0x34 May 17 21:19:42 client-5 kernel: [] do_select+0x3fe/0x462 May 17 21:19:42 client-5 kernel: [] __pollwait+0x0/0xe2 May 17 21:19:42 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:42 client-5 kernel: [] :jbd:journal_dirty_metadata+0x1b5/0x1e3 May 17 21:19:42 client-5 kernel: [] __do_page_cache_readahead+0x82/0x179 May 17 21:19:42 client-5 kernel: [] current_fs_time+0x3b/0x40 May 17 21:19:42 client-5 kernel: [] blockable_page_cache_readahead+0x53/0xb2 May 17 21:19:42 client-5 kernel: [] touch_atime+0x67/0xaa May 17 21:19:42 client-5 kernel: [] zone_statistics+0x3e/0x6d May 17 21:19:42 client-5 kernel: [] get_page_from_freelist+0x378/0x43a May 17 21:19:42 client-5 kernel: [] zone_statistics+0x3e/0x6d May 17 21:19:42 client-5 kernel: [] get_page_from_freelist+0x378/0x43a May 17 21:19:42 client-5 kernel: [] __rmqueue+0x91/0xc6 May 17 21:19:42 client-5 kernel: [] __bitmap_weight+0x65/0x76 May 17 21:19:42 client-5 kernel: [] core_sys_select+0x1bc/0x265 May 17 21:19:42 client-5 kernel: [] attach_pid+0x8c/0xb9 May 17 21:19:42 client-5 kernel: [] enqueue_task+0x41/0x56 May 17 21:19:42 client-5 kernel: [] __activate_task+0x56/0x6d May 17 21:19:42 client-5 kernel: [] wake_up_new_task+0x287/0x296 May 17 21:19:42 client-5 kernel: [] sys_select+0xc1/0x17c May 17 21:19:42 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:19:42 client-5 kernel: May 17 21:19:42 client-5 kernel: pcscd S ffff810001004420 0 3276 1 3273 3265 (NOTLB) May 17 21:19:42 client-5 kernel: ffff810329241eb8 0000000000000082 ffff810329241e58 ffff810001006500 May 17 21:19:42 client-5 kernel: 000000003b9aca00 0000000000000007 ffff81033aca8820 ffffffff80308b60 May 17 21:19:42 client-5 kernel: 000041f27f4493ba 0000000000048d5a ffff81033aca8a08 00000000f08e1f97 May 17 21:19:42 client-5 kernel: Call Trace: May 17 21:19:42 client-5 kernel: [] enqueue_hrtimer+0x55/0x70 May 17 21:19:42 client-5 kernel: [] do_nanosleep+0x3f/0x70 May 17 21:19:42 client-5 kernel: [] hrtimer_nanosleep+0x58/0x118 May 17 21:19:42 client-5 kernel: [] hrtimer_wakeup+0x0/0x22 May 17 21:19:42 client-5 kernel: [] sys_nanosleep+0x4c/0x62 May 17 21:19:42 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:19:42 client-5 kernel: May 17 21:19:42 client-5 kernel: acpid S 0000000000000000 0 3265 1 3276 3256 (NOTLB) May 17 21:19:42 client-5 kernel: ffff81032940db38 0000000000000086 ffff81033eb1c800 ffffffff8804d916 May 17 21:19:42 client-5 kernel: ffff81032940dad8 0000000000000002 ffff81033add97a0 ffff81033a91a040 May 17 21:19:42 client-5 kernel: 0000001fb16e00c1 000000000000f49a ffff81033add9988 000000030b7752d8 May 17 21:19:42 client-5 kernel: Call Trace: May 17 21:19:42 client-5 kernel: [] :ext3:ext3_mark_iloc_dirty+0x2f8/0x360 May 17 21:19:42 client-5 kernel: [] schedule_timeout+0x1e/0xad May 17 21:19:42 client-5 kernel: [] add_wait_queue+0x24/0x34 May 17 21:19:42 client-5 kernel: [] do_sys_poll+0x277/0x360 May 17 21:19:42 client-5 kernel: [] __pollwait+0x0/0xe2 May 17 21:19:42 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:42 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:42 client-5 kernel: [] __generic_file_aio_write_nolock+0x369/0x3b6 May 17 21:19:42 client-5 kernel: [] generic_file_aio_write+0x70/0xc1 May 17 21:19:42 client-5 kernel: [] :ext3:ext3_file_write+0x16/0x91 May 17 21:19:42 client-5 kernel: [] do_sync_write+0xc7/0x104 May 17 21:19:42 client-5 kernel: [] sock_getsockopt+0x32b/0x34d May 17 21:19:42 client-5 kernel: [] autoremove_wake_function+0x0/0x2e May 17 21:19:42 client-5 kernel: [] sys_accept+0x1a4/0x1ea May 17 21:19:42 client-5 kernel: [] audit_syscall_entry+0x180/0x1b3 May 17 21:19:42 client-5 kernel: [] sys_poll+0x2d/0x34 May 17 21:19:42 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:19:42 client-5 kernel: May 17 21:19:42 client-5 kernel: hald S ffff81000101d7a0 0 3273 1 3274 3315 3276 (NOTLB) May 17 21:19:42 client-5 kernel: ffff8103290b7b38 0000000000000082 0000000000000001 0000000000000000 May 17 21:19:42 client-5 kernel: 0000000000000000 000000000000000a ffff81033c1a37a0 ffff81010b770100 May 17 21:19:42 client-5 kernel: 000041ef587dd139 000000000002b685 ffff81033c1a3988 0000000300000002 May 17 21:19:42 client-5 kernel: Call Trace: May 17 21:19:42 client-5 kernel: [] get_pid_task+0x5/0xf May 17 21:19:42 client-5 kernel: [] schedule_timeout+0x8a/0xad May 17 21:19:42 client-5 kernel: [] process_timeout+0x0/0x5 May 17 21:19:42 client-5 kernel: [] do_sys_poll+0x277/0x360 May 17 21:19:42 client-5 kernel: [] __pollwait+0x0/0xe2 May 17 21:19:42 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:42 client-5 last message repeated 8 times May 17 21:19:42 client-5 kernel: [] free_pages_and_swap_cache+0x67/0x7e May 17 21:19:42 client-5 kernel: [] audit_syscall_entry+0x180/0x1b3 May 17 21:19:42 client-5 kernel: [] sys_poll+0x2d/0x34 May 17 21:19:42 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:19:42 client-5 kernel: May 17 21:19:42 client-5 kernel: hald-runner S ffff810001015120 0 3274 3273 3282 (NOTLB) May 17 21:19:42 client-5 kernel: ffff8103290f7b38 0000000000000082 ffff8103290f7aa8 ffffffff8008c9a2 May 17 21:19:42 client-5 kernel: 0000001fd62a30aa 0000000000000007 ffff81033acc1080 ffff81010b764080 May 17 21:19:42 client-5 kernel: 0000001fd63ab01c 00000000000060c9 ffff81033acc1268 0000000200000001 May 17 21:19:42 client-5 kernel: Call Trace: May 17 21:19:42 client-5 kernel: [] __activate_task+0x56/0x6d May 17 21:19:42 client-5 kernel: [] schedule_timeout+0x1e/0xad May 17 21:19:42 client-5 kernel: [] add_wait_queue+0x24/0x34 May 17 21:19:42 client-5 kernel: [] do_sys_poll+0x277/0x360 May 17 21:19:42 client-5 kernel: [] __pollwait+0x0/0xe2 May 17 21:19:42 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:42 client-5 kernel: [] get_page_from_freelist+0x378/0x43a May 17 21:19:42 client-5 kernel: [] do_sock_write+0xc6/0x102 May 17 21:19:42 client-5 kernel: [] sock_writev+0xb7/0xd1 May 17 21:19:42 client-5 kernel: [] flush_tlb_page+0xac/0xda May 17 21:19:42 client-5 kernel: [] do_wp_page+0x3fd/0x902 May 17 21:19:42 client-5 kernel: [] __handle_mm_fault+0xf6b/0x1039 May 17 21:19:42 client-5 kernel: [] math_state_restore+0x23/0x4c May 17 21:19:42 client-5 kernel: [] error_exit+0x0/0x84 May 17 21:19:42 client-5 kernel: [] __up_read+0x19/0x7f May 17 21:19:42 client-5 kernel: [] do_page_fault+0x4fe/0x874 May 17 21:19:42 client-5 kernel: [] thread_return+0x7a/0xfe May 17 21:19:42 client-5 kernel: [] release_task+0x3a5/0x3cb May 17 21:19:42 client-5 kernel: [] audit_syscall_entry+0x180/0x1b3 May 17 21:19:42 client-5 kernel: [] sys_poll+0x2d/0x34 May 17 21:19:42 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:19:42 client-5 kernel: May 17 21:19:42 client-5 kernel: hald-addon-ac S ffff81000101d7a0 0 3282 3274 3289 (NOTLB) May 17 21:19:42 client-5 kernel: ffff810328d95bd8 0000000000000086 ffff810328d95b48 0000000000000092 May 17 21:19:42 client-5 kernel: 00000000fffc9956 0000000000000001 ffff81033a91a040 ffff81010b770100 May 17 21:19:42 client-5 kernel: 0000001fb16e2d01 0000000000002c40 ffff81033a91a228 00000003802ac375 May 17 21:19:42 client-5 kernel: Call Trace: May 17 21:19:42 client-5 kernel: [] schedule_timeout+0x1e/0xad May 17 21:19:42 client-5 kernel: [] prepare_to_wait+0x34/0x61 May 17 21:19:42 client-5 kernel: [] unix_stream_recvmsg+0x245/0x56f May 17 21:19:42 client-5 kernel: [] autoremove_wake_function+0x0/0x2e May 17 21:19:42 client-5 kernel: [] dput+0x2c/0x114 May 17 21:19:42 client-5 kernel: [] enqueue_task+0x41/0x56 May 17 21:19:42 client-5 kernel: [] do_sock_read+0xcf/0x110 May 17 21:19:42 client-5 kernel: [] sock_aio_read+0x4f/0x5e May 17 21:19:42 client-5 kernel: [] do_sync_read+0xc7/0x104 May 17 21:19:42 client-5 kernel: [] autoremove_wake_function+0x0/0x2e May 17 21:19:42 client-5 kernel: [] do_mmap_pgoff+0x615/0x780 May 17 21:19:42 client-5 kernel: [] vfs_read+0xde/0x171 May 17 21:19:42 client-5 kernel: [] sys_read+0x45/0x6e May 17 21:19:42 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:19:42 client-5 kernel: May 17 21:19:42 client-5 kernel: hald-addon-ke S ffff81000100caa0 0 3289 3274 3282 (NOTLB) May 17 21:19:42 client-5 kernel: ffff810328dd7ea8 0000000000000086 0000000000000000 00002ab9a9b24000 May 17 21:19:42 client-5 kernel: 00002ab9a9925000 0000000000000001 ffff81033b724100 ffff81010b70e100 May 17 21:19:42 client-5 kernel: 0000001fc3753e10 000000000004120e ffff81033b7242e8 00000001a9b25000 May 17 21:19:42 client-5 kernel: Call Trace: May 17 21:19:42 client-5 kernel: [] vma_merge+0x141/0x1ee May 17 21:19:42 client-5 kernel: [] evdev_read+0x112/0x247 May 17 21:19:42 client-5 kernel: [] autoremove_wake_function+0x0/0x2e May 17 21:19:42 client-5 kernel: [] vfs_read+0xcb/0x171 May 17 21:19:42 client-5 kernel: [] sys_read+0x45/0x6e May 17 21:19:42 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:19:42 client-5 kernel: May 17 21:19:42 client-5 kernel: hidd S 0000000000000000 0 3315 1 3344 3273 (NOTLB) May 17 21:19:42 client-5 kernel: ffff8103298d1ad8 0000000000000082 00002b5c71e5e000 ffff81033a993b80 May 17 21:19:42 client-5 kernel: ffff810329740c78 0000000000000001 ffff81033c516040 ffff81033ad4f860 May 17 21:19:42 client-5 kernel: 0000001fdc936b05 00000000000159e1 ffff81033c516228 000000000ca5eaa0 May 17 21:19:42 client-5 kernel: Call Trace: May 17 21:19:42 client-5 kernel: [] find_get_page+0x21/0x51 May 17 21:19:42 client-5 kernel: [] schedule_timeout+0x1e/0xad May 17 21:19:42 client-5 kernel: [] add_wait_queue+0x24/0x34 May 17 21:19:42 client-5 kernel: [] do_sys_poll+0x277/0x360 May 17 21:19:42 client-5 kernel: [] __pollwait+0x0/0xe2 May 17 21:19:42 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:42 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:42 client-5 kernel: [] skb_queue_tail+0x17/0x3e May 17 21:19:42 client-5 kernel: [] sock_def_readable+0x10/0x62 May 17 21:19:42 client-5 kernel: [] unix_dgram_sendmsg+0x44e/0x4e0 May 17 21:19:42 client-5 kernel: [] touch_atime+0x67/0xaa May 17 21:19:42 client-5 kernel: [] sock_sendmsg+0xf8/0x14a May 17 21:19:42 client-5 kernel: [] dput+0x2c/0x114 May 17 21:19:42 client-5 kernel: [] find_get_page+0x21/0x51 May 17 21:19:42 client-5 kernel: [] filemap_nopage+0x193/0x360 May 17 21:19:42 client-5 kernel: [] __handle_mm_fault+0x5f3/0x1039 May 17 21:19:42 client-5 kernel: [] _atomic_dec_and_lock+0x39/0x57 May 17 21:19:42 client-5 kernel: [] __up_read+0x19/0x7f May 17 21:19:42 client-5 kernel: [] do_page_fault+0x4fe/0x874 May 17 21:19:42 client-5 kernel: [] sys_ppoll+0xf1/0x1f2 May 17 21:19:42 client-5 kernel: [] tracesys+0x71/0xe0 May 17 21:19:42 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:19:42 client-5 kernel: May 17 21:19:42 client-5 kernel: automount S ffffffff8006271a 0 3344 1 3345 3315 (NOTLB) May 17 21:19:42 client-5 kernel: ffff810328ebfe58 0000000000000082 0000000000000286 0000000000000286 May 17 21:19:42 client-5 kernel: 0000000000010800 0000000000000005 ffff81033a97d100 ffff81033fc57860 May 17 21:19:42 client-5 kernel: 000000202b7ff6e0 0000000000003e3b ffff81033a97d2e8 0000000300000000 May 17 21:19:42 client-5 kernel: Call Trace: May 17 21:19:42 client-5 kernel: [] schedule_timeout+0x1e/0xad May 17 21:19:42 client-5 kernel: [] recalc_sigpending+0xe/0x25 May 17 21:19:42 client-5 kernel: [] dequeue_signal+0x47/0xcd May 17 21:19:42 client-5 kernel: [] sys_rt_sigtimedwait+0x1ce/0x2b4 May 17 21:19:42 client-5 kernel: [] audit_syscall_entry+0x180/0x1b3 May 17 21:19:42 client-5 kernel: [] tracesys+0x71/0xe0 May 17 21:19:42 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:19:42 client-5 kernel: May 17 21:19:42 client-5 kernel: automount S 0000000000000000 0 3345 1 3346 3344 (NOTLB) May 17 21:19:42 client-5 kernel: ffff810328edfd28 0000000000000082 000000013bb6e088 0000000000000001 May 17 21:19:42 client-5 kernel: 000000013e91c040 000000000000000a ffff81033aace040 ffff810327a210c0 May 17 21:19:42 client-5 kernel: 000041e5af8c40e0 0000000000000be8 ffff81033aace228 000000018002b398 May 17 21:19:42 client-5 kernel: Call Trace: May 17 21:19:42 client-5 kernel: [] lock_timer_base+0x1b/0x3c May 17 21:19:42 client-5 kernel: [] __mod_timer+0x100/0x10f May 17 21:19:42 client-5 kernel: [] schedule_timeout+0x8a/0xad May 17 21:19:42 client-5 kernel: [] process_timeout+0x0/0x5 May 17 21:19:42 client-5 kernel: [] do_futex+0x25e/0xcf3 May 17 21:19:42 client-5 kernel: [] __handle_mm_fault+0xf6b/0x1039 May 17 21:19:42 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:42 client-5 kernel: [] sys_futex+0x10a/0x12b May 17 21:19:42 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:19:42 client-5 kernel: May 17 21:19:42 client-5 kernel: automount S ffff81000100caa0 0 3346 1 3349 3345 (NOTLB) May 17 21:19:42 client-5 kernel: ffff810328ee1d28 0000000000000082 0000000000000000 0000000000000000 May 17 21:19:42 client-5 kernel: ffff8102f1a67df0 000000000000000a ffff81033bb430c0 ffff81010b70e100 May 17 21:19:42 client-5 kernel: 000041f2ba755c18 0000000000001d58 ffff81033bb432a8 0000000100000000 May 17 21:19:42 client-5 kernel: Call Trace: May 17 21:19:42 client-5 kernel: [] schedule_timeout+0x8a/0xad May 17 21:19:42 client-5 kernel: [] process_timeout+0x0/0x5 May 17 21:19:42 client-5 kernel: [] do_futex+0x25e/0xcf3 May 17 21:19:42 client-5 kernel: [] __alloc_pages+0x78/0x308 May 17 21:19:42 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:42 client-5 kernel: [] wake_up_new_task+0x287/0x296 May 17 21:19:42 client-5 kernel: [] do_fork+0x148/0x1c1 May 17 21:19:42 client-5 kernel: [] sys_futex+0x10a/0x12b May 17 21:19:42 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:19:42 client-5 kernel: May 17 21:19:42 client-5 kernel: automount S ffff810001004420 0 3349 1 3352 3346 (NOTLB) May 17 21:19:42 client-5 kernel: ffff810328835b38 0000000000000082 0000000000000000 0000000000000000 May 17 21:19:42 client-5 kernel: 0000000000000008 0000000000000002 ffff81033acc17e0 ffffffff80308b60 May 17 21:19:42 client-5 kernel: 0000002028e6e22b 00000000000009bd ffff81033acc19c8 0000000000000001 May 17 21:19:42 client-5 kernel: Call Trace: May 17 21:19:42 client-5 kernel: [] schedule_timeout+0x1e/0xad May 17 21:19:42 client-5 kernel: [] add_wait_queue+0x24/0x34 May 17 21:19:42 client-5 kernel: [] pipe_poll+0x2d/0x90 May 17 21:19:42 client-5 kernel: [] do_sys_poll+0x277/0x360 May 17 21:19:42 client-5 kernel: [] __pollwait+0x0/0xe2 May 17 21:19:42 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:42 client-5 last message repeated 2 times May 17 21:19:42 client-5 kernel: [] __wake_up_common+0x3e/0x68 May 17 21:19:42 client-5 kernel: [] __wake_up+0x38/0x4f May 17 21:19:42 client-5 kernel: [] wake_futex+0x39/0x43 May 17 21:19:42 client-5 kernel: [] futex_wake+0x95/0xd4 May 17 21:19:42 client-5 kernel: [] do_futex+0x2c2/0xcf3 May 17 21:19:42 client-5 kernel: [] __handle_mm_fault+0xf6b/0x1039 May 17 21:19:42 client-5 kernel: [] do_page_fault+0x4fe/0x874 May 17 21:19:42 client-5 kernel: [] thread_return+0x62/0xfe May 17 21:19:42 client-5 kernel: [] do_ioctl+0x21/0x6b May 17 21:19:42 client-5 kernel: [] audit_syscall_entry+0x180/0x1b3 May 17 21:19:42 client-5 kernel: [] sys_poll+0x2d/0x34 May 17 21:19:42 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:19:42 client-5 kernel: May 17 21:19:42 client-5 kernel: automount S 0000000000000000 0 3352 1 3353 3349 (NOTLB) May 17 21:19:42 client-5 kernel: ffff810328837b38 0000000000000082 0000000000000000 0000000000000000 May 17 21:19:42 client-5 kernel: 0000000000000008 0000000000000004 ffff81033c3277e0 ffff81033aa46820 May 17 21:19:42 client-5 kernel: 0000002028fd07f6 00000000000008d0 ffff81033c3279c8 0000000000000001 May 17 21:19:42 client-5 kernel: Call Trace: May 17 21:19:42 client-5 kernel: [] match_number+0x81/0x8f May 17 21:19:42 client-5 kernel: [] schedule_timeout+0x1e/0xad May 17 21:19:42 client-5 kernel: [] add_wait_queue+0x24/0x34 May 17 21:19:42 client-5 kernel: [] pipe_poll+0x2d/0x90 May 17 21:19:42 client-5 kernel: [] do_sys_poll+0x277/0x360 May 17 21:19:42 client-5 kernel: [] __pollwait+0x0/0xe2 May 17 21:19:42 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:42 client-5 last message repeated 2 times May 17 21:19:42 client-5 kernel: [] __wake_up_common+0x3e/0x68 May 17 21:19:42 client-5 kernel: [] __wake_up+0x38/0x4f May 17 21:19:42 client-5 kernel: [] wake_futex+0x39/0x43 May 17 21:19:42 client-5 kernel: [] futex_wake+0x95/0xd4 May 17 21:19:42 client-5 kernel: [] do_futex+0x2c2/0xcf3 May 17 21:19:42 client-5 kernel: [] :autofs4:autofs_dev_ioctl_openmount+0xae/0x12d May 17 21:19:42 client-5 kernel: [] :autofs4:autofs_dev_ioctl_timeout+0x0/0x2c May 17 21:19:42 client-5 kernel: [] :autofs4:autofs_dev_ioctl+0x2dc/0x33f May 17 21:19:42 client-5 kernel: [] thread_return+0x62/0xfe May 17 21:19:42 client-5 kernel: [] do_ioctl+0x21/0x6b May 17 21:19:42 client-5 kernel: [] audit_syscall_entry+0x180/0x1b3 May 17 21:19:42 client-5 kernel: [] sys_poll+0x2d/0x34 May 17 21:19:42 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:19:42 client-5 kernel: May 17 21:19:42 client-5 kernel: automount S 0000000000000000 0 3353 1 3354 3352 (NOTLB) May 17 21:19:42 client-5 kernel: ffff810328857b38 0000000000000082 0000000000000000 0000000000000000 May 17 21:19:42 client-5 kernel: ffff810328857ae8 0000000000000007 ffff81033aa46820 ffff81033b724860 May 17 21:19:42 client-5 kernel: 000041db3c87fd95 000000000000150b ffff81033aa46a08 000000003cae6780 May 17 21:19:42 client-5 kernel: Call Trace: May 17 21:19:42 client-5 kernel: [] error_exit+0x0/0x84 May 17 21:19:42 client-5 kernel: [] flush_tlb_page+0xac/0xda May 17 21:19:42 client-5 kernel: [] schedule_timeout+0x1e/0xad May 17 21:19:42 client-5 kernel: [] add_wait_queue+0x24/0x34 May 17 21:19:42 client-5 kernel: [] pipe_poll+0x2d/0x90 May 17 21:19:42 client-5 kernel: [] do_sys_poll+0x277/0x360 May 17 21:19:42 client-5 kernel: [] __pollwait+0x0/0xe2 May 17 21:19:42 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:42 client-5 last message repeated 2 times May 17 21:19:42 client-5 kernel: [] lock_timer_base+0x1b/0x3c May 17 21:19:42 client-5 kernel: [] try_to_del_timer_sync+0x7f/0x88 May 17 21:19:42 client-5 kernel: [] futex_wake+0x50/0xd4 May 17 21:19:42 client-5 kernel: [] do_futex+0x2c2/0xcf3 May 17 21:19:42 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:42 client-5 kernel: [] wake_up_new_task+0x287/0x296 May 17 21:19:42 client-5 kernel: [] do_fork+0x148/0x1c1 May 17 21:19:42 client-5 kernel: [] audit_syscall_entry+0x180/0x1b3 May 17 21:19:42 client-5 kernel: [] sys_poll+0x2d/0x34 May 17 21:19:42 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:19:42 client-5 kernel: May 17 21:19:42 client-5 kernel: automount S ffff81000101d7a0 0 3354 1 3365 3353 (NOTLB) May 17 21:19:42 client-5 kernel: ffff810328897b38 0000000000000082 0000000000000000 0000000000000000 May 17 21:19:42 client-5 kernel: 0000000000000008 0000000000000004 ffff81033fc57860 ffff81010b770100 May 17 21:19:42 client-5 kernel: 000000202b801433 0000000000001d53 ffff81033fc57a48 0000000300000001 May 17 21:19:42 client-5 kernel: Call Trace: May 17 21:19:42 client-5 kernel: [] schedule_timeout+0x1e/0xad May 17 21:19:42 client-5 kernel: [] add_wait_queue+0x24/0x34 May 17 21:19:42 client-5 kernel: [] pipe_poll+0x2d/0x90 May 17 21:19:42 client-5 kernel: [] do_sys_poll+0x277/0x360 May 17 21:19:42 client-5 kernel: [] __pollwait+0x0/0xe2 May 17 21:19:42 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:42 client-5 last message repeated 2 times May 17 21:19:42 client-5 kernel: [] __wake_up_common+0x3e/0x68 May 17 21:19:42 client-5 kernel: [] __wake_up+0x38/0x4f May 17 21:19:42 client-5 kernel: [] wake_futex+0x39/0x43 May 17 21:19:42 client-5 kernel: [] futex_wake+0x95/0xd4 May 17 21:19:42 client-5 kernel: [] do_futex+0x2c2/0xcf3 May 17 21:19:42 client-5 kernel: [] :autofs4:autofs_dev_ioctl_openmount+0xae/0x12d May 17 21:19:42 client-5 kernel: [] :autofs4:autofs_dev_ioctl_timeout+0x0/0x2c May 17 21:19:42 client-5 kernel: [] :autofs4:autofs_dev_ioctl+0x2dc/0x33f May 17 21:19:42 client-5 kernel: [] thread_return+0x62/0xfe May 17 21:19:42 client-5 kernel: [] do_ioctl+0x21/0x6b May 17 21:19:42 client-5 kernel: [] audit_syscall_entry+0x180/0x1b3 May 17 21:19:42 client-5 kernel: [] sys_poll+0x2d/0x34 May 17 21:19:42 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:19:42 client-5 kernel: May 17 21:19:42 client-5 kernel: sssd S ffff810001004420 0 3365 1 3373 3379 3354 (NOTLB) May 17 21:19:42 client-5 kernel: ffff810328ac9e78 0000000000000086 0000000000000001 0000000000000000 May 17 21:19:42 client-5 kernel: 0000000000000000 000000000000000a ffff81033ad4f860 ffffffff80308b60 May 17 21:19:42 client-5 kernel: 000041f2d50ea500 00000000000029e8 ffff81033ad4fa48 000000008000ceb5 May 17 21:19:42 client-5 kernel: Call Trace: May 17 21:19:42 client-5 kernel: [] schedule_timeout+0x8a/0xad May 17 21:19:42 client-5 kernel: [] process_timeout+0x0/0x5 May 17 21:19:42 client-5 kernel: [] sys_epoll_wait+0x184/0x3f9 May 17 21:19:42 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:42 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:19:42 client-5 kernel: May 17 21:19:42 client-5 kernel: sssd_be S ffff810001015120 0 3373 3365 3376 (NOTLB) May 17 21:19:42 client-5 kernel: ffff810328ba1e78 0000000000000082 ffff810328ba1e18 ffff81033f757ac0 May 17 21:19:42 client-5 kernel: 000000000000001c 000000000000000a ffff81033a91a7a0 ffff81010b764080 May 17 21:19:42 client-5 kernel: 000041f1ab8d8b93 0000000000000edc ffff81033a91a988 00000002800182c3 May 17 21:19:42 client-5 kernel: Call Trace: May 17 21:19:42 client-5 kernel: [] kmem_cache_alloc+0x6e/0x76 May 17 21:19:42 client-5 kernel: [] schedule_timeout+0x8a/0xad May 17 21:19:42 client-5 kernel: [] process_timeout+0x0/0x5 May 17 21:19:42 client-5 kernel: [] sys_epoll_wait+0x184/0x3f9 May 17 21:19:42 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:42 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:19:42 client-5 kernel: May 17 21:19:42 client-5 kernel: sssd_nss S ffff81033fcc0100 0 3376 3365 3377 3373 (NOTLB) May 17 21:19:42 client-5 kernel: ffff81032851fe78 0000000000000082 ffff81032851fe18 0000000000000000 May 17 21:19:42 client-5 kernel: 0000000000000000 000000000000000a ffff81033a97d860 ffff81033fcc0100 May 17 21:19:42 client-5 kernel: 000041f1ab8d8115 0000000000000566 ffff81033a97da48 000000008000ceb5 May 17 21:19:42 client-5 kernel: Call Trace: May 17 21:19:42 client-5 kernel: [] lock_timer_base+0x1b/0x3c May 17 21:19:42 client-5 kernel: [] __mod_timer+0x100/0x10f May 17 21:19:42 client-5 kernel: [] schedule_timeout+0x8a/0xad May 17 21:19:42 client-5 kernel: [] process_timeout+0x0/0x5 May 17 21:19:42 client-5 kernel: [] sys_epoll_wait+0x184/0x3f9 May 17 21:19:42 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:42 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:19:42 client-5 kernel: May 17 21:19:42 client-5 kernel: sssd_pam S ffff810001004420 0 3377 3365 3376 (NOTLB) May 17 21:19:42 client-5 kernel: ffff810328521e78 0000000000000086 ffff810328521e18 0000000000000000 May 17 21:19:42 client-5 kernel: 0000000000000000 000000000000000a ffff81033fcc0100 ffffffff80308b60 May 17 21:19:42 client-5 kernel: 000041f1ab8d8535 0000000000000420 ffff81033fcc02e8 000000008000ceb5 May 17 21:19:42 client-5 kernel: Call Trace: May 17 21:19:42 client-5 kernel: [] schedule_timeout+0x8a/0xad May 17 21:19:42 client-5 kernel: [] process_timeout+0x0/0x5 May 17 21:19:42 client-5 kernel: [] sys_epoll_wait+0x184/0x3f9 May 17 21:19:42 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:42 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:19:42 client-5 kernel: May 17 21:19:42 client-5 kernel: sshd S ffff810001015120 0 3379 1 3391 3365 (NOTLB) May 17 21:19:42 client-5 kernel: ffff81032871fa18 0000000000000082 0000000000000000 ffff810000021c10 May 17 21:19:42 client-5 kernel: 00000000000280d2 0000000000000006 ffff81033fc0f100 ffff81010b764080 May 17 21:19:42 client-5 kernel: 0000015917bf9cd4 00000000000085df ffff81033fc0f2e8 00000002000280d2 May 17 21:19:42 client-5 kernel: Call Trace: May 17 21:19:42 client-5 kernel: [] schedule_timeout+0x1e/0xad May 17 21:19:42 client-5 kernel: [] add_wait_queue+0x24/0x34 May 17 21:19:42 client-5 kernel: [] tcp_poll+0x1c/0x132 May 17 21:19:42 client-5 kernel: [] do_select+0x3fe/0x462 May 17 21:19:42 client-5 kernel: [] __pollwait+0x0/0xe2 May 17 21:19:42 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:42 client-5 last message repeated 2 times May 17 21:19:42 client-5 kernel: [] __alloc_skb+0x77/0x12d May 17 21:19:42 client-5 kernel: [] sock_alloc_send_pskb+0x7d/0x282 May 17 21:19:42 client-5 kernel: [] memcpy_fromiovec+0x36/0x66 May 17 21:19:42 client-5 kernel: [] skb_queue_tail+0x17/0x3e May 17 21:19:42 client-5 kernel: [] sock_def_readable+0x10/0x62 May 17 21:19:42 client-5 kernel: [] unix_stream_sendmsg+0x281/0x346 May 17 21:19:42 client-5 kernel: [] flush_tlb_page+0xac/0xda May 17 21:19:42 client-5 kernel: [] vsnprintf+0x5e7/0x62f May 17 21:19:42 client-5 kernel: [] core_sys_select+0x1bc/0x265 May 17 21:19:42 client-5 kernel: [] math_state_restore+0x23/0x4c May 17 21:19:42 client-5 kernel: [] error_exit+0x0/0x84 May 17 21:19:42 client-5 kernel: [] sys_rt_sigreturn+0x283/0x356 May 17 21:19:42 client-5 kernel: [] sys_rt_sigreturn+0x323/0x356 May 17 21:19:42 client-5 kernel: [] sys_select+0xc1/0x17c May 17 21:19:42 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:19:42 client-5 kernel: May 17 21:19:42 client-5 kernel: xinetd S 0000000000000040 0 3391 1 11742 3402 3379 (NOTLB) May 17 21:19:42 client-5 kernel: ffff8103287d1a18 0000000000000082 ffff81033b464880 ffff81033b464880 May 17 21:19:42 client-5 kernel: 0000000010008040 0000000000000009 ffff81033a4dd100 ffff81017de92080 May 17 21:19:42 client-5 kernel: 0000401690fd3a6f 000000000000cc2d ffff81033a4dd2e8 0000000300000009 May 17 21:19:42 client-5 kernel: Call Trace: May 17 21:19:42 client-5 kernel: [] schedule_timeout+0x1e/0xad May 17 21:19:42 client-5 kernel: [] add_wait_queue+0x24/0x34 May 17 21:19:42 client-5 kernel: [] tcp_poll+0x1c/0x132 May 17 21:19:42 client-5 kernel: [] do_select+0x3fe/0x462 May 17 21:19:42 client-5 kernel: [] __pollwait+0x0/0xe2 May 17 21:19:42 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:42 client-5 last message repeated 2 times May 17 21:19:42 client-5 kernel: [] sock_def_readable+0x37/0x62 May 17 21:19:42 client-5 kernel: [] unix_dgram_sendmsg+0x44e/0x4e0 May 17 21:19:42 client-5 kernel: [] flush_tlb_page+0xac/0xda May 17 21:19:42 client-5 kernel: [] do_wp_page+0x3fd/0x902 May 17 21:19:42 client-5 kernel: [] autoremove_wake_function+0x0/0x2e May 17 21:19:42 client-5 kernel: [] core_sys_select+0x1bc/0x265 May 17 21:19:42 client-5 kernel: [] _atomic_dec_and_lock+0x39/0x57 May 17 21:19:42 client-5 kernel: [] mntput_no_expire+0x19/0x89 May 17 21:19:42 client-5 kernel: [] __up_read+0x19/0x7f May 17 21:19:42 client-5 kernel: [] do_page_fault+0x4fe/0x874 May 17 21:19:42 client-5 kernel: [] sys_select+0xc1/0x17c May 17 21:19:42 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:19:42 client-5 kernel: May 17 21:19:42 client-5 kernel: ntpd S ffff81000100caa0 0 3402 1 3420 3391 (NOTLB) May 17 21:19:42 client-5 kernel: ffff810328105a18 0000000000000086 ffff81033c0ca000 ffffffff80064bbd May 17 21:19:42 client-5 kernel: ffff81033c0ca000 000000000000000a ffff81033c5467e0 ffff81010b70e100 May 17 21:19:42 client-5 kernel: 000041f2dce9a0af 000000000000366e ffff81033c5469c8 0000000128105ac8 May 17 21:19:42 client-5 kernel: Call Trace: May 17 21:19:42 client-5 kernel: [] _write_lock_bh+0x9/0x1c May 17 21:19:42 client-5 kernel: [] schedule_timeout+0x1e/0xad May 17 21:19:42 client-5 kernel: [] datagram_poll+0x18/0xcb May 17 21:19:42 client-5 kernel: [] udp_poll+0x15/0xf3 May 17 21:19:42 client-5 kernel: [] do_select+0x3fe/0x462 May 17 21:19:42 client-5 kernel: [] __pollwait+0x0/0xe2 May 17 21:19:42 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:42 client-5 last message repeated 7 times May 17 21:19:42 client-5 kernel: [] mntput_no_expire+0x19/0x89 May 17 21:19:42 client-5 kernel: [] core_sys_select+0x1bc/0x265 May 17 21:19:42 client-5 kernel: [] math_state_restore+0x23/0x4c May 17 21:19:42 client-5 kernel: [] error_exit+0x0/0x84 May 17 21:19:42 client-5 kernel: [] sys_rt_sigreturn+0x283/0x356 May 17 21:19:42 client-5 kernel: [] sys_rt_sigreturn+0x323/0x356 May 17 21:19:42 client-5 kernel: [] sys_select+0xc1/0x17c May 17 21:19:42 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:19:42 client-5 kernel: May 17 21:19:42 client-5 kernel: sendmail S ffff81000101d7a0 0 3420 1 3428 3402 (NOTLB) May 17 21:19:42 client-5 kernel: ffff8103283a5a18 0000000000000086 ffff81033b9dc3c0 ffff81033b9dc3c0 May 17 21:19:42 client-5 kernel: 0000000010008040 000000000000000a ffff81033b591820 ffff81010b770100 May 17 21:19:42 client-5 kernel: 000041f018fb7b78 000000000000ba0b ffff81033b591a08 000000030000000a May 17 21:19:42 client-5 kernel: Call Trace: May 17 21:19:42 client-5 kernel: [] tcp_poll+0x0/0x132 May 17 21:19:42 client-5 kernel: [] schedule_timeout+0x8a/0xad May 17 21:19:42 client-5 kernel: [] process_timeout+0x0/0x5 May 17 21:19:42 client-5 kernel: [] do_select+0x3fe/0x462 May 17 21:19:42 client-5 kernel: [] __pollwait+0x0/0xe2 May 17 21:19:42 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:42 client-5 kernel: [] cache_alloc_refill+0x106/0x186 May 17 21:19:42 client-5 kernel: [] proc_alloc_inode+0x40/0x66 May 17 21:19:42 client-5 kernel: [] alloc_inode+0xeb/0x192 May 17 21:19:42 client-5 kernel: [] iget_locked+0xc5/0x149 May 17 21:19:42 client-5 kernel: [] wake_up_bit+0x11/0x22 May 17 21:19:42 client-5 kernel: [] inotify_d_instantiate+0x12/0x3c May 17 21:19:42 client-5 kernel: [] d_instantiate+0x4a/0x82 May 17 21:19:42 client-5 kernel: [] d_rehash+0x21/0x34 May 17 21:19:42 client-5 kernel: [] vsnprintf+0x5e7/0x62f May 17 21:19:42 client-5 kernel: [] dput+0x2c/0x114 May 17 21:19:42 client-5 kernel: [] sprintf+0x51/0x59 May 17 21:19:42 client-5 kernel: [] core_sys_select+0x1bc/0x265 May 17 21:19:42 client-5 kernel: [] loadavg_read_proc+0xae/0xda May 17 21:19:42 client-5 kernel: [] lock_kernel+0x1b/0x32 May 17 21:19:42 client-5 kernel: [] de_put+0xe/0x63 May 17 21:19:42 client-5 kernel: [] sys_select+0x153/0x17c May 17 21:19:42 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:19:42 client-5 kernel: May 17 21:19:42 client-5 kernel: sendmail S ffff81000100caa0 0 3428 1 3437 3420 (NOTLB) May 17 21:19:42 client-5 kernel: ffff810327d6ff68 0000000000000086 0000000000000000 ffffffff80043b90 May 17 21:19:42 client-5 kernel: ffff81033b2520c0 0000000000000007 ffff81033b2520c0 ffff81010b70e100 May 17 21:19:42 client-5 kernel: 0000418088fb323c 00000000000043db ffff81033b2522a8 00000001800b7649 May 17 21:19:42 client-5 kernel: Call Trace: May 17 21:19:42 client-5 kernel: [] sys_rt_sigreturn+0x323/0x356 May 17 21:19:42 client-5 kernel: [] sys_pause+0x19/0x22 May 17 21:19:42 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:19:42 client-5 kernel: May 17 21:19:42 client-5 kernel: gpm S 0000000000000020 0 3437 1 3453 3428 (NOTLB) May 17 21:19:42 client-5 kernel: ffff8103281e3a18 0000000000000086 0000000000000008 0000000000000000 May 17 21:19:42 client-5 kernel: 0000000000000000 0000000000000001 ffff81033f5c1860 ffff81033b90e7e0 May 17 21:19:42 client-5 kernel: 000000204fbf296d 0000000000028bb8 ffff81033f5c1a48 0000000000000000 May 17 21:19:42 client-5 kernel: Call Trace: May 17 21:19:42 client-5 kernel: [] vsnprintf+0x5e7/0x62f May 17 21:19:42 client-5 kernel: [] lock_timer_base+0x1b/0x3c May 17 21:19:42 client-5 kernel: [] __mod_timer+0x100/0x10f May 17 21:19:42 client-5 kernel: [] schedule_timeout+0x8a/0xad May 17 21:19:42 client-5 kernel: [] process_timeout+0x0/0x5 May 17 21:19:42 client-5 kernel: [] do_select+0x3fe/0x462 May 17 21:19:42 client-5 kernel: [] __pollwait+0x0/0xe2 May 17 21:19:42 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:42 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:42 client-5 kernel: [] vsnprintf+0x5e7/0x62f May 17 21:19:42 client-5 kernel: [] vsnprintf+0x400/0x62f May 17 21:19:42 client-5 kernel: [] sprintf+0x51/0x59 May 17 21:19:42 client-5 kernel: [] __wake_up+0x38/0x4f May 17 21:19:42 client-5 kernel: [] __next_cpu+0x19/0x28 May 17 21:19:42 client-5 kernel: [] smp_send_reschedule+0x4e/0x53 May 17 21:19:42 client-5 kernel: [] find_get_page+0x21/0x51 May 17 21:19:42 client-5 kernel: [] enqueue_task+0x41/0x56 May 17 21:19:42 client-5 kernel: [] filemap_nopage+0x193/0x360 May 17 21:19:42 client-5 kernel: [] core_sys_select+0x1bc/0x265 May 17 21:19:42 client-5 kernel: [] __wake_up_common+0x3e/0x68 May 17 21:19:42 client-5 kernel: [] __up_read+0x19/0x7f May 17 21:19:42 client-5 kernel: [] do_page_fault+0x4fe/0x874 May 17 21:19:42 client-5 kernel: [] __up_wakeup+0x35/0x67 May 17 21:19:42 client-5 kernel: [] class_device_create_release+0x0/0x5 May 17 21:19:42 client-5 kernel: [] mousedev_read+0x153/0x188 May 17 21:19:42 client-5 kernel: [] sys_select+0x153/0x17c May 17 21:19:42 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:19:42 client-5 kernel: May 17 21:19:42 client-5 kernel: crond S ffff810001004420 0 3453 1 3472 3437 (NOTLB) May 17 21:19:42 client-5 kernel: ffff810327d65eb8 0000000000000086 ffff810327d65e58 ffff810001006500 May 17 21:19:42 client-5 kernel: 0000000df8475800 000000000000000a ffff81033fbaf040 ffffffff80308b60 May 17 21:19:42 client-5 kernel: 000041ead7c01a50 000000000000d9f8 ffff81033fbaf228 00000000027b73a7 May 17 21:19:42 client-5 kernel: Call Trace: May 17 21:19:42 client-5 kernel: [] enqueue_hrtimer+0x55/0x70 May 17 21:19:42 client-5 kernel: [] do_nanosleep+0x3f/0x70 May 17 21:19:42 client-5 kernel: [] hrtimer_nanosleep+0x58/0x118 May 17 21:19:42 client-5 kernel: [] hrtimer_wakeup+0x0/0x22 May 17 21:19:42 client-5 kernel: [] sys_nanosleep+0x4c/0x62 May 17 21:19:42 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:19:42 client-5 kernel: May 17 21:19:42 client-5 kernel: atd S ffff810001004420 0 3472 1 3509 3453 (NOTLB) May 17 21:19:42 client-5 kernel: ffff810327e99eb8 0000000000000086 ffff810327e99e58 ffff810001006500 May 17 21:19:42 client-5 kernel: 00000045d964b800 0000000000000007 ffff81033b740100 ffffffff80308b60 May 17 21:19:42 client-5 kernel: 000041c65669a529 000000000000a24b ffff81033b7402e8 0000000052dbb0bf May 17 21:19:42 client-5 kernel: Call Trace: May 17 21:19:42 client-5 kernel: [] enqueue_hrtimer+0x55/0x70 May 17 21:19:42 client-5 kernel: [] do_nanosleep+0x3f/0x70 May 17 21:19:42 client-5 kernel: [] hrtimer_nanosleep+0x58/0x118 May 17 21:19:42 client-5 kernel: [] hrtimer_wakeup+0x0/0x22 May 17 21:19:42 client-5 kernel: [] sys_nanosleep+0x4c/0x62 May 17 21:19:42 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:19:42 client-5 kernel: May 17 21:19:42 client-5 kernel: smartd S ffff810001015120 0 3509 1 3513 3472 (NOTLB) May 17 21:19:42 client-5 kernel: ffff810327a3deb8 0000000000000082 ffff810327a3de58 ffff810001017200 May 17 21:19:42 client-5 kernel: 000001a3185c5000 0000000000000007 ffff810327a21820 ffff81010b764080 May 17 21:19:42 client-5 kernel: 00004180abb582fa 0000000000004989 ffff810327a21a08 00000002c9baf4b7 May 17 21:19:42 client-5 kernel: Call Trace: May 17 21:19:42 client-5 kernel: [] enqueue_hrtimer+0x55/0x70 May 17 21:19:42 client-5 kernel: [] do_nanosleep+0x3f/0x70 May 17 21:19:42 client-5 kernel: [] hrtimer_nanosleep+0x58/0x118 May 17 21:19:42 client-5 kernel: [] hrtimer_wakeup+0x0/0x22 May 17 21:19:42 client-5 kernel: [] sys_nanosleep+0x4c/0x62 May 17 21:19:42 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:19:42 client-5 kernel: May 17 21:19:42 client-5 kernel: mingetty S ffff81000100caa0 0 3513 1 3514 3509 (NOTLB) May 17 21:19:42 client-5 kernel: ffff810327a1bdb8 0000000000000086 0000000000000000 000000000000000c May 17 21:19:42 client-5 kernel: 0000000000000001 0000000000000007 ffff81033fc57100 ffff81010b70e100 May 17 21:19:42 client-5 kernel: 000000206e60dadf 000000000000182b ffff81033fc572e8 0000000180063054 May 17 21:19:42 client-5 kernel: Call Trace: May 17 21:19:42 client-5 kernel: [] schedule_timeout+0x1e/0xad May 17 21:19:42 client-5 kernel: [] add_wait_queue+0x24/0x34 May 17 21:19:42 client-5 kernel: [] read_chan+0x3d4/0x694 May 17 21:19:42 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:42 client-5 kernel: [] tty_ldisc_ref_wait+0xe/0x9b May 17 21:19:42 client-5 kernel: [] tty_read+0x72/0xc8 May 17 21:19:42 client-5 kernel: [] vfs_read+0xcb/0x171 May 17 21:19:42 client-5 kernel: [] sys_read+0x45/0x6e May 17 21:19:42 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:19:42 client-5 kernel: May 17 21:19:42 client-5 kernel: mingetty S ffff81033ea9c158 0 3514 1 3515 3513 (NOTLB) May 17 21:19:42 client-5 kernel: ffff810327e5ddb8 0000000000000086 0000000000000000 000000000000000c May 17 21:19:42 client-5 kernel: 0000000000000001 0000000000000007 ffff810332cd7860 ffff810327f437e0 May 17 21:19:42 client-5 kernel: 000000206e7fbb84 0000000000009756 ffff810332cd7a48 00000002800172da May 17 21:19:42 client-5 kernel: Call Trace: May 17 21:19:42 client-5 kernel: [] do_con_write+0x1692/0x16c6 May 17 21:19:42 client-5 kernel: [] schedule_timeout+0x1e/0xad May 17 21:19:42 client-5 kernel: [] add_wait_queue+0x24/0x34 May 17 21:19:42 client-5 kernel: [] read_chan+0x3d4/0x694 May 17 21:19:42 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:42 client-5 kernel: [] tty_ldisc_ref_wait+0xe/0x9b May 17 21:19:42 client-5 kernel: [] tty_read+0x72/0xc8 May 17 21:19:42 client-5 kernel: [] vfs_read+0xcb/0x171 May 17 21:19:42 client-5 kernel: [] sys_read+0x45/0x6e May 17 21:19:42 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:19:42 client-5 kernel: May 17 21:19:42 client-5 kernel: mingetty S ffff810001004420 0 3515 1 3516 3514 (NOTLB) May 17 21:19:42 client-5 kernel: ffff810327eeddb8 0000000000000082 0000000000000000 000000000000000c May 17 21:19:42 client-5 kernel: 0000000000000001 0000000000000006 ffff810332cd7100 ffffffff80308b60 May 17 21:19:42 client-5 kernel: 000000206e88a46c 000000000009558d ffff810332cd72e8 00000000800172da May 17 21:19:42 client-5 kernel: Call Trace: May 17 21:19:42 client-5 kernel: [] schedule_timeout+0x1e/0xad May 17 21:19:42 client-5 kernel: [] add_wait_queue+0x24/0x34 May 17 21:19:42 client-5 kernel: [] read_chan+0x3d4/0x694 May 17 21:19:42 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:42 client-5 kernel: [] tty_ldisc_ref_wait+0xe/0x9b May 17 21:19:42 client-5 kernel: [] tty_read+0x72/0xc8 May 17 21:19:42 client-5 kernel: [] vfs_read+0xcb/0x171 May 17 21:19:42 client-5 kernel: [] sys_read+0x45/0x6e May 17 21:19:42 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:19:42 client-5 kernel: May 17 21:19:42 client-5 kernel: mingetty S ffff81033ea5ea18 0 3516 1 3519 3515 (NOTLB) May 17 21:19:42 client-5 kernel: ffff810327fcddb8 0000000000000086 0000000000000000 000000000000000c May 17 21:19:42 client-5 kernel: 0000000000000001 0000000000000005 ffff810333be57a0 ffff81010b6fd7a0 May 17 21:19:42 client-5 kernel: 000000206e6dd636 00000000000dbeff ffff810333be5988 00000000800172da May 17 21:19:42 client-5 kernel: Call Trace: May 17 21:19:42 client-5 kernel: [] do_con_write+0x1692/0x16c6 May 17 21:19:42 client-5 kernel: [] schedule_timeout+0x1e/0xad May 17 21:19:42 client-5 kernel: [] add_wait_queue+0x24/0x34 May 17 21:19:42 client-5 kernel: [] read_chan+0x3d4/0x694 May 17 21:19:42 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:42 client-5 kernel: [] tty_ldisc_ref_wait+0xe/0x9b May 17 21:19:42 client-5 kernel: [] tty_read+0x72/0xc8 May 17 21:19:42 client-5 kernel: [] vfs_read+0xcb/0x171 May 17 21:19:42 client-5 kernel: [] sys_read+0x45/0x6e May 17 21:19:42 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:19:42 client-5 kernel: May 17 21:19:42 client-5 kernel: mingetty S ffff81033ea5e718 0 3519 1 3520 3516 (NOTLB) May 17 21:19:42 client-5 kernel: ffff810327f2bdb8 0000000000000082 0000000000000000 000000000000000c May 17 21:19:42 client-5 kernel: 0000000000000001 0000000000000007 ffff810333be5040 ffff810332cd7100 May 17 21:19:42 client-5 kernel: 000000206e7cfc61 0000000000020a0b ffff810333be5228 00000000800172da May 17 21:19:42 client-5 kernel: Call Trace: May 17 21:19:42 client-5 kernel: [] do_con_write+0x1692/0x16c6 May 17 21:19:42 client-5 kernel: [] schedule_timeout+0x1e/0xad May 17 21:19:42 client-5 kernel: [] add_wait_queue+0x24/0x34 May 17 21:19:42 client-5 kernel: [] read_chan+0x3d4/0x694 May 17 21:19:42 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:42 client-5 kernel: [] tty_ldisc_ref_wait+0xe/0x9b May 17 21:19:42 client-5 kernel: [] tty_read+0x72/0xc8 May 17 21:19:42 client-5 kernel: [] vfs_read+0xcb/0x171 May 17 21:19:42 client-5 kernel: [] sys_read+0x45/0x6e May 17 21:19:42 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:19:42 client-5 kernel: May 17 21:19:42 client-5 kernel: mingetty S ffff81033ea5e418 0 3520 1 3582 3519 (NOTLB) May 17 21:19:42 client-5 kernel: ffff810327c0bdb8 0000000000000082 0000000000000000 000000000000000c May 17 21:19:42 client-5 kernel: 0000000000000001 0000000000000002 ffff81033b040080 ffff810327f53820 May 17 21:19:42 client-5 kernel: 000000206e94b0be 000000000001be48 ffff81033b040268 00000000800172da May 17 21:19:42 client-5 kernel: Call Trace: May 17 21:19:42 client-5 kernel: [] do_con_write+0x1692/0x16c6 May 17 21:19:42 client-5 kernel: [] schedule_timeout+0x1e/0xad May 17 21:19:42 client-5 kernel: [] add_wait_queue+0x24/0x34 May 17 21:19:42 client-5 kernel: [] read_chan+0x3d4/0x694 May 17 21:19:42 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:42 client-5 kernel: [] tty_ldisc_ref_wait+0xe/0x9b May 17 21:19:42 client-5 kernel: [] tty_read+0x72/0xc8 May 17 21:19:42 client-5 kernel: [] vfs_read+0xcb/0x171 May 17 21:19:42 client-5 kernel: [] sys_read+0x45/0x6e May 17 21:19:42 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:19:42 client-5 kernel: May 17 21:19:42 client-5 kernel: yum-updatesd S ffff810001004420 0 3582 1 3584 3520 (NOTLB) May 17 21:19:42 client-5 kernel: ffff8103267bdb38 0000000000000082 000041814507b2cf ffff810001004420 May 17 21:19:42 client-5 kernel: ffff8103267bdab8 000000000000000a ffff810327ad0080 ffffffff80308b60 May 17 21:19:42 client-5 kernel: 00004181531c9c93 000000000001981c ffff810327ad0268 0000000080046edb May 17 21:19:42 client-5 kernel: Call Trace: May 17 21:19:42 client-5 kernel: [] schedule_timeout+0x8a/0xad May 17 21:19:42 client-5 kernel: [] process_timeout+0x0/0x5 May 17 21:19:42 client-5 kernel: [] do_sys_poll+0x277/0x360 May 17 21:19:42 client-5 kernel: [] __pollwait+0x0/0xe2 May 17 21:19:42 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:42 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:42 client-5 kernel: [] flush_tlb_page+0x57/0xda May 17 21:19:42 client-5 kernel: [] flush_tlb_page+0xac/0xda May 17 21:19:42 client-5 kernel: [] do_wp_page+0x3fd/0x902 May 17 21:19:42 client-5 kernel: [] __handle_mm_fault+0xf6b/0x1039 May 17 21:19:42 client-5 kernel: [] do_page_fault+0x4fe/0x874 May 17 21:19:42 client-5 kernel: [] release_task+0x3a5/0x3cb May 17 21:19:42 client-5 kernel: [] remove_wait_queue+0x1c/0x2c May 17 21:19:42 client-5 kernel: [] audit_syscall_entry+0x180/0x1b3 May 17 21:19:42 client-5 kernel: [] sys_poll+0x2d/0x34 May 17 21:19:42 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:19:42 client-5 kernel: May 17 21:19:42 client-5 kernel: gam_server S ffff810001004420 0 3584 1 3630 3582 (NOTLB) May 17 21:19:42 client-5 kernel: ffff810325b91b38 0000000000000086 ffff810000020c00 00000002000204d0 May 17 21:19:42 client-5 kernel: 0000000100000000 000000000000000a ffff810327b88100 ffffffff80308b60 May 17 21:19:42 client-5 kernel: 000041f323300713 000000000000348f ffff810327b882e8 0000000000000000 May 17 21:19:42 client-5 kernel: Call Trace: May 17 21:19:42 client-5 kernel: [] schedule_timeout+0x8a/0xad May 17 21:19:42 client-5 kernel: [] process_timeout+0x0/0x5 May 17 21:19:42 client-5 kernel: [] do_sys_poll+0x277/0x360 May 17 21:19:42 client-5 kernel: [] __pollwait+0x0/0xe2 May 17 21:19:42 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:42 client-5 last message repeated 4 times May 17 21:19:42 client-5 kernel: [] do_sock_write+0xc6/0x102 May 17 21:19:42 client-5 kernel: [] sock_aio_write+0x4f/0x5e May 17 21:19:42 client-5 kernel: [] _atomic_dec_and_lock+0x39/0x57 May 17 21:19:42 client-5 kernel: [] do_sync_write+0xc7/0x104 May 17 21:19:42 client-5 kernel: [] prepare_to_wait+0x34/0x61 May 17 21:19:42 client-5 kernel: [] autoremove_wake_function+0x0/0x2e May 17 21:19:42 client-5 kernel: [] autoremove_wake_function+0x0/0x2e May 17 21:19:42 client-5 kernel: [] audit_syscall_entry+0x180/0x1b3 May 17 21:19:42 client-5 kernel: [] sys_poll+0x2d/0x34 May 17 21:19:42 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:19:42 client-5 kernel: May 17 21:19:42 client-5 kernel: login S ffff810001015120 0 3630 1 3631 28792 3584 (NOTLB) May 17 21:19:42 client-5 kernel: ffff810325f7dee8 0000000000000086 0000000000000292 ffffffff80022205 May 17 21:19:42 client-5 kernel: ffff81033c448500 0000000000000008 ffff81033aabe7a0 ffff81010b764080 May 17 21:19:42 client-5 kernel: 0000002c1049f6f9 0000000000004bf2 ffff81033aabe988 000000023aabe7a0 May 17 21:19:42 client-5 kernel: Call Trace: May 17 21:19:42 client-5 kernel: [] __up_read+0x19/0x7f May 17 21:19:42 client-5 kernel: [] do_wait+0xa03/0xad7 May 17 21:19:42 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:42 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:19:42 client-5 kernel: May 17 21:19:42 client-5 kernel: bash S 0000000000000001 0 3631 3630 3677 (NOTLB) May 17 21:19:42 client-5 kernel: ffff810327fb9ee8 0000000000000082 ffff81033f7d1800 0000000000000000 May 17 21:19:42 client-5 kernel: ffff810327fb9e88 0000000000000009 ffff81033b547100 ffff81033b6f67e0 May 17 21:19:42 client-5 kernel: 00000036f5d18ca9 000000000000ea52 ffff81033b5472e8 0000000200000001 May 17 21:19:42 client-5 kernel: Call Trace: May 17 21:19:42 client-5 kernel: [] do_wait+0xa03/0xad7 May 17 21:19:42 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:42 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:19:42 client-5 kernel: May 17 21:19:42 client-5 kernel: sh S 0000000000000001 0 3677 3631 29246 (NOTLB) May 17 21:19:42 client-5 kernel: ffff810325803ee8 0000000000000082 0000000000000292 ffffffff80022205 May 17 21:19:42 client-5 kernel: ffff81033c4fdc40 0000000000000002 ffff81033b6f67e0 ffff810321d6c0c0 May 17 21:19:42 client-5 kernel: 00003ffe05ba4b64 000000000000dfcf ffff81033b6f69c8 000000023b6f67e0 May 17 21:19:42 client-5 kernel: Call Trace: May 17 21:19:42 client-5 kernel: [] __up_read+0x19/0x7f May 17 21:19:42 client-5 kernel: [] do_wait+0xa03/0xad7 May 17 21:19:42 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:42 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:19:42 client-5 kernel: May 17 21:19:42 client-5 kernel: nfsiod S 0000000000000282 0 932 123 3131 (L-TLB) May 17 21:19:42 client-5 kernel: ffff810310adfe70 0000000000000046 00002b320234ac06 0000000000000001 May 17 21:19:42 client-5 kernel: ffff81030be69860 0000000000000006 ffff81030be69860 ffff81033fb790c0 May 17 21:19:42 client-5 kernel: 00002b32023b9049 000000000000064a ffff81030be69a48 0000000132d65540 May 17 21:19:42 client-5 kernel: Call Trace: May 17 21:19:42 client-5 kernel: [] thread_return+0x89/0xfe May 17 21:19:42 client-5 kernel: [] recalc_sigpending_and_wake+0x9/0x1a May 17 21:19:42 client-5 kernel: [] do_sigaction+0x14d/0x199 May 17 21:19:42 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:42 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:19:42 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:19:42 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:42 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:42 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:42 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:19:42 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:19:42 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:19:42 client-5 kernel: [] kthread+0x0/0x132 May 17 21:19:42 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:19:42 client-5 kernel: May 17 21:19:42 client-5 kernel: in.rlogind S ffff810001015120 0 11742 3391 11743 (NOTLB) May 17 21:19:42 client-5 kernel: ffff81031936da18 0000000000000086 ffff81033cb4b0c0 0000000000000000 May 17 21:19:42 client-5 kernel: ffffffff80013b95 000000000000000a ffff81033fc69820 ffff81010b764080 May 17 21:19:42 client-5 kernel: 000041f06da3d0da 00000000000048cf ffff81033fc69a08 0000000200000282 May 17 21:19:42 client-5 kernel: Call Trace: May 17 21:19:42 client-5 kernel: [] tcp_poll+0x0/0x132 May 17 21:19:42 client-5 kernel: [] schedule_timeout+0x1e/0xad May 17 21:19:42 client-5 kernel: [] tty_poll+0x5f/0x6d May 17 21:19:42 client-5 kernel: [] do_select+0x3fe/0x462 May 17 21:19:42 client-5 kernel: [] __pollwait+0x0/0xe2 May 17 21:19:42 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:42 client-5 last message repeated 2 times May 17 21:19:42 client-5 kernel: [] lock_timer_base+0x1b/0x3c May 17 21:19:42 client-5 kernel: [] __mod_timer+0x100/0x10f May 17 21:19:42 client-5 kernel: [] sk_reset_timer+0xf/0x19 May 17 21:19:42 client-5 kernel: [] __tcp_push_pending_frames+0x782/0x85c May 17 21:19:42 client-5 kernel: [] __alloc_skb+0x77/0x12d May 17 21:19:42 client-5 kernel: [] _spin_lock_bh+0x9/0x14 May 17 21:19:42 client-5 kernel: [] release_sock+0x13/0xaa May 17 21:19:42 client-5 kernel: [] tcp_sendmsg+0xa07/0xb0b May 17 21:19:42 client-5 kernel: [] do_sock_write+0xc6/0x102 May 17 21:19:42 client-5 kernel: [] core_sys_select+0x1bc/0x265 May 17 21:19:42 client-5 kernel: [] remove_wait_queue+0x1c/0x2c May 17 21:19:42 client-5 kernel: [] do_sync_write+0xc7/0x104 May 17 21:19:42 client-5 kernel: [] read_chan+0x626/0x694 May 17 21:19:42 client-5 kernel: [] autoremove_wake_function+0x0/0x2e May 17 21:19:42 client-5 kernel: [] tty_ldisc_deref+0x68/0x7b May 17 21:19:42 client-5 kernel: [] sys_select+0xc1/0x17c May 17 21:19:42 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:19:42 client-5 kernel: May 17 21:19:42 client-5 kernel: login S ffff810001004420 0 11743 11742 11744 (NOTLB) May 17 21:19:42 client-5 kernel: ffff8103119a9ee8 0000000000000082 0000000000000292 ffffffff80022205 May 17 21:19:42 client-5 kernel: ffff81033cb4bb40 0000000000000007 ffff8102d94087e0 ffffffff80308b60 May 17 21:19:42 client-5 kernel: 00003de93ce0f0aa 0000000000005961 ffff8102d94089c8 00000000d94087e0 May 17 21:19:42 client-5 kernel: Call Trace: May 17 21:19:42 client-5 kernel: [] __up_read+0x19/0x7f May 17 21:19:42 client-5 kernel: [] do_wait+0xa03/0xad7 May 17 21:19:42 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:42 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:19:42 client-5 kernel: May 17 21:19:42 client-5 kernel: bash R running task 0 11744 11743 (NOTLB) May 17 21:19:42 client-5 kernel: 00000000000d0b2c 00000000000d0b55 fffffffffffe0000 000000000000002c May 17 21:19:42 client-5 kernel: ffff81010b7bffc0 ffffffff80091c0f 00000000000d0b55 00000000000d0b55 May 17 21:19:42 client-5 kernel: 0000000000000046 ffffffff800172da 0000000000000082 0000000000000000 May 17 21:19:42 client-5 kernel: Call Trace: May 17 21:19:42 client-5 kernel: [] __call_console_drivers+0x5b/0x69 May 17 21:19:42 client-5 kernel: [] release_console_sem+0x1ba/0x20e May 17 21:19:42 client-5 kernel: [] serial8250_console_putchar+0x3f/0xa5 May 17 21:19:42 client-5 kernel: [] serial8250_console_putchar+0x0/0xa5 May 17 21:19:42 client-5 last message repeated 2 times May 17 21:19:42 client-5 kernel: [] printk+0x52/0xbd May 17 21:19:42 client-5 kernel: [] printk+0x52/0xbd May 17 21:19:42 client-5 kernel: [] release_console_sem+0x1ba/0x20e May 17 21:19:42 client-5 kernel: [] kallsyms_lookup+0xe6/0x1ae May 17 21:19:42 client-5 last message repeated 3 times May 17 21:19:42 client-5 kernel: [] printk_address+0x9f/0xab May 17 21:19:42 client-5 kernel: [] gart_map_sg+0x1cf/0x401 May 17 21:19:42 client-5 kernel: [] printk+0x52/0xbd May 17 21:19:42 client-5 kernel: [] module_text_address+0x33/0x3c May 17 21:19:42 client-5 kernel: [] kernel_text_address+0x1a/0x26 May 17 21:19:42 client-5 kernel: [] dump_trace+0x206/0x22f May 17 21:19:42 client-5 kernel: [] show_trace+0x34/0x47 May 17 21:19:42 client-5 kernel: [] _show_stack+0xdb/0xea May 17 21:19:42 client-5 kernel: [] show_state+0x48/0x86 May 17 21:19:42 client-5 kernel: [] __handle_sysrq+0x90/0x121 May 17 21:19:42 client-5 kernel: [] write_sysrq_trigger+0x2a/0x32 May 17 21:19:42 client-5 kernel: [] vfs_write+0xce/0x174 May 17 21:19:42 client-5 kernel: [] sys_write+0x45/0x6e May 17 21:19:42 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:19:42 client-5 kernel: May 17 21:19:42 client-5 kernel: cfs_wi_sd000 S 0000000000000000 0 28792 1 28793 3630 (L-TLB) May 17 21:19:42 client-5 kernel: ffff8103192f1ec0 0000000000000046 00000000ffffffff 0000000000000001 May 17 21:19:42 client-5 kernel: 0000000000000003 0000000000000001 ffff81027425b0c0 ffff810309801860 May 17 21:19:42 client-5 kernel: 00003ffddde6c79b 0000000000001469 ffff81027425b2a8 00000000192f1f18 May 17 21:19:42 client-5 kernel: Call Trace: May 17 21:19:42 client-5 kernel: [] :libcfs:cfs_wi_scheduler+0x239/0x2a0 May 17 21:19:42 client-5 kernel: [] autoremove_wake_function+0x0/0x2e May 17 21:19:42 client-5 kernel: [] audit_syscall_exit+0x336/0x362 May 17 21:19:42 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:19:42 client-5 kernel: [] :libcfs:cfs_wi_scheduler+0x0/0x2a0 May 17 21:19:42 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:19:42 client-5 kernel: May 17 21:19:42 client-5 kernel: cfs_wi_sd001 S 0000000000000000 0 28793 1 28794 28792 (L-TLB) May 17 21:19:42 client-5 kernel: ffff81030269dec0 0000000000000046 00000000ffffffff 0000000000000001 May 17 21:19:42 client-5 kernel: 0000000000000003 0000000000000001 ffff810309801860 ffff8102f44587e0 May 17 21:19:42 client-5 kernel: 00003ffddde6cfcf 0000000000000834 ffff810309801a48 000000000269df18 May 17 21:19:42 client-5 kernel: Call Trace: May 17 21:19:42 client-5 kernel: [] :libcfs:cfs_wi_scheduler+0x239/0x2a0 May 17 21:19:42 client-5 kernel: [] autoremove_wake_function+0x0/0x2e May 17 21:19:42 client-5 kernel: [] audit_syscall_exit+0x336/0x362 May 17 21:19:42 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:19:42 client-5 kernel: [] :libcfs:cfs_wi_scheduler+0x0/0x2a0 May 17 21:19:42 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:19:42 client-5 kernel: May 17 21:19:42 client-5 kernel: cfs_wi_sd002 S 0000000000000000 0 28794 1 28795 28793 (L-TLB) May 17 21:19:42 client-5 kernel: ffff810314b51ec0 0000000000000046 00000000ffffffff 0000000000000001 May 17 21:19:42 client-5 kernel: 0000000000000003 0000000000000001 ffff8102f44587e0 ffff81027425b820 May 17 21:19:42 client-5 kernel: 00003ffddde6d84b 000000000000087c ffff8102f44589c8 0000000014b51f18 May 17 21:19:42 client-5 kernel: Call Trace: May 17 21:19:42 client-5 kernel: [] :libcfs:cfs_wi_scheduler+0x239/0x2a0 May 17 21:19:42 client-5 kernel: [] autoremove_wake_function+0x0/0x2e May 17 21:19:42 client-5 kernel: [] audit_syscall_exit+0x336/0x362 May 17 21:19:42 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:19:42 client-5 kernel: [] :libcfs:cfs_wi_scheduler+0x0/0x2a0 May 17 21:19:42 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:19:42 client-5 kernel: May 17 21:19:42 client-5 kernel: cfs_wi_sd003 S 0000000000000000 0 28795 1 28796 28794 (L-TLB) May 17 21:19:42 client-5 kernel: ffff81031bc73ec0 0000000000000046 00000000ffffffff 0000000000000001 May 17 21:19:42 client-5 kernel: 0000000000000003 0000000000000001 ffff81027425b820 ffff81031a8f10c0 May 17 21:19:42 client-5 kernel: 00003ffddde6dfc5 000000000000077a ffff81027425ba08 0000000000000030 May 17 21:19:42 client-5 kernel: Call Trace: May 17 21:19:42 client-5 kernel: [] :libcfs:cfs_wi_scheduler+0x239/0x2a0 May 17 21:19:42 client-5 kernel: [] autoremove_wake_function+0x0/0x2e May 17 21:19:42 client-5 kernel: [] audit_syscall_exit+0x336/0x362 May 17 21:19:42 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:19:42 client-5 kernel: [] :libcfs:cfs_wi_scheduler+0x0/0x2a0 May 17 21:19:42 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:19:42 client-5 kernel: May 17 21:19:42 client-5 kernel: wi_serial_sd S 0000000000000000 0 28796 1 28859 28795 (L-TLB) May 17 21:19:42 client-5 kernel: ffff810326cf1ec0 0000000000000046 ffff81031d5ee878 000000000000003f May 17 21:19:42 client-5 kernel: ffffffff80359d38 0000000000000001 ffff81031a8f10c0 ffff81031526e820 May 17 21:19:42 client-5 kernel: 00003ffddde6e706 0000000000000741 ffff81031a8f12a8 0000000000000000 May 17 21:19:42 client-5 kernel: Call Trace: May 17 21:19:42 client-5 kernel: [] :libcfs:cfs_wi_scheduler+0x239/0x2a0 May 17 21:19:42 client-5 kernel: [] autoremove_wake_function+0x0/0x2e May 17 21:19:42 client-5 kernel: [] audit_syscall_exit+0x336/0x362 May 17 21:19:42 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:19:42 client-5 kernel: [] :libcfs:cfs_wi_scheduler+0x0/0x2a0 May 17 21:19:42 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:19:42 client-5 kernel: May 17 21:19:42 client-5 kernel: obd_zombid S 00002ad503746010 0 28859 1 28878 28796 (L-TLB) May 17 21:19:42 client-5 kernel: ffff81031f049ec0 0000000000000046 0000000000061a04 000000000036c752 May 17 21:19:42 client-5 kernel: 0000000100000001 0000000000000006 ffff81031ec2e860 ffff81031931b0c0 May 17 21:19:42 client-5 kernel: 00003ffde5e89e0c 00000000000020bf ffff81031ec2ea48 0000000000000000 May 17 21:19:42 client-5 kernel: Call Trace: May 17 21:19:42 client-5 kernel: [] __wake_up_common+0x3e/0x68 May 17 21:19:42 client-5 kernel: [] :obdclass:obd_zombie_impexp_check+0x11/0xc0 May 17 21:19:42 client-5 kernel: [] :obdclass:obd_zombie_impexp_thread+0x18d/0x2a0 May 17 21:19:42 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:42 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:19:42 client-5 kernel: [] :obdclass:obd_zombie_impexp_thread+0x0/0x2a0 May 17 21:19:42 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:19:42 client-5 kernel: May 17 21:19:42 client-5 kernel: ptlrpc_hr_0 S ffff8102e80ad820 0 28878 1 28879 28859 (L-TLB) May 17 21:19:42 client-5 kernel: ffff8102c611be40 0000000000000046 ffff8102c611bd90 0000000000000086 May 17 21:19:42 client-5 kernel: 0000000000445299 0000000000000001 ffff81031931b0c0 ffff8102e80ad820 May 17 21:19:42 client-5 kernel: 00003ffdea1bd3b5 0000000000002e3c ffff81031931b2a8 000000008008b3c7 May 17 21:19:42 client-5 kernel: Call Trace: May 17 21:19:42 client-5 kernel: [] __wake_up+0x38/0x4f May 17 21:19:42 client-5 kernel: [] :ptlrpc:ptlrpc_hr_main+0x19f/0x650 May 17 21:19:42 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:42 client-5 kernel: [] audit_syscall_exit+0x336/0x362 May 17 21:19:42 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:19:42 client-5 kernel: [] :ptlrpc:ptlrpc_hr_main+0x0/0x650 May 17 21:19:42 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:19:42 client-5 kernel: May 17 21:19:42 client-5 kernel: ptlrpc_hr_1 S ffff8102e80ad820 0 28879 1 28880 28878 (L-TLB) May 17 21:19:42 client-5 kernel: ffff8103174b9e40 0000000000000046 ffff8103174b9d90 0000000000000086 May 17 21:19:42 client-5 kernel: 000000000023201d 0000000000000001 ffff81033fb647e0 ffff8102e80ad820 May 17 21:19:42 client-5 kernel: 00003ffdea1bfbee 000000000000119b ffff81033fb649c8 000000008008b3c7 May 17 21:19:42 client-5 kernel: Call Trace: May 17 21:19:42 client-5 kernel: [] __wake_up+0x38/0x4f May 17 21:19:42 client-5 kernel: [] :ptlrpc:ptlrpc_hr_main+0x19f/0x650 May 17 21:19:42 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:42 client-5 kernel: [] audit_syscall_exit+0x336/0x362 May 17 21:19:42 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:19:42 client-5 kernel: [] :ptlrpc:ptlrpc_hr_main+0x0/0x650 May 17 21:19:42 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:19:42 client-5 kernel: May 17 21:19:42 client-5 kernel: ptlrpc_hr_2 S ffff8102e80ad820 0 28880 1 28881 28879 (L-TLB) May 17 21:19:42 client-5 kernel: ffff81030f55be40 0000000000000046 ffff81030f55bd90 0000000000000086 May 17 21:19:42 client-5 kernel: 000000180000000a 0000000000000001 ffff81033b27c7e0 ffff8102e80ad820 May 17 21:19:42 client-5 kernel: 00003ffdea1c2521 0000000000000bd1 ffff81033b27c9c8 000000008008b3c7 May 17 21:19:42 client-5 kernel: Call Trace: May 17 21:19:42 client-5 kernel: [] __wake_up+0x38/0x4f May 17 21:19:42 client-5 kernel: [] :ptlrpc:ptlrpc_hr_main+0x19f/0x650 May 17 21:19:42 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:42 client-5 kernel: [] audit_syscall_exit+0x336/0x362 May 17 21:19:42 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:19:42 client-5 kernel: [] :ptlrpc:ptlrpc_hr_main+0x0/0x650 May 17 21:19:42 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:19:42 client-5 kernel: May 17 21:19:42 client-5 kernel: ptlrpc_hr_3 S ffff8102e80ad820 0 28881 1 28889 28880 (L-TLB) May 17 21:19:42 client-5 kernel: ffff81032304fe40 0000000000000046 ffff81032304fd90 0000000000000086 May 17 21:19:42 client-5 kernel: 0000000000440eca 0000000000000001 ffff81033b158820 ffff8102e80ad820 May 17 21:19:42 client-5 kernel: 00003ffdea1dd9aa 0000000000019756 ffff81033b158a08 000000008008b3c7 May 17 21:19:42 client-5 kernel: Call Trace: May 17 21:19:42 client-5 kernel: [] __wake_up+0x38/0x4f May 17 21:19:42 client-5 kernel: [] :ptlrpc:ptlrpc_hr_main+0x19f/0x650 May 17 21:19:42 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:42 client-5 kernel: [] audit_syscall_exit+0x336/0x362 May 17 21:19:42 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:19:42 client-5 kernel: [] :ptlrpc:ptlrpc_hr_main+0x0/0x650 May 17 21:19:42 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:19:42 client-5 kernel: May 17 21:19:42 client-5 kernel: kiblnd_sd_00 S ffff81000100caa0 0 28889 1 28890 28881 (L-TLB) May 17 21:19:42 client-5 kernel: ffff810321e97e50 0000000000000046 ffff8103066b5a40 000000c0000000c0 May 17 21:19:42 client-5 kernel: 0000001200000282 000000000000000a ffff810309801100 ffff81010b70e100 May 17 21:19:42 client-5 kernel: 000041f2ae57c655 000000000000027a ffff8103098012e8 0000000188a0ed9b May 17 21:19:42 client-5 kernel: Call Trace: May 17 21:19:42 client-5 kernel: [] :ko2iblnd:kiblnd_scheduler+0x66c/0x6e0 May 17 21:19:42 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:42 client-5 kernel: [] audit_syscall_exit+0x336/0x362 May 17 21:19:42 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:19:42 client-5 kernel: [] :ko2iblnd:kiblnd_scheduler+0x0/0x6e0 May 17 21:19:42 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:19:42 client-5 kernel: May 17 21:19:42 client-5 kernel: kiblnd_sd_01 S ffff810309801100 0 28890 1 28891 28889 (L-TLB) May 17 21:19:42 client-5 kernel: ffff810310681e50 0000000000000046 ffff8102f45cec40 0000000000000082 May 17 21:19:42 client-5 kernel: 0000001200000000 000000000000000a ffff81031997a040 ffff810309801100 May 17 21:19:42 client-5 kernel: 000041f2ae57c3db 0000000000000315 ffff81031997a228 0000000188a0e2bf May 17 21:19:42 client-5 kernel: Call Trace: May 17 21:19:42 client-5 kernel: [] :ko2iblnd:kiblnd_scheduler+0x66c/0x6e0 May 17 21:19:42 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:42 client-5 kernel: [] audit_syscall_exit+0x336/0x362 May 17 21:19:42 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:19:42 client-5 kernel: [] :ko2iblnd:kiblnd_scheduler+0x0/0x6e0 May 17 21:19:42 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:19:42 client-5 kernel: May 17 21:19:42 client-5 kernel: kiblnd_sd_02 S ffff81000100caa0 0 28891 1 28892 28890 (L-TLB) May 17 21:19:42 client-5 kernel: ffff81032f0b5e50 0000000000000046 ffff8103066b50c0 000000c0000000c0 May 17 21:19:42 client-5 kernel: 0000001200000000 000000000000000a ffff81031977e080 ffff81010b70e100 May 17 21:19:42 client-5 kernel: 000041f2ae570cd6 00000000000001f5 ffff81031977e268 0000000188a0ed9b May 17 21:19:42 client-5 kernel: Call Trace: May 17 21:19:42 client-5 kernel: [] :ko2iblnd:kiblnd_scheduler+0x66c/0x6e0 May 17 21:19:42 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:42 client-5 kernel: [] audit_syscall_exit+0x336/0x362 May 17 21:19:42 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:19:42 client-5 kernel: [] :ko2iblnd:kiblnd_scheduler+0x0/0x6e0 May 17 21:19:42 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:19:42 client-5 kernel: May 17 21:19:42 client-5 kernel: kiblnd_sd_03 S ffff81031997a040 0 28892 1 28893 28891 (L-TLB) May 17 21:19:42 client-5 kernel: ffff8102fa2b5e50 0000000000000046 ffff810252c21d40 000000c0000000c0 May 17 21:19:42 client-5 kernel: 0000001200000000 000000000000000a ffff81026c179080 ffff81031997a040 May 17 21:19:42 client-5 kernel: 000041f2ae57c0c6 0000000000002f54 ffff81026c179268 0000000188a0ed9b May 17 21:19:42 client-5 kernel: Call Trace: May 17 21:19:42 client-5 kernel: [] :mlx4_ib:mlx4_ib_poll_cq+0x57e/0x5b9 May 17 21:19:42 client-5 kernel: [] :ko2iblnd:kiblnd_scheduler+0x66c/0x6e0 May 17 21:19:42 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:42 client-5 kernel: [] audit_syscall_exit+0x336/0x362 May 17 21:19:42 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:19:42 client-5 kernel: [] :ko2iblnd:kiblnd_scheduler+0x0/0x6e0 May 17 21:19:42 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:19:42 client-5 kernel: May 17 21:19:42 client-5 kernel: kiblnd_connd S 000000000000005b 0 28893 1 28896 28892 (L-TLB) May 17 21:19:42 client-5 kernel: ffff8102fc199e50 0000000000000046 ffff81010b66bc80 ffffffff800939b4 May 17 21:19:42 client-5 kernel: 0000003000000008 000000000000000a ffff8102cc559860 ffff81033f5c1100 May 17 21:19:42 client-5 kernel: 000041f045ab1f03 0000000000000c03 ffff8102cc559a48 0000000388a0d4e6 May 17 21:19:42 client-5 kernel: Call Trace: May 17 21:19:42 client-5 kernel: [] daemonize+0x2fa/0x304 May 17 21:19:42 client-5 kernel: [] lock_timer_base+0x1b/0x3c May 17 21:19:42 client-5 kernel: [] __mod_timer+0x100/0x10f May 17 21:19:42 client-5 kernel: [] schedule_timeout+0x8a/0xad May 17 21:19:42 client-5 kernel: [] process_timeout+0x0/0x5 May 17 21:19:42 client-5 kernel: [] :ko2iblnd:kiblnd_connd+0x2d3/0x360 May 17 21:19:42 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:42 client-5 kernel: [] audit_syscall_exit+0x336/0x362 May 17 21:19:42 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:19:42 client-5 kernel: [] :ko2iblnd:kiblnd_connd+0x0/0x360 May 17 21:19:42 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:19:42 client-5 kernel: May 17 21:19:42 client-5 kernel: socknal_sd00 S 0000000000000000 0 28894 1 28898 28896 (L-TLB) May 17 21:19:42 client-5 kernel: ffff810310f53e60 0000000000000046 00000000ffffffff 0000000000000001 May 17 21:19:42 client-5 kernel: 0000000000000002 0000000000000001 ffff8102cab51080 ffff81033c54a040 May 17 21:19:42 client-5 kernel: 00003ffdeafd8e11 0000000000001606 ffff8102cab51268 00000000800646f9 May 17 21:19:42 client-5 kernel: Call Trace: May 17 21:19:42 client-5 kernel: [] :ksocklnd:ksocknal_scheduler+0x58b/0x640 May 17 21:19:42 client-5 kernel: [] autoremove_wake_function+0x0/0x2e May 17 21:19:42 client-5 kernel: [] audit_syscall_exit+0x336/0x362 May 17 21:19:42 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:19:42 client-5 kernel: [] :ksocklnd:ksocknal_scheduler+0x0/0x640 May 17 21:19:42 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:19:42 client-5 kernel: May 17 21:19:42 client-5 kernel: socknal_sd01 S 0000000000000000 0 28895 1 28897 28903 (L-TLB) May 17 21:19:42 client-5 kernel: ffff8102fc389e60 0000000000000046 00000000ffffffff 0000000000000001 May 17 21:19:42 client-5 kernel: 0000000000000002 0000000000000001 ffff81033fb790c0 ffff8102e80ad820 May 17 21:19:42 client-5 kernel: 00003ffdeafe41fb 0000000000000bb1 ffff81033fb792a8 00000000800646f9 May 17 21:19:42 client-5 kernel: Call Trace: May 17 21:19:42 client-5 kernel: [] :ksocklnd:ksocknal_scheduler+0x58b/0x640 May 17 21:19:42 client-5 kernel: [] autoremove_wake_function+0x0/0x2e May 17 21:19:42 client-5 kernel: [] audit_syscall_exit+0x336/0x362 May 17 21:19:42 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:19:42 client-5 kernel: [] :ksocklnd:ksocknal_scheduler+0x0/0x640 May 17 21:19:42 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:19:42 client-5 kernel: May 17 21:19:42 client-5 kernel: socknal_sd02 S ffff810001015120 0 28896 1 28894 28893 (L-TLB) May 17 21:19:42 client-5 kernel: ffff810318cd3e60 0000000000000046 ffffffff8032bc28 0000000000000000 May 17 21:19:42 client-5 kernel: 0000000000000001 0000000000000001 ffff81033ab1b860 ffff81010b764080 May 17 21:19:42 client-5 kernel: 00003ffdeafc4a09 0000000000004854 ffff81033ab1ba48 00000002112320b0 May 17 21:19:42 client-5 kernel: Call Trace: May 17 21:19:42 client-5 kernel: [] :ksocklnd:ksocknal_scheduler+0x58b/0x640 May 17 21:19:42 client-5 kernel: [] autoremove_wake_function+0x0/0x2e May 17 21:19:42 client-5 kernel: [] audit_syscall_exit+0x336/0x362 May 17 21:19:42 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:19:42 client-5 kernel: [] :ksocklnd:ksocknal_scheduler+0x0/0x640 May 17 21:19:42 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:19:42 client-5 kernel: May 17 21:19:42 client-5 kernel: socknal_sd03 S 0000000000000000 0 28897 1 28904 28895 (L-TLB) May 17 21:19:42 client-5 kernel: ffff8102e863de60 0000000000000046 00000000ffffffff 0000000000000001 May 17 21:19:42 client-5 kernel: 0000000000000002 0000000000000001 ffff8102efe8d820 ffff81030318b820 May 17 21:19:42 client-5 kernel: 00003ffdeaffad68 0000000000000fb0 ffff8102efe8da08 00000000800646f9 May 17 21:19:42 client-5 kernel: Call Trace: May 17 21:19:42 client-5 kernel: [] :ksocklnd:ksocknal_scheduler+0x58b/0x640 May 17 21:19:42 client-5 kernel: [] autoremove_wake_function+0x0/0x2e May 17 21:19:42 client-5 kernel: [] audit_syscall_exit+0x336/0x362 May 17 21:19:42 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:19:42 client-5 kernel: [] :ksocklnd:ksocknal_scheduler+0x0/0x640 May 17 21:19:42 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:19:42 client-5 kernel: May 17 21:19:42 client-5 kernel: socknal_cd00 S ffff810330eebee0 0 28898 1 28899 28894 (L-TLB) May 17 21:19:42 client-5 kernel: ffff810330eebe40 0000000000000046 ffff81010b66bc80 ffffffff800939b4 May 17 21:19:42 client-5 kernel: 0000003000000008 0000000000000001 ffff81033c54a040 ffff8102cc559100 May 17 21:19:42 client-5 kernel: 00003ffdeafd9dfd 0000000000000fec ffff81033c54a228 0000000030eebf20 May 17 21:19:42 client-5 kernel: Call Trace: May 17 21:19:42 client-5 kernel: [] daemonize+0x2fa/0x304 May 17 21:19:42 client-5 kernel: [] schedule_timeout+0x1e/0xad May 17 21:19:42 client-5 kernel: [] add_wait_queue_exclusive+0x28/0x38 May 17 21:19:42 client-5 kernel: [] :ksocklnd:ksocknal_connd+0x415/0x4b0 May 17 21:19:42 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:42 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:19:42 client-5 kernel: [] :ksocklnd:ksocknal_connd+0x0/0x4b0 May 17 21:19:42 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:19:42 client-5 kernel: May 17 21:19:42 client-5 kernel: socknal_cd01 S ffff810327ba5ee0 0 28899 1 28900 28898 (L-TLB) May 17 21:19:42 client-5 kernel: ffff810327ba5e40 0000000000000046 ffff81010b66bc80 ffffffff800939b4 May 17 21:19:42 client-5 kernel: 0000003000000008 0000000000000001 ffff8102cc559100 ffff81031931b820 May 17 21:19:42 client-5 kernel: 00003ffdeafda8b9 0000000000000abc ffff8102cc5592e8 0000000027ba5f20 May 17 21:19:42 client-5 kernel: Call Trace: May 17 21:19:42 client-5 kernel: [] daemonize+0x2fa/0x304 May 17 21:19:42 client-5 kernel: [] schedule_timeout+0x1e/0xad May 17 21:19:42 client-5 kernel: [] add_wait_queue_exclusive+0x28/0x38 May 17 21:19:42 client-5 kernel: [] :ksocklnd:ksocknal_connd+0x415/0x4b0 May 17 21:19:42 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:42 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:19:42 client-5 kernel: [] :ksocklnd:ksocknal_connd+0x0/0x4b0 May 17 21:19:42 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:19:42 client-5 kernel: May 17 21:19:42 client-5 kernel: socknal_cd02 S ffff81030a269ee0 0 28900 1 28901 28899 (L-TLB) May 17 21:19:42 client-5 kernel: ffff81030a269e40 0000000000000046 ffff81010b66bc80 ffffffff800939b4 May 17 21:19:42 client-5 kernel: 0000003000000008 0000000000000001 ffff81031931b820 ffff810327ad07e0 May 17 21:19:42 client-5 kernel: 00003ffdeafdb279 00000000000009c0 ffff81031931ba08 000000000a269f20 May 17 21:19:42 client-5 kernel: Call Trace: May 17 21:19:42 client-5 kernel: [] daemonize+0x2fa/0x304 May 17 21:19:42 client-5 kernel: [] schedule_timeout+0x1e/0xad May 17 21:19:42 client-5 kernel: [] add_wait_queue_exclusive+0x28/0x38 May 17 21:19:42 client-5 kernel: [] :ksocklnd:ksocknal_connd+0x415/0x4b0 May 17 21:19:42 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:42 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:19:42 client-5 kernel: [] :ksocklnd:ksocknal_connd+0x0/0x4b0 May 17 21:19:42 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:19:42 client-5 kernel: May 17 21:19:42 client-5 kernel: socknal_cd03 S ffff810323945ee0 0 28901 1 28902 28900 (L-TLB) May 17 21:19:42 client-5 kernel: ffff810323945e40 0000000000000046 ffff81010b66bc80 ffffffff800939b4 May 17 21:19:42 client-5 kernel: 0000003000000008 0000000000000001 ffff810327ad07e0 ffff810321d6c820 May 17 21:19:42 client-5 kernel: 00003ffdeafdbbc0 0000000000000947 ffff810327ad09c8 0000000023945f20 May 17 21:19:42 client-5 kernel: Call Trace: May 17 21:19:42 client-5 kernel: [] daemonize+0x2fa/0x304 May 17 21:19:42 client-5 kernel: [] schedule_timeout+0x1e/0xad May 17 21:19:42 client-5 kernel: [] add_wait_queue_exclusive+0x28/0x38 May 17 21:19:42 client-5 kernel: [] :ksocklnd:ksocknal_connd+0x415/0x4b0 May 17 21:19:42 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:42 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:19:42 client-5 kernel: [] :ksocklnd:ksocknal_connd+0x0/0x4b0 May 17 21:19:42 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:19:42 client-5 kernel: May 17 21:19:42 client-5 kernel: socknal_reape S ffff81033c327080 0 28902 1 28903 28901 (L-TLB) May 17 21:19:42 client-5 kernel: ffff810303bf7e60 0000000000000046 ffff81010b66bc80 ffffffff800939b4 May 17 21:19:42 client-5 kernel: 0000003000000008 000000000000000a ffff810321d6c820 ffff81033c327080 May 17 21:19:42 client-5 kernel: 000041f386fa1baf 0000000000000a9e ffff810321d6ca08 0000000000000000 May 17 21:19:42 client-5 kernel: Call Trace: May 17 21:19:42 client-5 kernel: [] daemonize+0x2fa/0x304 May 17 21:19:42 client-5 kernel: [] :ksocklnd:ksocknal_check_peer_timeouts+0x2b/0x410 May 17 21:19:42 client-5 kernel: [] lock_timer_base+0x1b/0x3c May 17 21:19:42 client-5 kernel: [] __mod_timer+0x100/0x10f May 17 21:19:42 client-5 kernel: [] schedule_timeout+0x8a/0xad May 17 21:19:42 client-5 kernel: [] process_timeout+0x0/0x5 May 17 21:19:42 client-5 kernel: [] :ksocklnd:ksocknal_reaper+0x2ec/0x370 May 17 21:19:42 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:42 client-5 kernel: [] audit_syscall_exit+0x336/0x362 May 17 21:19:42 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:19:42 client-5 kernel: [] :ksocklnd:ksocknal_reaper+0x0/0x370 May 17 21:19:42 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:19:42 client-5 kernel: May 17 21:19:42 client-5 kernel: acceptor_7988 S ffff810289b09f18 0 28903 1 28895 28902 (L-TLB) May 17 21:19:42 client-5 kernel: ffff810289b09e30 0000000000000046 00000000000000d0 ffffffff80064b29 May 17 21:19:42 client-5 kernel: ffff810306b2d940 0000000000000001 ffff81033c546080 ffff81033fb790c0 May 17 21:19:42 client-5 kernel: 00003ffdeafe364a 0000000000006ac3 ffff81033c546268 00000000803199d0 May 17 21:19:42 client-5 kernel: Call Trace: May 17 21:19:42 client-5 kernel: [] _spin_lock_bh+0x9/0x14 May 17 21:19:42 client-5 kernel: [] new_inode+0x4e/0x7f May 17 21:19:42 client-5 kernel: [] :libcfs:libcfs_sock_accept+0xf9/0x170 May 17 21:19:42 client-5 kernel: [] __wake_up_common+0x3e/0x68 May 17 21:19:42 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:42 client-5 kernel: [] :lnet:lnet_acceptor+0x208/0x4a0 May 17 21:19:42 client-5 kernel: [] audit_syscall_exit+0x336/0x362 May 17 21:19:42 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:19:42 client-5 kernel: [] :lnet:lnet_acceptor+0x0/0x4a0 May 17 21:19:42 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:19:42 client-5 kernel: May 17 21:19:42 client-5 kernel: ptlrpcd-brw S ffff810001004420 0 28904 1 28905 28897 (L-TLB) May 17 21:19:42 client-5 kernel: ffff8102c2907e00 0000000000000046 0004dd3406d007bc 000000000000039b May 17 21:19:42 client-5 kernel: 000000004dd340d7 000000000000000a ffff81030318b820 ffffffff80308b60 May 17 21:19:42 client-5 kernel: 000041f39fbb4cd3 0000000000001198 ffff81030318ba08 0000000017269768 May 17 21:19:42 client-5 kernel: Call Trace: May 17 21:19:42 client-5 kernel: [] schedule_timeout+0x8a/0xad May 17 21:19:42 client-5 kernel: [] process_timeout+0x0/0x5 May 17 21:19:42 client-5 kernel: [] :ptlrpc:ptlrpcd+0x294/0x382 May 17 21:19:42 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:42 client-5 kernel: [] audit_syscall_exit+0x336/0x362 May 17 21:19:42 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:19:42 client-5 kernel: [] :ptlrpc:ptlrpcd+0x0/0x382 May 17 21:19:42 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:19:42 client-5 kernel: May 17 21:19:42 client-5 kernel: ptlrpcd-brw-r S ffff810001004420 0 28905 1 28906 28904 (L-TLB) May 17 21:19:42 client-5 kernel: ffff81032790fe00 0000000000000046 0000000000000086 ffffffff80046edb May 17 21:19:42 client-5 kernel: 00000000ffffffff 000000000000000a ffff81033c327080 ffffffff80308b60 May 17 21:19:42 client-5 kernel: 000041f386fa27ad 0000000000000bfe ffff81033c327268 00000000887524b8 May 17 21:19:42 client-5 kernel: Call Trace: May 17 21:19:42 client-5 kernel: [] try_to_wake_up+0x472/0x484 May 17 21:19:42 client-5 kernel: [] schedule_timeout+0x8a/0xad May 17 21:19:42 client-5 kernel: [] process_timeout+0x0/0x5 May 17 21:19:42 client-5 kernel: [] :ptlrpc:ptlrpcd+0x294/0x382 May 17 21:19:42 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:42 client-5 kernel: [] audit_syscall_exit+0x336/0x362 May 17 21:19:42 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:19:42 client-5 kernel: [] :ptlrpc:ptlrpcd+0x0/0x382 May 17 21:19:42 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:19:42 client-5 kernel: May 17 21:19:42 client-5 kernel: ptlrpcd S ffff810001004420 0 28906 1 28907 28905 (L-TLB) May 17 21:19:42 client-5 kernel: ffff8102e5593e00 0000000000000046 ffff810291ba8338 00000000000070ea May 17 21:19:42 client-5 kernel: ffff8102f2a3f8d8 000000000000000a ffff810136568080 ffffffff80308b60 May 17 21:19:42 client-5 kernel: 000041f39c56e746 0000000000000fd4 ffff810136568268 0000000000000000 May 17 21:19:42 client-5 kernel: Call Trace: May 17 21:19:42 client-5 kernel: [] schedule_timeout+0x8a/0xad May 17 21:19:42 client-5 kernel: [] process_timeout+0x0/0x5 May 17 21:19:42 client-5 kernel: [] :ptlrpc:ptlrpcd+0x294/0x382 May 17 21:19:42 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:42 client-5 kernel: [] audit_syscall_exit+0x336/0x362 May 17 21:19:42 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:19:42 client-5 kernel: [] :ptlrpc:ptlrpcd+0x0/0x382 May 17 21:19:42 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:19:42 client-5 kernel: May 17 21:19:42 client-5 kernel: ptlrpcd-rcv S ffff81000101d7a0 0 28907 1 28908 28906 (L-TLB) May 17 21:19:42 client-5 kernel: ffff8102d563fe00 0000000000000046 ffff8102ff3fc238 ffffffff000070eb May 17 21:19:42 client-5 kernel: ffff81032faa2cd8 000000000000000a ffff810327b88860 ffff81010b770100 May 17 21:19:42 client-5 kernel: 000041f05ed6e7a0 0000000000001101 ffff810327b88a48 0000000300000000 May 17 21:19:42 client-5 kernel: Call Trace: May 17 21:19:42 client-5 kernel: [] :obdclass:cl_key_exit+0x0/0xe0 May 17 21:19:42 client-5 kernel: [] schedule_timeout+0x8a/0xad May 17 21:19:42 client-5 kernel: [] process_timeout+0x0/0x5 May 17 21:19:42 client-5 kernel: [] :ptlrpc:ptlrpcd+0x294/0x382 May 17 21:19:42 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:42 client-5 kernel: [] audit_syscall_exit+0x336/0x362 May 17 21:19:42 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:19:42 client-5 kernel: [] :ptlrpc:ptlrpcd+0x0/0x382 May 17 21:19:42 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:19:42 client-5 kernel: May 17 21:19:42 client-5 kernel: ll_ping S ffff810001015120 0 28908 1 28909 28907 (L-TLB) May 17 21:19:42 client-5 kernel: ffff81033fae5e10 0000000000000046 0000000000000001 0000000000000009 May 17 21:19:42 client-5 kernel: 0000000000000286 000000000000000a ffff81033fbc00c0 ffff81010b764080 May 17 21:19:42 client-5 kernel: 000041f2ae522e21 0000000000007e9c ffff81033fbc02a8 0000000200000190 May 17 21:19:42 client-5 kernel: Call Trace: May 17 21:19:42 client-5 kernel: [] :ptlrpc:lustre_msg_size+0x11a/0x130 May 17 21:19:42 client-5 kernel: [] schedule_timeout+0x8a/0xad May 17 21:19:42 client-5 kernel: [] process_timeout+0x0/0x5 May 17 21:19:42 client-5 kernel: [] :ptlrpc:ptlrpc_pinger_main+0x68a/0x820 May 17 21:19:42 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:42 client-5 kernel: [] audit_syscall_exit+0x336/0x362 May 17 21:19:42 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:19:42 client-5 kernel: [] :ptlrpc:ptlrpc_pinger_main+0x0/0x820 May 17 21:19:42 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:19:42 client-5 kernel: May 17 21:19:42 client-5 kernel: sptlrpc_gc S ffff810001004420 0 28909 1 29054 28908 (L-TLB) May 17 21:19:42 client-5 kernel: ffff81030f423e60 0000000000000046 000000000f423dc0 0000000000000003 May 17 21:19:42 client-5 kernel: 00000000ffffffff 000000000000000a ffff81033c7d9820 ffffffff80308b60 May 17 21:19:42 client-5 kernel: 000041a050353016 0000000000000d8c ffff81033c7d9a08 0000000000000000 May 17 21:19:42 client-5 kernel: Call Trace: May 17 21:19:42 client-5 kernel: [] schedule_timeout+0x8a/0xad May 17 21:19:42 client-5 kernel: [] process_timeout+0x0/0x5 May 17 21:19:42 client-5 kernel: [] :ptlrpc:sec_gc_main+0x2cf/0x360 May 17 21:19:42 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:42 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:19:42 client-5 kernel: [] :ptlrpc:sec_gc_main+0x0/0x360 May 17 21:19:42 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:19:42 client-5 kernel: May 17 21:19:42 client-5 kernel: ll_capa S 00002b1c55064010 0 29054 1 30445 28909 (L-TLB) May 17 21:19:42 client-5 kernel: ffff81031a459e30 0000000000000046 0000000000000000 ffff81031d0a3e98 May 17 21:19:42 client-5 kernel: 0000000000000001 0000000000000001 ffff81025b211040 ffff81031977e7e0 May 17 21:19:42 client-5 kernel: 00003ffdfacacf98 0000000000001998 ffff81025b211228 000000028008b3c7 May 17 21:19:42 client-5 kernel: Call Trace: May 17 21:19:42 client-5 kernel: [] __wake_up+0x38/0x4f May 17 21:19:42 client-5 kernel: [] :lustre:capa_thread_main+0x4d2/0x15a0 May 17 21:19:42 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:42 client-5 kernel: [] audit_syscall_exit+0x336/0x362 May 17 21:19:42 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:19:42 client-5 kernel: [] :lustre:capa_thread_main+0x0/0x15a0 May 17 21:19:42 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:19:42 client-5 kernel: May 17 21:19:42 client-5 kernel: bash S 0000000000000001 0 29246 3677 31816 (NOTLB) May 17 21:19:42 client-5 kernel: ffff81032e5c7ee8 0000000000000082 0000000000000292 ffffffff80022205 May 17 21:19:42 client-5 kernel: ffff81033b5f8040 0000000000000001 ffff810321d6c0c0 ffff81017de92080 May 17 21:19:42 client-5 kernel: 0000401193a6c079 000000000000a280 ffff810321d6c2a8 0000000321d6c0c0 May 17 21:19:42 client-5 kernel: Call Trace: May 17 21:19:42 client-5 kernel: [] __up_read+0x19/0x7f May 17 21:19:42 client-5 kernel: [] do_wait+0xa03/0xad7 May 17 21:19:42 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:42 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:19:42 client-5 kernel: May 17 21:19:42 client-5 kernel: ldlm_bl_00 S ffff810001015120 0 30445 1 30446 29054 (L-TLB) May 17 21:19:42 client-5 kernel: ffff8102f4023ea0 0000000000000046 00000000ffffffff ffff81031585ab40 May 17 21:19:42 client-5 kernel: ffffc200112dd000 000000000000000a ffff81031977e7e0 ffff81010b764080 May 17 21:19:42 client-5 kernel: 0000400fd78c90c8 0000000000007a63 ffff81031977e9c8 0000000200000000 May 17 21:19:42 client-5 kernel: Call Trace: May 17 21:19:42 client-5 kernel: [] :ptlrpc:ldlm_bl_thread_main+0x18e/0x420 May 17 21:19:42 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:42 client-5 kernel: [] audit_syscall_exit+0x336/0x362 May 17 21:19:42 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:19:42 client-5 kernel: [] :ptlrpc:ldlm_bl_thread_main+0x0/0x420 May 17 21:19:42 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:19:42 client-5 kernel: May 17 21:19:42 client-5 kernel: ldlm_bl_01 S ffff810001004420 0 30446 1 30447 30445 (L-TLB) May 17 21:19:42 client-5 kernel: ffff81026a87dea0 0000000000000046 ffff81031585a900 0000000000000286 May 17 21:19:42 client-5 kernel: ffff81031585a900 0000000000000009 ffff81030318b0c0 ffffffff80308b60 May 17 21:19:42 client-5 kernel: 000040156e73dea6 000000000000b1de ffff81030318b2a8 0000000000000000 May 17 21:19:42 client-5 kernel: Call Trace: May 17 21:19:42 client-5 kernel: [] __wake_up_common+0x3e/0x68 May 17 21:19:42 client-5 kernel: [] :ptlrpc:ldlm_bl_thread_main+0x18e/0x420 May 17 21:19:42 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:42 client-5 kernel: [] audit_syscall_exit+0x336/0x362 May 17 21:19:42 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:19:42 client-5 kernel: [] :ptlrpc:ldlm_bl_thread_main+0x0/0x420 May 17 21:19:42 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:19:42 client-5 kernel: May 17 21:19:42 client-5 kernel: ldlm_cn_00 S ffff810331e7f860 0 30447 1 30448 30446 (L-TLB) May 17 21:19:42 client-5 kernel: ffff810319fc7df0 0000000000000046 ffffffff88784a68 0000000000000000 May 17 21:19:42 client-5 kernel: ffff810319fc7d80 0000000000000001 ffff81033a3bf100 ffff810331e7f860 May 17 21:19:42 client-5 kernel: 0000400cf73e6235 0000000000000b54 ffff81033a3bf2e8 0000000200000003 May 17 21:19:42 client-5 kernel: Call Trace: May 17 21:19:42 client-5 kernel: [] __wake_up+0x38/0x4f May 17 21:19:42 client-5 kernel: [] _spin_lock_bh+0x9/0x14 May 17 21:19:43 client-5 kernel: [] :libcfs:lc_watchdog_del_pending+0x16/0x70 May 17 21:19:43 client-5 kernel: [] :ptlrpc:ptlrpc_wait_event+0x1b9/0x3e0 May 17 21:19:43 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:43 client-5 kernel: [] __wake_up+0x38/0x4f May 17 21:19:43 client-5 kernel: [] :ptlrpc:ptlrpc_main+0x150c/0x1560 May 17 21:19:43 client-5 kernel: [] audit_syscall_exit+0x336/0x362 May 17 21:19:43 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:19:43 client-5 kernel: [] :ptlrpc:ptlrpc_main+0x0/0x1560 May 17 21:19:43 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:19:43 client-5 kernel: May 17 21:19:43 client-5 kernel: lc_watchdogd S ffff8103048011c8 0 30448 1 30449 30447 (L-TLB) May 17 21:19:43 client-5 kernel: ffff810302a7de60 0000000000000046 0000000000000000 0000000000000000 May 17 21:19:43 client-5 kernel: ffff810304801380 0000000000000001 ffff8103024350c0 ffff8101365687e0 May 17 21:19:43 client-5 kernel: 0000400cf73e33d7 0000000000000cf7 ffff8103024352a8 0000000200000000 May 17 21:19:43 client-5 kernel: Call Trace: May 17 21:19:43 client-5 kernel: [] __wake_up_common+0x3e/0x68 May 17 21:19:43 client-5 kernel: [] :libcfs:lcw_dispatch_main+0x166/0x890 May 17 21:19:43 client-5 kernel: [] autoremove_wake_function+0x0/0x2e May 17 21:19:43 client-5 kernel: [] audit_syscall_exit+0x336/0x362 May 17 21:19:43 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:19:43 client-5 kernel: [] :libcfs:lcw_dispatch_main+0x0/0x890 May 17 21:19:43 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:19:43 client-5 kernel: May 17 21:19:43 client-5 kernel: ldlm_cn_01 S 0000000000000000 0 30449 1 30450 30448 (L-TLB) May 17 21:19:43 client-5 kernel: ffff8103155eddf0 0000000000000046 0000000300000000 ffffffff88784a60 May 17 21:19:43 client-5 kernel: 0000000000000246 0000000000000001 ffff8101365687e0 ffff81033b5910c0 May 17 21:19:43 client-5 kernel: 0000400cf73e88ee 00000000000003ff ffff8101365689c8 00000002365687e0 May 17 21:19:43 client-5 kernel: Call Trace: May 17 21:19:43 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:43 client-5 kernel: [] _spin_lock_bh+0x9/0x14 May 17 21:19:43 client-5 kernel: [] :libcfs:lc_watchdog_del_pending+0x16/0x70 May 17 21:19:43 client-5 kernel: [] :ptlrpc:ptlrpc_wait_event+0x1b9/0x3e0 May 17 21:19:43 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:43 client-5 kernel: [] __wake_up+0x38/0x4f May 17 21:19:43 client-5 kernel: [] :ptlrpc:ptlrpc_main+0x150c/0x1560 May 17 21:19:43 client-5 kernel: [] audit_syscall_exit+0x336/0x362 May 17 21:19:43 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:19:43 client-5 kernel: [] :ptlrpc:ptlrpc_main+0x0/0x1560 May 17 21:19:43 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:19:43 client-5 kernel: May 17 21:19:43 client-5 kernel: ldlm_cb_00 S ffff810001015120 0 30450 1 30451 30449 (L-TLB) May 17 21:19:43 client-5 kernel: ffff8102e368fdf0 0000000000000046 0000400cf73ea378 0000000000000002 May 17 21:19:43 client-5 kernel: ffff810331e7f860 000000000000000a ffff81033b5910c0 ffff81010b764080 May 17 21:19:43 client-5 kernel: 0000400ff3022e9b 00000000000004de ffff81033b5912a8 0000000200000001 May 17 21:19:43 client-5 kernel: Call Trace: May 17 21:19:43 client-5 kernel: [] :ptlrpc:ptlrpc_wait_event+0x1b9/0x3e0 May 17 21:19:43 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:43 client-5 kernel: [] __wake_up+0x38/0x4f May 17 21:19:43 client-5 kernel: [] :ptlrpc:ptlrpc_main+0x150c/0x1560 May 17 21:19:43 client-5 kernel: [] audit_syscall_exit+0x336/0x362 May 17 21:19:43 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:19:43 client-5 kernel: [] :ptlrpc:ptlrpc_main+0x0/0x1560 May 17 21:19:43 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:19:43 client-5 kernel: May 17 21:19:43 client-5 kernel: ldlm_cb_01 S ffff810001015120 0 30451 1 30452 30450 (L-TLB) May 17 21:19:43 client-5 kernel: ffff810313a2bdf0 0000000000000046 0000000000000000 0000000000000000 May 17 21:19:43 client-5 kernel: ffffffff8898b478 000000000000000a ffff81025b2117a0 ffff81010b764080 May 17 21:19:43 client-5 kernel: 0000401158087ed0 0000000000001a03 ffff81025b211988 00000002d94c5648 May 17 21:19:43 client-5 kernel: Call Trace: May 17 21:19:43 client-5 kernel: [] __mod_timer+0x100/0x10f May 17 21:19:43 client-5 kernel: [] :ptlrpc:ptlrpc_wait_event+0x1b9/0x3e0 May 17 21:19:43 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:43 client-5 kernel: [] :ptlrpc:ptlrpc_main+0x150c/0x1560 May 17 21:19:43 client-5 kernel: [] audit_syscall_exit+0x336/0x362 May 17 21:19:43 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:19:43 client-5 kernel: [] :ptlrpc:ptlrpc_main+0x0/0x1560 May 17 21:19:43 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:19:43 client-5 kernel: May 17 21:19:43 client-5 kernel: ldlm_elt S ffff8102c4623ef0 0 30452 1 30453 30451 (L-TLB) May 17 21:19:43 client-5 kernel: ffff8102c4623eb0 0000000000000046 0000000000000500 ffff81013e897588 May 17 21:19:43 client-5 kernel: 0000000000000001 0000000000000001 ffff8102c8efa080 ffff810331e7f860 May 17 21:19:43 client-5 kernel: 0000400cf73f09f8 0000000000000c25 ffff8102c8efa268 000000028008b3c7 May 17 21:19:43 client-5 kernel: Call Trace: May 17 21:19:43 client-5 kernel: [] __wake_up+0x38/0x4f May 17 21:19:43 client-5 kernel: [] :ptlrpc:expired_lock_main+0x311/0x8b0 May 17 21:19:43 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:43 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:19:43 client-5 kernel: [] :ptlrpc:expired_lock_main+0x0/0x8b0 May 17 21:19:43 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:19:43 client-5 kernel: May 17 21:19:43 client-5 kernel: ldlm_poold S ffff81000101d7a0 0 30453 1 30456 30452 (L-TLB) May 17 21:19:43 client-5 kernel: ffff81031ffe7e60 0000000000000046 ffffffff889f9d20 0000000000000002 May 17 21:19:43 client-5 kernel: 0000000000000282 000000000000000a ffff8102d9408080 ffff81010b770100 May 17 21:19:43 client-5 kernel: 000041f0366f72c1 0000000000001ca3 ffff8102d9408268 000000030b7e4068 May 17 21:19:43 client-5 kernel: Call Trace: May 17 21:19:43 client-5 kernel: [] schedule_timeout+0x8a/0xad May 17 21:19:43 client-5 kernel: [] process_timeout+0x0/0x5 May 17 21:19:43 client-5 kernel: [] :ptlrpc:ldlm_pools_thread_main+0x1c4/0x2b3 May 17 21:19:43 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:43 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:19:43 client-5 kernel: [] :ptlrpc:ldlm_pools_thread_main+0x0/0x2b3 May 17 21:19:43 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:19:43 client-5 kernel: May 17 21:19:43 client-5 kernel: ll_close S ffff81031a0a93f0 0 30456 1 31115 30453 (L-TLB) May 17 21:19:43 client-5 kernel: ffff810173109e80 0000000000000046 ffffffff88c483d2 ffff810173109db0 May 17 21:19:43 client-5 kernel: 00000000ffffffff 0000000000000006 ffff810318e3b820 ffff810331e7f860 May 17 21:19:43 client-5 kernel: 0000400e24274c37 0000000000001de2 ffff810318e3ba08 0000000200000000 May 17 21:19:43 client-5 kernel: Call Trace: May 17 21:19:43 client-5 kernel: [] __wake_up_common+0x3e/0x68 May 17 21:19:43 client-5 kernel: [] :lustre:ll_close_thread+0x169/0xfb0 May 17 21:19:43 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:43 client-5 kernel: [] audit_syscall_exit+0x336/0x362 May 17 21:19:43 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:19:43 client-5 kernel: [] :lustre:ll_close_thread+0x0/0xfb0 May 17 21:19:43 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:19:43 client-5 kernel: May 17 21:19:43 client-5 kernel: ldlm_bl_02 S ffff810001015120 0 31115 1 413 30456 (L-TLB) May 17 21:19:43 client-5 kernel: ffff8102c2b97ea0 0000000000000046 00000000ffffffff ffff8103229826c0 May 17 21:19:43 client-5 kernel: ffffc200112dd000 0000000000000009 ffff81033fb6f820 ffff81010b764080 May 17 21:19:43 client-5 kernel: 0000400fd78a40c7 0000000000006f3a ffff81033fb6fa08 0000000200000000 May 17 21:19:43 client-5 kernel: Call Trace: May 17 21:19:43 client-5 kernel: [] :ptlrpc:ldlm_bl_thread_main+0x18e/0x420 May 17 21:19:43 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:43 client-5 kernel: [] audit_syscall_exit+0x336/0x362 May 17 21:19:43 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:19:43 client-5 kernel: [] :ptlrpc:ldlm_bl_thread_main+0x0/0x420 May 17 21:19:43 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:19:43 client-5 kernel: May 17 21:19:43 client-5 kernel: bash S ffff810001004420 0 31816 29246 32745 31817 (NOTLB) May 17 21:19:43 client-5 kernel: ffff8102ff5f1ee8 0000000000000082 0000000000000292 ffffffff80022205 May 17 21:19:43 client-5 kernel: ffff8102f2791b80 0000000000000002 ffff81033b657860 ffffffff80308b60 May 17 21:19:43 client-5 kernel: 0000401690fde012 000000000000da35 ffff81033b657a48 000000003b657860 May 17 21:19:43 client-5 kernel: Call Trace: May 17 21:19:43 client-5 kernel: [] __up_read+0x19/0x7f May 17 21:19:43 client-5 kernel: [] do_wait+0xa03/0xad7 May 17 21:19:43 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:43 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:19:43 client-5 kernel: May 17 21:19:43 client-5 kernel: tee S ffff81000101d7a0 0 31817 29246 31816 (NOTLB) May 17 21:19:43 client-5 kernel: ffff81026c17fdf8 0000000000000086 0000000000000000 ffff81031ec2e100 May 17 21:19:43 client-5 kernel: ffffffff8008c937 0000000000000007 ffff81017de92080 ffff81010b770100 May 17 21:19:43 client-5 kernel: 0000401690fd9f42 00000000000064d3 ffff81017de92268 0000000311c6ae40 May 17 21:19:43 client-5 kernel: Call Trace: May 17 21:19:43 client-5 kernel: [] enqueue_task+0x41/0x56 May 17 21:19:43 client-5 kernel: [] generic_file_aio_write+0x70/0xc1 May 17 21:19:43 client-5 kernel: [] pipe_wait+0x69/0x93 May 17 21:19:43 client-5 kernel: [] autoremove_wake_function+0x0/0x2e May 17 21:19:43 client-5 kernel: [] mutex_lock+0xd/0x1d May 17 21:19:43 client-5 kernel: [] pipe_readv+0x324/0x3a9 May 17 21:19:43 client-5 kernel: [] pipe_read+0x1a/0x1f May 17 21:19:43 client-5 kernel: [] vfs_read+0xcb/0x171 May 17 21:19:43 client-5 kernel: [] sys_read+0x45/0x6e May 17 21:19:43 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:19:43 client-5 kernel: May 17 21:19:43 client-5 kernel: dd S ffff81000100caa0 0 32745 31816 (NOTLB) May 17 21:19:43 client-5 kernel: ffff8102fb8df558 0000000000000086 ffff810319f1c6c0 ffff8102ff3fc6a8 May 17 21:19:43 client-5 kernel: ffff810319f1c710 0000000000000001 ffff8101f9b0e040 ffff81010b70e100 May 17 21:19:43 client-5 kernel: 000040169dd977a9 0000000000200a54 ffff8101f9b0e228 0000000188b0dc18 May 17 21:19:43 client-5 kernel: Call Trace: May 17 21:19:43 client-5 kernel: [] :osc:osc_queue_async_io+0xa23/0xed0 May 17 21:19:43 client-5 kernel: [] radix_tree_insert+0xb3/0x190 May 17 21:19:43 client-5 kernel: [] :libcfs:libcfs_debug_vmsg2+0x75b/0x9f0 May 17 21:19:43 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:43 client-5 kernel: [] :osc:osc_page_cache_add+0x120/0x1d0 May 17 21:19:43 client-5 kernel: [] :obdclass:cl_page_invoke+0xb8/0x170 May 17 21:19:43 client-5 kernel: [] radix_tree_node_alloc+0x18/0x57 May 17 21:19:43 client-5 kernel: [] :obdclass:cl_page_cache_add+0x90/0x1d0 May 17 21:19:43 client-5 kernel: [] :lustre:vvp_write_pending+0x8c/0x130 May 17 21:19:43 client-5 kernel: [] :lustre:vvp_io_commit_write+0x26f/0x500 May 17 21:19:43 client-5 kernel: [] :lov:lov_io_prepare_write+0x12c/0x1c0 May 17 21:19:43 client-5 kernel: [] :obdclass:cl_page_at_trusted+0x67/0x1c0 May 17 21:19:43 client-5 kernel: [] :obdclass:cl_io_commit_write+0x125/0x1c0 May 17 21:19:43 client-5 kernel: [] :obdclass:cl_env_get+0x25/0x300 May 17 21:19:43 client-5 kernel: [] :lustre:ll_commit_write+0x1b4/0x280 May 17 21:19:43 client-5 kernel: [] :lustre:ll_prepare_write+0xfb/0x1b0 May 17 21:19:43 client-5 kernel: [] generic_file_buffered_write+0x4df/0x675 May 17 21:19:43 client-5 kernel: [] __generic_file_aio_write_nolock+0x369/0x3b6 May 17 21:19:43 client-5 kernel: [] __generic_file_write_nolock+0x8f/0xa8 May 17 21:19:43 client-5 kernel: [] :obdclass:cl_lock_trace0+0x13e/0x180 May 17 21:19:43 client-5 kernel: [] autoremove_wake_function+0x0/0x2e May 17 21:19:43 client-5 kernel: [] mutex_lock+0xd/0x1d May 17 21:19:43 client-5 kernel: [] generic_file_writev+0x48/0xa3 May 17 21:19:43 client-5 kernel: [] :lustre:vvp_io_write_start+0x157/0x240 May 17 21:19:43 client-5 kernel: [] :obdclass:cl_io_start+0xbc/0x160 May 17 21:19:43 client-5 kernel: [] :obdclass:cl_io_loop+0xad/0x1a0 May 17 21:19:43 client-5 kernel: [] :lustre:ll_file_io_generic+0x459/0x4a0 May 17 21:19:43 client-5 kernel: [] thread_return+0x62/0xfe May 17 21:19:43 client-5 kernel: [] :obdclass:cl_env_get+0x25/0x300 May 17 21:19:43 client-5 kernel: [] :lustre:ll_file_writev+0x1e0/0x280 May 17 21:19:43 client-5 kernel: [] :obdclass:cl_env_get+0x162/0x300 May 17 21:19:43 client-5 kernel: [] :obdclass:cl_env_put+0x279/0x2d0 May 17 21:19:43 client-5 kernel: [] :lustre:ll_file_write+0x164/0x210 May 17 21:19:43 client-5 kernel: [] read_zero+0x1cd/0x226 May 17 21:19:43 client-5 kernel: [] vfs_write+0xce/0x174 May 17 21:19:43 client-5 kernel: [] sys_write+0x45/0x6e May 17 21:19:43 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:19:43 client-5 kernel: May 17 21:19:43 client-5 kernel: lockd S ffff81000100caa0 0 413 1 31115 (L-TLB) May 17 21:19:43 client-5 kernel: ffff8102f49b1e60 0000000000000046 0000000000000246 ffffffff8004765b May 17 21:19:43 client-5 kernel: 0000000000002cc0 0000000000000005 ffff81033b5957a0 ffff81010b70e100 May 17 21:19:43 client-5 kernel: 000041db426cfcf6 0000000000002534 ffff81033b595988 0000000100000286 May 17 21:19:43 client-5 kernel: Call Trace: May 17 21:19:43 client-5 kernel: [] skb_dequeue+0x48/0x50 May 17 21:19:43 client-5 kernel: [] schedule_timeout+0x1e/0xad May 17 21:19:43 client-5 kernel: [] _spin_lock_bh+0x9/0x14 May 17 21:19:43 client-5 kernel: [] add_wait_queue+0x24/0x34 May 17 21:19:43 client-5 kernel: [] :sunrpc:svc_recv+0x303/0x495 May 17 21:19:43 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:19:43 client-5 kernel: [] __mod_timer+0x100/0x10f May 17 21:19:43 client-5 kernel: [] :lockd:lockd+0x0/0x2bf May 17 21:19:43 client-5 kernel: [] :lockd:lockd+0x18e/0x2bf May 17 21:19:43 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:19:43 client-5 kernel: [] :lockd:lockd+0x0/0x2bf May 17 21:19:43 client-5 kernel: [] :lockd:lockd+0x0/0x2bf May 17 21:19:43 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:19:43 client-5 kernel: May 17 21:21:22 client-5 kernel: Lustre: 28904:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369115429177275 sent from lustre-OST0001-osc-ffff81031957b000 to NID 192.168.4.129@o2ib has timed out for slow reply: [sent 1305691255] [real_sent 1305691255] [current 1305692482] [deadline 1227s] [delay 0s] req@ffff8102eb50ec00 x1369115429177275/t0(0) o-1->lustre-OST0001_UUID@192.168.4.129@o2ib:6/4 lens 456/416 e 6 to 1 dl 1305692482 ref 2 fl Rpc:X/ffffffff/ffffffff rc 0/-1 May 17 21:21:22 client-5 kernel: Lustre: lustre-OST0001-osc-ffff81031957b000: Connection to service lustre-OST0001 via nid 192.168.4.129@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 21:21:22 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.129@o2ib. The ost_connect operation failed with -16 May 17 21:21:27 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.129@o2ib. The ost_connect operation failed with -16 May 17 21:21:32 client-5 kernel: Lustre: lustre-OST0001-osc-ffff81031957b000: Connection restored to service lustre-OST0001 using nid 192.168.4.129@o2ib. May 17 21:29:02 client-5 kernel: SysRq : Show State May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: sibling May 17 21:29:02 client-5 kernel: task PC pid father child younger older May 17 21:29:02 client-5 kernel: init S ffff810001015120 0 1 0 2 (NOTLB) May 17 21:29:02 client-5 kernel: ffff81010b717a18 0000000000000086 ffff81033aae3500 ffff81033aae3500 May 17 21:29:02 client-5 kernel: 0000000010008040 000000000000000a ffff81010b6fd7a0 ffff81010b764080 May 17 21:29:02 client-5 kernel: 00004275d3fedc3e 00000000000035d8 ffff81010b6fd988 000000020000000a May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] schedule_timeout+0x8a/0xad May 17 21:29:02 client-5 kernel: [] process_timeout+0x0/0x5 May 17 21:29:02 client-5 kernel: [] do_select+0x3fe/0x462 May 17 21:29:02 client-5 kernel: [] __pollwait+0x0/0xe2 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] __alloc_pages+0x78/0x308 May 17 21:29:02 client-5 kernel: [] dummy_inode_permission+0x0/0x3 May 17 21:29:02 client-5 kernel: [] zone_statistics+0x3e/0x6d May 17 21:29:02 client-5 kernel: [] __d_lookup+0xb0/0xff May 17 21:29:02 client-5 kernel: [] do_lookup+0x65/0x1e6 May 17 21:29:02 client-5 kernel: [] dput+0x2c/0x114 May 17 21:29:02 client-5 kernel: [] __link_path_walk+0xf32/0xf5b May 17 21:29:02 client-5 kernel: [] core_sys_select+0x1bc/0x265 May 17 21:29:02 client-5 kernel: [] mntput_no_expire+0x19/0x89 May 17 21:29:02 client-5 kernel: [] cp_new_stat+0xe5/0xfd May 17 21:29:02 client-5 kernel: [] sys_select+0x153/0x17c May 17 21:29:02 client-5 kernel: [] system_call+0x7e/0x83 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: migration/0 S ffff810001004420 0 2 1 3 (L-TLB) May 17 21:29:02 client-5 kernel: ffff81010b71bea0 0000000000000046 ffff81010b71bea0 ffffffff800767ae May 17 21:29:02 client-5 kernel: 0000000000000008 0000000000000001 ffff81010b6fd040 ffffffff80308b60 May 17 21:29:02 client-5 kernel: 0000420b94285e0e 000000000000052f ffff81010b6fd228 0000000000000000 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] smp_send_reschedule+0x4e/0x53 May 17 21:29:02 client-5 kernel: [] migration_thread+0x1a2/0x23a May 17 21:29:02 client-5 kernel: [] migration_thread+0x0/0x23a May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: ksoftirqd/0 S ffff810001004420 0 3 1 4 2 (L-TLB) May 17 21:29:02 client-5 kernel: ffff81010b71fed0 0000000000000046 ffff81010b6ff7e0 ffff8102e9a12040 May 17 21:29:02 client-5 kernel: 00003ffdfd943379 000000000000000a ffff81010b6ff7e0 ffffffff80308b60 May 17 21:29:02 client-5 kernel: 0000426cb1cce35b 0000000000005242 ffff81010b6ff9c8 0000000080bc7000 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] ksoftirqd+0x0/0xbf May 17 21:29:02 client-5 kernel: [] ksoftirqd+0x3b/0xbf May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: watchdog/0 S ffff810001004420 0 4 1 5 3 (L-TLB) May 17 21:29:02 client-5 kernel: ffff81010b721ec0 0000000000000046 ffff810001004498 ffff81010b721e30 May 17 21:29:02 client-5 kernel: ffff81010b6ff080 0000000000000001 ffff81010b6ff080 ffffffff80308b60 May 17 21:29:02 client-5 kernel: 000042763e72d228 0000000000000382 ffff81010b6ff268 00000000800a4d70 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] watchdog+0x0/0x5f May 17 21:29:02 client-5 kernel: [] watchdog+0x4f/0x5f May 17 21:29:02 client-5 kernel: [] watchdog+0x0/0x5f May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: migration/1 S ffff81010b7000c0 0 5 1 6 4 (L-TLB) May 17 21:29:02 client-5 kernel: ffff81010b723ea0 0000000000000046 0000000000000001 0000000000000000 May 17 21:29:02 client-5 kernel: 0000000000000000 0000000000000001 ffff81010b700820 ffff81010b7000c0 May 17 21:29:02 client-5 kernel: 00004265bb9f5a3b 0000000000000ade ffff81010b700a08 0000000100000001 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] migration_thread+0x1a2/0x23a May 17 21:29:02 client-5 kernel: [] migration_thread+0x0/0x23a May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: ksoftirqd/1 S ffff81000100caa0 0 6 1 7 5 (L-TLB) May 17 21:29:02 client-5 kernel: ffff81010b727ed0 0000000000000046 ffff81010b7000c0 ffff8102c8efa7e0 May 17 21:29:02 client-5 kernel: 000041634bd2d7be 000000000000000a ffff81010b7000c0 ffff81010b70e100 May 17 21:29:02 client-5 kernel: 00004265bbfb1205 0000000000002b5d ffff81010b7002a8 000000018008c4fc May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] ksoftirqd+0x0/0xbf May 17 21:29:02 client-5 kernel: [] ksoftirqd+0x3b/0xbf May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: watchdog/1 S ffff81000100caa0 0 7 1 8 6 (L-TLB) May 17 21:29:02 client-5 kernel: ffff81010b72dec0 0000000000000046 ffff81000100cb18 ffff81010b72de30 May 17 21:29:02 client-5 kernel: ffff81010b70e860 0000000000000009 ffff81010b70e860 ffff81010b70e100 May 17 21:29:02 client-5 kernel: 000042762acffe6e 0000000000000481 ffff81010b70ea48 00000001800a4d70 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] watchdog+0x0/0x5f May 17 21:29:02 client-5 kernel: [] watchdog+0x4f/0x5f May 17 21:29:02 client-5 kernel: [] watchdog+0x0/0x5f May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: migration/2 S ffff810001015120 0 8 1 9 7 (L-TLB) May 17 21:29:02 client-5 kernel: ffff81010b759ea0 0000000000000046 ffff81010b759ea0 ffffffff800767ae May 17 21:29:02 client-5 kernel: 0000000000000001 0000000000000001 ffff81010b7117a0 ffff81010b764080 May 17 21:29:02 client-5 kernel: 0000420b94280c42 0000000000000691 ffff81010b711988 0000000200000000 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] smp_send_reschedule+0x4e/0x53 May 17 21:29:02 client-5 kernel: [] migration_thread+0x1a2/0x23a May 17 21:29:02 client-5 kernel: [] migration_thread+0x0/0x23a May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: ksoftirqd/2 S ffff810001015120 0 9 1 10 8 (L-TLB) May 17 21:29:02 client-5 kernel: ffff81010b77ded0 0000000000000046 ffff81010b711040 ffff810314d847a0 May 17 21:29:02 client-5 kernel: 00003ff091e0b57e 000000000000000a ffff81010b711040 ffff81010b764080 May 17 21:29:02 client-5 kernel: 000041ab6e283ccf 00000000000052bd ffff81010b711228 0000000200000000 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] ksoftirqd+0x0/0xbf May 17 21:29:02 client-5 kernel: [] ksoftirqd+0x3b/0xbf May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: watchdog/2 S ffff810001015120 0 10 1 11 9 (L-TLB) May 17 21:29:02 client-5 kernel: ffff81010b781ec0 0000000000000046 ffff810001015198 ffff81010b781e30 May 17 21:29:02 client-5 kernel: ffff81010b7647e0 0000000000000009 ffff81010b7647e0 ffff81010b764080 May 17 21:29:02 client-5 kernel: 0000427603a86c3e 000000000000047a ffff81010b7649c8 0000000200000000 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] watchdog+0x0/0x5f May 17 21:29:02 client-5 kernel: [] watchdog+0x4f/0x5f May 17 21:29:02 client-5 kernel: [] watchdog+0x0/0x5f May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: migration/3 S ffff81000101d7a0 0 11 1 12 10 (L-TLB) May 17 21:29:02 client-5 kernel: ffff81010b7adea0 0000000000000046 ffff81010b7adea0 ffffffff800767ae May 17 21:29:02 client-5 kernel: 0000000000000001 0000000000000001 ffff81010b766820 ffff81010b770100 May 17 21:29:02 client-5 kernel: 0000420b9428971c 00000000000006ef ffff81010b766a08 0000000300000000 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] smp_send_reschedule+0x4e/0x53 May 17 21:29:02 client-5 kernel: [] migration_thread+0x1a2/0x23a May 17 21:29:02 client-5 kernel: [] migration_thread+0x0/0x23a May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: ksoftirqd/3 S ffff81000101d7a0 0 12 1 13 11 (L-TLB) May 17 21:29:02 client-5 kernel: ffff81010b7afed0 0000000000000046 ffff81010b7660c0 ffff8102c3c54820 May 17 21:29:02 client-5 kernel: 00003ffdfdb20261 000000000000000a ffff81010b7660c0 ffff81010b770100 May 17 21:29:02 client-5 kernel: 00004230e4aefe1a 0000000000004045 ffff81010b7662a8 0000000300000220 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] ksoftirqd+0x0/0xbf May 17 21:29:02 client-5 kernel: [] ksoftirqd+0x3b/0xbf May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: watchdog/3 S ffff81000101d7a0 0 13 1 14 12 (L-TLB) May 17 21:29:02 client-5 kernel: ffff81010b7b7ec0 0000000000000046 ffff81000101d818 ffff81010b7b7e30 May 17 21:29:02 client-5 kernel: ffff81010b770860 0000000000000009 ffff81010b770860 ffff81010b770100 May 17 21:29:02 client-5 kernel: 000042765f88be90 0000000000000490 ffff81010b770a48 00000003800a4d70 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] watchdog+0x0/0x5f May 17 21:29:02 client-5 kernel: [] watchdog+0x4f/0x5f May 17 21:29:02 client-5 kernel: [] watchdog+0x0/0x5f May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: events/0 S ffff810001004420 0 14 1 15 13 (L-TLB) May 17 21:29:02 client-5 kernel: ffff81033fb3de70 0000000000000046 0000000000000000 ffff81010b64edc0 May 17 21:29:02 client-5 kernel: 0000000300000000 000000000000000a ffff81033fb2f7a0 ffffffff80308b60 May 17 21:29:02 client-5 kernel: 000042765c1544ea 00000000000049cd ffff81033fb2f988 000000008002e23f May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:29:02 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: events/1 S ffff81000100caa0 0 15 1 16 14 (L-TLB) May 17 21:29:02 client-5 kernel: ffff81033fb41e70 0000000000000046 0000000000000000 ffff81010b64edc0 May 17 21:29:02 client-5 kernel: 0000000300000000 000000000000000a ffff81033fb2f040 ffff81010b70e100 May 17 21:29:02 client-5 kernel: 0000427602b55582 0000000000003ee4 ffff81033fb2f228 000000018002e23f May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:29:02 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: events/2 S ffff810001015120 0 16 1 17 15 (L-TLB) May 17 21:29:02 client-5 kernel: ffff81033fb43e70 0000000000000046 0000000000000000 ffff81010b64e2c0 May 17 21:29:02 client-5 kernel: 0000000300000000 000000000000000a ffff81033fb307e0 ffff81010b764080 May 17 21:29:02 client-5 kernel: 000042766a7bfe8a 000000000000491e ffff81033fb309c8 000000028002e23f May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:29:02 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: events/3 S ffff81000101d7a0 0 17 1 18 16 (L-TLB) May 17 21:29:02 client-5 kernel: ffff81033fb47e70 0000000000000046 0000000000000000 ffff81010b64e3c0 May 17 21:29:02 client-5 kernel: 0000000300000000 000000000000000a ffff81033fb30080 ffff81010b770100 May 17 21:29:02 client-5 kernel: 000042763edde745 0000000000003a41 ffff81033fb30268 000000038002e23f May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:29:02 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: khelper S ffff81010b64cf60 0 18 1 123 17 (L-TLB) May 17 21:29:02 client-5 kernel: ffff81033fb4be70 0000000000000046 0000000000000000 ffff81033fb4a000 May 17 21:29:02 client-5 kernel: 0000000300000000 000000000000000a ffff81033fb32820 ffff8102c8efa080 May 17 21:29:02 client-5 kernel: 00003ffdf74a4ccc 0000000000001885 ffff81033fb32a08 000000008002e23f May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] __call_usermodehelper+0x0/0x4f May 17 21:29:02 client-5 kernel: [] run_workqueue+0xba/0xe4 May 17 21:29:02 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:29:02 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: kthread S ffff81033fbacea0 0 123 1 130 692 18 (L-TLB) May 17 21:29:02 client-5 kernel: ffff81033fcb9e70 0000000000000046 0000000000007188 ffff81030aa83d68 May 17 21:29:02 client-5 kernel: 0000000300000000 0000000000000003 ffff81033fc690c0 ffff81025b2117a0 May 17 21:29:02 client-5 kernel: 00003ffdfbb7c898 0000000000000712 ffff81033fc692a8 000000028002e23f May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] run_workqueue+0xba/0xe4 May 17 21:29:02 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:29:02 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: kblockd/0 S ffff810001004420 0 130 123 131 (L-TLB) May 17 21:29:02 client-5 kernel: ffff81033fb85e70 0000000000000046 ffffffff8014e000 ffffffff80143260 May 17 21:29:02 client-5 kernel: 0000000300000000 000000000000000a ffff81033fc09080 ffffffff80308b60 May 17 21:29:02 client-5 kernel: 0000426ddb1743f9 0000000000000497 ffff81033fc09268 000000008002e23f May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] cfq_kick_queue+0x0/0x89 May 17 21:29:02 client-5 kernel: [] elv_next_request+0x168/0x178 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:29:02 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] request_module+0x0/0x14d May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: kblockd/1 S ffff81000100caa0 0 131 123 132 130 (L-TLB) May 17 21:29:02 client-5 kernel: ffff81033fb8be70 0000000000000046 ffffffff8014e000 ffffffff80143260 May 17 21:29:02 client-5 kernel: 0000000300000000 000000000000000a ffff81033fc097e0 ffff81010b70e100 May 17 21:29:02 client-5 kernel: 00004266e3f62114 000000000000070c ffff81033fc099c8 000000018002e23f May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] cfq_kick_queue+0x0/0x89 May 17 21:29:02 client-5 kernel: [] elv_next_request+0x168/0x178 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:29:02 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] request_module+0x0/0x14d May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: kblockd/2 S ffff810001015120 0 132 123 133 131 (L-TLB) May 17 21:29:02 client-5 kernel: ffff81033fb8de70 0000000000000046 ffffffff8014e000 ffffffff80143260 May 17 21:29:02 client-5 kernel: 0000000300000000 000000000000000a ffff81033fc08040 ffff81010b764080 May 17 21:29:02 client-5 kernel: 000042767664d68e 0000000000000744 ffff81033fc08228 000000028002e23f May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] cfq_kick_queue+0x0/0x89 May 17 21:29:02 client-5 kernel: [] elv_next_request+0x168/0x178 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:29:02 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] request_module+0x0/0x14d May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: kblockd/3 S ffff81000101d7a0 0 133 123 134 132 (L-TLB) May 17 21:29:02 client-5 kernel: ffff81033fb8fe70 0000000000000046 ffffffff8014e000 ffffffff80143260 May 17 21:29:02 client-5 kernel: 0000000300000000 000000000000000a ffff81033fc087a0 ffff81010b770100 May 17 21:29:02 client-5 kernel: 0000425b6f036b3f 00000000000006f6 ffff81033fc08988 000000038002e23f May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] cfq_kick_queue+0x0/0x89 May 17 21:29:02 client-5 kernel: [] elv_next_request+0x168/0x178 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:29:02 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] request_module+0x0/0x14d May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: kacpid S ffff810001004420 0 134 123 295 133 (L-TLB) May 17 21:29:02 client-5 kernel: ffff81033fb93e70 0000000000000046 ffff81033fc39640 ffff81033fc39648 May 17 21:29:02 client-5 kernel: 0000000300000000 0000000000000006 ffff81033fc05100 ffffffff80308b60 May 17 21:29:02 client-5 kernel: 00000013e04021c5 00000000000003e0 ffff81033fc052e8 000000008002e23f May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:29:02 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] request_module+0x0/0x14d May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: cqueue/0 S ffff810001004420 0 295 123 296 134 (L-TLB) May 17 21:29:02 client-5 kernel: ffff81033fc8be70 0000000000000046 00000013e8ce21c0 0000000000000001 May 17 21:29:02 client-5 kernel: ffff81033fbbd7e0 0000000000000007 ffff81033fbbd7e0 ffffffff80308b60 May 17 21:29:02 client-5 kernel: 00000013e8ce35c2 00000000000004ce ffff81033fbbd9c8 0000000080308800 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:29:02 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] request_module+0x0/0x14d May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: cqueue/1 S 0000000000000282 0 296 123 297 295 (L-TLB) May 17 21:29:02 client-5 kernel: ffff81033fe27e70 0000000000000046 00000013e8ce47b4 0000000000000001 May 17 21:29:02 client-5 kernel: ffff81033fbbd080 0000000000000005 ffff81033fbbd080 ffff81033fbc0820 May 17 21:29:02 client-5 kernel: 00000013e8ce66a7 00000000000002a1 ffff81033fbbd268 0000000180308800 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] thread_return+0x62/0xfe May 17 21:29:02 client-5 kernel: [] recalc_sigpending_and_wake+0x9/0x1a May 17 21:29:02 client-5 kernel: [] do_sigaction+0x14d/0x199 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:29:02 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: cqueue/2 S ffff810001015120 0 297 123 298 296 (L-TLB) May 17 21:29:02 client-5 kernel: ffff81033fe0fe70 0000000000000046 00000013e8ce692d 0000000000000001 May 17 21:29:02 client-5 kernel: ffff81033fbc0820 0000000000000004 ffff81033fbc0820 ffff81010b764080 May 17 21:29:02 client-5 kernel: 00000013e8ce7929 00000000000003fe ffff81033fbc0a08 0000000280308800 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:29:02 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] request_module+0x0/0x14d May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: cqueue/3 S ffff81000101d7a0 0 298 123 301 297 (L-TLB) May 17 21:29:02 client-5 kernel: ffff81033fdf1e70 0000000000000046 00000013e8ce88c0 0000000000000001 May 17 21:29:02 client-5 kernel: ffff81033fbc37a0 0000000000000003 ffff81033fbc37a0 ffff81010b770100 May 17 21:29:02 client-5 kernel: 00000013e8ce97ff 00000000000003c5 ffff81033fbc3988 0000000380308800 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:29:02 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] request_module+0x0/0x14d May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: khubd S ffff81033eb37820 0 301 123 303 298 (L-TLB) May 17 21:29:02 client-5 kernel: ffff81033fe25e10 0000000000000046 ffffffff8008cf9d 8000018000000000 May 17 21:29:02 client-5 kernel: ffff81033eb80000 000000000000000a ffff81033fbb8860 ffff81033fc53820 May 17 21:29:02 client-5 kernel: 00000014aed2f197 0000000000003fbb ffff81033fbb8a48 000000003ebb7d60 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] hub_port_status+0x59/0xcf May 17 21:29:02 client-5 kernel: [] hub_thread+0xa83/0xb01 May 17 21:29:02 client-5 kernel: [] autoremove_wake_function+0x0/0x2e May 17 21:29:02 client-5 kernel: [] hub_thread+0x0/0xb01 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] request_module+0x0/0x14d May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: kseriod S 0000000000000282 0 303 123 394 301 (L-TLB) May 17 21:29:02 client-5 kernel: ffff81033fe2fe90 0000000000000046 0000000000008080 ffffffff8010d8eb May 17 21:29:02 client-5 kernel: ffffffff8033ba20 000000000000000a ffff81033fbc3040 ffff81033b0aa7e0 May 17 21:29:02 client-5 kernel: 0000001a9de323d0 00000000000045c7 ffff81033fbc3228 0000000288187950 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] sysfs_make_dirent+0x1b/0x8f May 17 21:29:02 client-5 kernel: [] driver_create_file+0x31/0x39 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] serio_thread+0x29c/0x2e7 May 17 21:29:02 client-5 kernel: [] autoremove_wake_function+0x0/0x2e May 17 21:29:02 client-5 kernel: [] serio_thread+0x0/0x2e7 May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: khungtaskd S ffff810001015120 0 394 123 395 303 (L-TLB) May 17 21:29:02 client-5 kernel: ffff81033fed5e60 0000000000000046 0000000000000000 0000000000000282 May 17 21:29:02 client-5 kernel: 0000000000000000 000000000000000a ffff81033fc0f860 ffff81010b764080 May 17 21:29:02 client-5 kernel: 0000426f278bb3bc 0000000000007f35 ffff81033fc0fa48 000000028009ef6a May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] schedule_timeout+0x8a/0xad May 17 21:29:02 client-5 kernel: [] process_timeout+0x0/0x5 May 17 21:29:02 client-5 kernel: [] watchdog+0x4b/0x18f May 17 21:29:02 client-5 kernel: [] watchdog+0x0/0x18f May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] request_module+0x0/0x14d May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: pdflush D ffff810001004420 0 395 123 396 394 (L-TLB) May 17 21:29:02 client-5 kernel: ffff81033e977b30 0000000000000046 ffff8102c2ec44a8 0000000000000008 May 17 21:29:02 client-5 kernel: ffff81032217d440 000000000000000a ffff81033fc0c0c0 ffffffff80308b60 May 17 21:29:02 client-5 kernel: 0000401e1441fdc5 0000000000002389 ffff81033fc0c2a8 00000000020bdfef May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] do_gettimeofday+0x40/0x90 May 17 21:29:02 client-5 kernel: [] sync_page+0x0/0x43 May 17 21:29:02 client-5 kernel: [] io_schedule+0x3f/0x67 May 17 21:29:02 client-5 kernel: [] sync_page+0x3e/0x43 May 17 21:29:02 client-5 kernel: [] __wait_on_bit_lock+0x36/0x66 May 17 21:29:02 client-5 kernel: [] __lock_page+0x5e/0x64 May 17 21:29:02 client-5 kernel: [] wake_bit_function+0x0/0x23 May 17 21:29:02 client-5 kernel: [] pagevec_lookup_tag+0x1a/0x21 May 17 21:29:02 client-5 kernel: [] mpage_writepages+0x14f/0x37d May 17 21:29:02 client-5 kernel: [] :lustre:ll_writepage+0x0/0x450 May 17 21:29:02 client-5 kernel: [] __next_cpu+0x19/0x28 May 17 21:29:02 client-5 kernel: [] find_busiest_group+0x20d/0x621 May 17 21:29:02 client-5 kernel: [] do_writepages+0x20/0x2f May 17 21:29:02 client-5 kernel: [] __writeback_single_inode+0x1ae/0x328 May 17 21:29:02 client-5 kernel: [] sync_sb_inodes+0x1b5/0x26f May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] writeback_inodes+0x82/0xd8 May 17 21:29:02 client-5 kernel: [] wb_kupdate+0xd4/0x14e May 17 21:29:02 client-5 kernel: [] pdflush+0x0/0x1fb May 17 21:29:02 client-5 kernel: [] pdflush+0x151/0x1fb May 17 21:29:02 client-5 kernel: [] wb_kupdate+0x0/0x14e May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: pdflush S ffff810001004420 0 396 123 397 395 (L-TLB) May 17 21:29:02 client-5 kernel: ffff81033fed3e70 0000000000000046 ffffffff8049bf80 ffffffff803178a0 May 17 21:29:02 client-5 kernel: 0000000000000286 000000000000000a ffff81033fc0c820 ffffffff80308b60 May 17 21:29:02 client-5 kernel: 000033f348fe9041 00000000000017dc ffff81033fc0ca08 0000000000000282 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] wb_kupdate+0x145/0x14e May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] pdflush+0x0/0x1fb May 17 21:29:02 client-5 kernel: [] pdflush+0xed/0x1fb May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] request_module+0x0/0x14d May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: kswapd0 S ffff81000101d7a0 0 397 123 398 396 (L-TLB) May 17 21:29:02 client-5 kernel: ffff81033feb9dd0 0000000000000046 0000000000000002 ffffffff8891ce3f May 17 21:29:02 client-5 kernel: ffff81010abf7770 000000000000000a ffff81033fbfc100 ffff81010b770100 May 17 21:29:02 client-5 kernel: 0000281327efa8bd 000000000059f608 ffff81033fbfc2e8 0000000300000007 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] :ptlrpc:ldlm_pools_shrink+0x1bf/0x300 May 17 21:29:02 client-5 kernel: [] __up_read+0x19/0x7f May 17 21:29:02 client-5 kernel: [] kswapd+0x12e/0x46c May 17 21:29:02 client-5 kernel: [] autoremove_wake_function+0x0/0x2e May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kswapd+0x0/0x46c May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] request_module+0x0/0x14d May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: aio/0 S ffff810001004420 0 398 123 399 397 (L-TLB) May 17 21:29:02 client-5 kernel: ffff81033febbe70 0000000000000046 00000013f411f52b 0000000000000001 May 17 21:29:02 client-5 kernel: ffff81033fbfc860 0000000000000001 ffff81033fbfc860 ffffffff80308b60 May 17 21:29:02 client-5 kernel: 00000013f41209a6 00000000000006a4 ffff81033fbfca48 0000000080308800 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:29:02 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] request_module+0x0/0x14d May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: aio/1 S 0000000000000282 0 399 123 400 398 (L-TLB) May 17 21:29:02 client-5 kernel: ffff81033febfe70 0000000000000046 00000013f41219e7 0000000000000001 May 17 21:29:02 client-5 kernel: ffff81033fbbb7a0 0000000000000001 ffff81033fbbb7a0 ffff81033fbb40c0 May 17 21:29:02 client-5 kernel: 00000013f41237ff 0000000000000307 ffff81033fbbb988 0000000180308800 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] thread_return+0x62/0xfe May 17 21:29:02 client-5 kernel: [] recalc_sigpending_and_wake+0x9/0x1a May 17 21:29:02 client-5 kernel: [] do_sigaction+0x14d/0x199 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:29:02 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] request_module+0x0/0x14d May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: aio/2 S ffff810001015120 0 400 123 401 399 (L-TLB) May 17 21:29:02 client-5 kernel: ffff81033e979e70 0000000000000046 00000013f4123b6b 0000000000000001 May 17 21:29:02 client-5 kernel: ffff81033fbb40c0 000000000000000a ffff81033fbb40c0 ffff81010b764080 May 17 21:29:02 client-5 kernel: 00000013f4124c7e 00000000000004c7 ffff81033fbb42a8 0000000280308800 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:29:02 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: aio/3 S ffff81000101d7a0 0 401 123 552 400 (L-TLB) May 17 21:29:02 client-5 kernel: ffff81033e97be70 0000000000000046 00000013f4126305 0000000000000001 May 17 21:29:02 client-5 kernel: ffff81033fbc1100 0000000000000001 ffff81033fbc1100 ffff81010b770100 May 17 21:29:02 client-5 kernel: 00000013f48abef6 00000000000004a3 ffff81033fbc12e8 0000000380308800 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:29:02 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: kpsmoused S 0000000000000282 0 552 123 597 401 (L-TLB) May 17 21:29:02 client-5 kernel: ffff81033e907e70 0000000000000046 0000001458d59855 0000000000000001 May 17 21:29:02 client-5 kernel: ffff81033fc05860 0000000000000009 ffff81033fc05860 ffff81033fbb4820 May 17 21:29:02 client-5 kernel: 0000001458d5c7fe 00000000000005bb ffff81033fc05a48 0000000180308800 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] thread_return+0x62/0xfe May 17 21:29:02 client-5 kernel: [] recalc_sigpending_and_wake+0x9/0x1a May 17 21:29:02 client-5 kernel: [] do_sigaction+0x14d/0x199 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:29:02 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] request_module+0x0/0x14d May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: ata/0 S ffff810001004420 0 597 123 598 552 (L-TLB) May 17 21:29:02 client-5 kernel: ffff81033e5ebe70 0000000000000046 00000014b12f9a53 0000000000000001 May 17 21:29:02 client-5 kernel: ffff81033fb79820 0000000000000008 ffff81033fb79820 ffffffff80308b60 May 17 21:29:02 client-5 kernel: 00000014b130c0ab 0000000000000958 ffff81033fb79a08 000000003e91cac0 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:29:02 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: ata/1 S 0000000000000282 0 598 123 599 597 (L-TLB) May 17 21:29:02 client-5 kernel: ffff81033e5ede70 0000000000000046 00000014b130b02b 0000000000000001 May 17 21:29:02 client-5 kernel: ffff81033fc040c0 0000000000000008 ffff81033fc040c0 ffff81033fc04820 May 17 21:29:02 client-5 kernel: 00000014b130dfce 0000000000000437 ffff81033fc042a8 000000013e91cac0 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] thread_return+0x62/0xfe May 17 21:29:02 client-5 kernel: [] recalc_sigpending_and_wake+0x9/0x1a May 17 21:29:02 client-5 kernel: [] do_sigaction+0x14d/0x199 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:29:02 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: ata/2 S 0000000000000282 0 599 123 600 598 (L-TLB) May 17 21:29:02 client-5 kernel: ffff81033e5efe70 0000000000000046 00000014b130e402 0000000000000001 May 17 21:29:02 client-5 kernel: ffff81033fc04820 0000000000000008 ffff81033fc04820 ffff81033fc4c100 May 17 21:29:02 client-5 kernel: 00000014b131e42b 00000000000003fe ffff81033fc04a08 0000000200000000 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] thread_return+0x62/0xfe May 17 21:29:02 client-5 kernel: [] recalc_sigpending_and_wake+0x9/0x1a May 17 21:29:02 client-5 kernel: [] do_sigaction+0x14d/0x199 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:29:02 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: ata/3 S ffff81000101d7a0 0 600 123 601 599 (L-TLB) May 17 21:29:02 client-5 kernel: ffff81033e5f3e70 0000000000000046 00000014b131f84d 0000000000000001 May 17 21:29:02 client-5 kernel: ffff81033fc01080 0000000000000008 ffff81033fc01080 ffff81010b770100 May 17 21:29:02 client-5 kernel: 00000014b1320fc0 000000000000069a ffff81033fc01268 000000033e91cac0 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:29:02 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: ata_aux S ffff81000100caa0 0 601 123 607 600 (L-TLB) May 17 21:29:02 client-5 kernel: ffff81033e5f5e70 0000000000000046 00000014b1323024 0000000000000001 May 17 21:29:02 client-5 kernel: ffff81033fb6f0c0 0000000000000008 ffff81033fb6f0c0 ffff81010b70e100 May 17 21:29:02 client-5 kernel: 00000014b1324279 000000000000028f ffff81033fb6f2a8 000000013e91cac0 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:29:02 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: scsi_eh_0 S ffff810001004420 0 607 123 608 601 (L-TLB) May 17 21:29:02 client-5 kernel: ffff81033e451e70 0000000000000046 ffff81033e55afc0 ffffffff8008b3c7 May 17 21:29:02 client-5 kernel: 0000000300000000 0000000000000009 ffff81033fb6d080 ffffffff80308b60 May 17 21:29:02 client-5 kernel: 000000150573b39f 00000000002ebf37 ffff81033fb6d268 000000008002e23f May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] __wake_up_common+0x3e/0x68 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] :scsi_mod:scsi_error_handler+0x6a/0x4ac May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] :scsi_mod:scsi_error_handler+0x0/0x4ac May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: scsi_eh_1 S ffff81000100caa0 0 608 123 609 607 (L-TLB) May 17 21:29:02 client-5 kernel: ffff81033e565e70 0000000000000046 ffff81033e55efc0 ffffffff8008b3c7 May 17 21:29:02 client-5 kernel: 0000000300000000 000000000000000a ffff81033fb6d7e0 ffff81010b70e100 May 17 21:29:02 client-5 kernel: 0000001517dccf97 000000000041391d ffff81033fb6d9c8 000000018002e23f May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] __wake_up_common+0x3e/0x68 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] :scsi_mod:scsi_error_handler+0x6a/0x4ac May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] :scsi_mod:scsi_error_handler+0x0/0x4ac May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: scsi_eh_2 S ffff81000100caa0 0 609 123 610 608 (L-TLB) May 17 21:29:02 client-5 kernel: ffff81033e5dfe70 0000000000000046 ffff81033e572fc0 ffffffff8008b3c7 May 17 21:29:02 client-5 kernel: 0000000300000000 000000000000000a ffff81033fb6c040 ffff81010b70e100 May 17 21:29:02 client-5 kernel: 000000152a3feacd 0000000000413762 ffff81033fb6c228 000000018002e23f May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] __wake_up_common+0x3e/0x68 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] :scsi_mod:scsi_error_handler+0x6a/0x4ac May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] :scsi_mod:scsi_error_handler+0x0/0x4ac May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: scsi_eh_3 S ffff81000100caa0 0 610 123 611 609 (L-TLB) May 17 21:29:02 client-5 kernel: ffff81033e5fde70 0000000000000046 ffff81033e576fc0 ffffffff8008b3c7 May 17 21:29:02 client-5 kernel: 0000000300000000 000000000000000a ffff81033fb6c7a0 ffff81010b70e100 May 17 21:29:02 client-5 kernel: 000000153ca30446 0000000000413237 ffff81033fb6c988 000000018002e23f May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] __wake_up_common+0x3e/0x68 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] :scsi_mod:scsi_error_handler+0x6a/0x4ac May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] :scsi_mod:scsi_error_handler+0x0/0x4ac May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: scsi_eh_4 S ffff81000100caa0 0 611 123 612 610 (L-TLB) May 17 21:29:02 client-5 kernel: ffff81033e5ffe70 0000000000000046 ffff81033e56afc0 ffffffff8008b3c7 May 17 21:29:02 client-5 kernel: 0000000300000000 000000000000000a ffff81033fb69100 ffff81010b70e100 May 17 21:29:02 client-5 kernel: 000000154f06245a 00000000004136db ffff81033fb692e8 000000018002e23f May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] __wake_up_common+0x3e/0x68 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] :scsi_mod:scsi_error_handler+0x6a/0x4ac May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] :scsi_mod:scsi_error_handler+0x0/0x4ac May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: scsi_eh_5 S ffff81000100caa0 0 612 123 623 611 (L-TLB) May 17 21:29:02 client-5 kernel: ffff81033e6c1e70 0000000000000046 ffff81033e56efc0 ffffffff8008b3c7 May 17 21:29:02 client-5 kernel: 0000000300000000 000000000000000a ffff81033fb69860 ffff81010b70e100 May 17 21:29:02 client-5 kernel: 0000001561691fb3 000000000041331f ffff81033fb69a48 000000018002e23f May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] __wake_up_common+0x3e/0x68 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] :scsi_mod:scsi_error_handler+0x6a/0x4ac May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] :scsi_mod:scsi_error_handler+0x0/0x4ac May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: kstriped S ffff81000100caa0 0 623 123 644 612 (L-TLB) May 17 21:29:02 client-5 kernel: ffff81033e6c7e70 0000000000000046 0000001565979b8d 0000000000000001 May 17 21:29:02 client-5 kernel: ffff81033fb64080 0000000000000009 ffff81033fb64080 ffff81010b70e100 May 17 21:29:02 client-5 kernel: 000000156597c680 0000000000000435 ffff81033fb64268 000000013e91cac0 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:29:02 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: kjournald D ffff81010b7752c0 0 644 123 664 623 (L-TLB) May 17 21:29:02 client-5 kernel: ffff81033cf9fcf0 0000000000000046 ffff81033e558000 ffffffff880bd0df May 17 21:29:02 client-5 kernel: 0000000000000286 000000000000000a ffff81033f5d17e0 ffff81010b764080 May 17 21:29:02 client-5 kernel: 000042775271dff9 ffff810001015120 ffff81033cf9fca0 ffffffff8008c9a2 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] :libata:ata_scsi_translate+0x140/0x16d May 17 21:29:02 client-5 kernel: [] enqueue_task+0x41/0x56 May 17 21:29:02 client-5 kernel: [] __activate_task+0x56/0x6d May 17 21:29:02 client-5 kernel: [] try_to_wake_up+0x472/0x484 May 17 21:29:02 client-5 kernel: [] sync_buffer+0x0/0x3f May 17 21:29:02 client-5 kernel: [] io_schedule+0x3f/0x67 May 17 21:29:02 client-5 kernel: [] sync_buffer+0x3b/0x3f May 17 21:29:02 client-5 kernel: [] autoremove_wake_function+0x9/0x2e May 17 21:29:02 client-5 kernel: [] dequeue_task+0x18/0x37 May 17 21:29:02 client-5 kernel: [] deactivate_task+0x28/0x5f May 17 21:29:02 client-5 kernel: [] :jbd:kjournald+0x192/0x213 May 17 21:29:02 client-5 kernel: [] autoremove_wake_function+0x0/0x2e May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] :jbd:kjournald+0x0/0x213 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] acpi_processor_idle_simple+0x10d/0x30e May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: kauditd S ffff81033aba4820 0 664 123 1510 644 (L-TLB) May 17 21:29:02 client-5 kernel: ffff81033e7a9e90 0000000000000046 ffff81032a9775b8 ffff81033e7a9e20 May 17 21:29:02 client-5 kernel: 0000000000000001 000000000000000a ffff81033fbb17e0 ffff81033aba4820 May 17 21:29:02 client-5 kernel: 000040efcc123ce1 0000000000000ac6 ffff81033fbb19c8 0000000300000001 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] __wake_up+0x38/0x4f May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kauditd_thread+0x144/0x17b May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] kauditd_thread+0x0/0x17b May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: udevd S ffff81000101d7a0 0 692 1 2526 123 (NOTLB) May 17 21:29:02 client-5 kernel: ffff81033f84da18 0000000000000086 ffff81033faccbc0 00002b24810a54c0 May 17 21:29:02 client-5 kernel: 0000000000000296 0000000000000002 ffff81033fc53820 ffff81010b770100 May 17 21:29:02 client-5 kernel: 00003ffe099d66de 000000000000e468 ffff81033fc53a08 0000000300000005 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] schedule_timeout+0x1e/0xad May 17 21:29:02 client-5 kernel: [] mutex_lock+0xd/0x1d May 17 21:29:02 client-5 kernel: [] inotify_poll+0x49/0x50 May 17 21:29:02 client-5 kernel: [] do_select+0x3fe/0x462 May 17 21:29:02 client-5 kernel: [] __pollwait+0x0/0xe2 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 last message repeated 3 times May 17 21:29:02 client-5 kernel: [] __alloc_pages+0x78/0x308 May 17 21:29:02 client-5 kernel: [] flush_tlb_page+0xac/0xda May 17 21:29:02 client-5 kernel: [] vsnprintf+0x5e7/0x62f May 17 21:29:02 client-5 kernel: [] core_sys_select+0x1bc/0x265 May 17 21:29:02 client-5 kernel: [] mntput_no_expire+0x19/0x89 May 17 21:29:02 client-5 kernel: [] shrink_dcache_parent+0x3e/0xe7 May 17 21:29:02 client-5 kernel: [] sys_select+0xc1/0x17c May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] sys_read+0x45/0x6e May 17 21:29:02 client-5 kernel: [] system_call+0x7e/0x83 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: mlx4 S ffff81000101d7a0 0 1510 123 1672 664 (L-TLB) May 17 21:29:02 client-5 kernel: ffff8103391bfe70 0000000000000046 0000001aaa2c0ac3 0000000000000003 May 17 21:29:02 client-5 kernel: ffff81033b547860 0000000000000005 ffff81033b547860 ffff81010b770100 May 17 21:29:02 client-5 kernel: 0000001aaa2c554a 0000000000000707 ffff81033b547a48 000000033eac60c0 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:29:02 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: mlx4_sense S ffff81000100caa0 0 1672 123 1698 1510 (L-TLB) May 17 21:29:02 client-5 kernel: ffff81033426fe70 0000000000000046 ffff81033bc429e0 0000000000000286 May 17 21:29:02 client-5 kernel: 0000000300000000 000000000000000a ffff81033ad91860 ffff81010b70e100 May 17 21:29:02 client-5 kernel: 0000427629163aad 0000000000000b29 ffff81033ad91a48 000000018002e23f May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:29:02 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] acpi_processor_idle_simple+0x17d/0x30e May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: mlx4_ib S 0000000000000282 0 1698 123 1699 1672 (L-TLB) May 17 21:29:02 client-5 kernel: ffff810333ddbe70 0000000000000046 0000000000000000 0000000000000000 May 17 21:29:02 client-5 kernel: ffff810333dda000 0000000000000008 ffff81033b956040 ffff81010b6fd7a0 May 17 21:29:02 client-5 kernel: 0000001b52723762 0000000000000691 ffff81033b956228 0000000300000000 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] thread_return+0x62/0xfe May 17 21:29:02 client-5 kernel: [] recalc_sigpending_and_wake+0x9/0x1a May 17 21:29:02 client-5 kernel: [] do_sigaction+0x14d/0x199 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:29:02 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: ib_mad1 S ffff810001004420 0 1699 123 1885 1698 (L-TLB) May 17 21:29:02 client-5 kernel: ffff810333ee1e70 0000000000000046 000001283ea27ce0 ffff81033f412000 May 17 21:29:02 client-5 kernel: 0000000300000000 000000000000000a ffff81033ae7b7e0 ffffffff80308b60 May 17 21:29:02 client-5 kernel: 00004261b07fe80c 00000000000015e2 ffff81033ae7b9c8 000000008002e23f May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:29:02 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] acpi_processor_idle_simple+0x10d/0x30e May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: kmpathd/0 S 0000000000000282 0 1885 123 1886 1699 (L-TLB) May 17 21:29:02 client-5 kernel: ffff810333e03e70 0000000000000046 0000001c39ed6f2a 0000000000000003 May 17 21:29:02 client-5 kernel: ffff81033fc530c0 0000000000000008 ffff81033fc530c0 ffff81033aace7a0 May 17 21:29:02 client-5 kernel: 0000001c39ef8642 0000000000000552 ffff81033fc532a8 0000000000000000 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] thread_return+0x62/0xfe May 17 21:29:02 client-5 kernel: [] recalc_sigpending_and_wake+0x9/0x1a May 17 21:29:02 client-5 kernel: [] do_sigaction+0x14d/0x199 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:29:02 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: kmpathd/1 S ffff81000100caa0 0 1886 123 1887 1885 (L-TLB) May 17 21:29:02 client-5 kernel: ffff810333e05e70 0000000000000046 0000001c39ef9d91 0000000000000003 May 17 21:29:02 client-5 kernel: ffff81033b740860 0000000000000008 ffff81033b740860 ffff81010b70e100 May 17 21:29:02 client-5 kernel: 0000001c39efe6cd 000000000000087d ffff81033b740a48 000000013b5f8ac0 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:29:02 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: kmpathd/2 S ffff810001015120 0 1887 123 1888 1886 (L-TLB) May 17 21:29:02 client-5 kernel: ffff810333e07e70 0000000000000046 0000001c39efe867 0000000000000003 May 17 21:29:02 client-5 kernel: ffff81033b90e080 0000000000000006 ffff81033b90e080 ffff81010b764080 May 17 21:29:02 client-5 kernel: 0000001c39f10293 00000000000008a1 ffff81033b90e268 000000023faccbc0 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:29:02 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: kmpathd/3 S ffff81000101d7a0 0 1888 123 1889 1887 (L-TLB) May 17 21:29:02 client-5 kernel: ffff810333e09e70 0000000000000046 0000001c39f0fd94 0000000000000003 May 17 21:29:02 client-5 kernel: ffff81033fbb4820 0000000000000005 ffff81033fbb4820 ffff81010b770100 May 17 21:29:02 client-5 kernel: 0000001c39f111ec 00000000000003b0 ffff81033fbb4a08 000000033eac60c0 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:29:02 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: kmpath_handle S ffff81000101d7a0 0 1889 123 2114 1888 (L-TLB) May 17 21:29:02 client-5 kernel: ffff810333e0be70 0000000000000046 0000001c39f1373a 0000000000000003 May 17 21:29:02 client-5 kernel: ffff81033ab037e0 0000000000000004 ffff81033ab037e0 ffff81010b770100 May 17 21:29:02 client-5 kernel: 0000001c39f14a96 000000000000033a ffff81033ab039c8 000000033eac60c0 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:29:02 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: mthcacatas S ffff81000101d7a0 0 2114 123 2132 1889 (L-TLB) May 17 21:29:02 client-5 kernel: ffff810332403e70 0000000000000046 0000001c8b232caf 0000000000000003 May 17 21:29:02 client-5 kernel: ffff81033bb43820 0000000000000009 ffff81033bb43820 ffff81010b770100 May 17 21:29:02 client-5 kernel: 0000001c8b241d5f 0000000000000551 ffff81033bb43a08 000000033b464880 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:29:02 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: mlx4_en S ffff81000101d7a0 0 2132 123 2176 2114 (L-TLB) May 17 21:29:02 client-5 kernel: ffff810332757e70 0000000000000046 0000001c8f01669c 0000000000000003 May 17 21:29:02 client-5 kernel: ffff81033b1580c0 0000000000000009 ffff81033b1580c0 ffff81010b770100 May 17 21:29:02 client-5 kernel: 0000001c8f0196a3 0000000000000424 ffff81033b1582a8 000000033be7bbc0 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:29:02 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: iw_cxgb3 S 0000000000000282 0 2176 123 2185 2132 (L-TLB) May 17 21:29:02 client-5 kernel: ffff81033f677e70 0000000000000046 0000001c97d61818 0000000000000003 May 17 21:29:02 client-5 kernel: ffff81033fbad100 0000000000000008 ffff81033fbad100 ffff81033fb787e0 May 17 21:29:02 client-5 kernel: 0000001c97d62416 00000000000002c7 ffff81033fbad2e8 0000000300000000 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] thread_return+0x62/0xfe May 17 21:29:02 client-5 kernel: [] recalc_sigpending_and_wake+0x9/0x1a May 17 21:29:02 client-5 kernel: [] do_sigaction+0x14d/0x199 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:29:02 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: nesewq S ffff81000101d7a0 0 2185 123 2186 2176 (L-TLB) May 17 21:29:02 client-5 kernel: ffff8103316ade70 0000000000000046 0000001c9a6256b4 0000000000000003 May 17 21:29:02 client-5 kernel: ffff81033b9567a0 0000000000000006 ffff81033b9567a0 ffff81010b770100 May 17 21:29:02 client-5 kernel: 0000001c9a629af2 000000000000054b ffff81033b956988 000000033a993b80 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:29:02 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: nesdwq S ffff81000101d7a0 0 2186 123 2188 2185 (L-TLB) May 17 21:29:02 client-5 kernel: ffff8103316afe70 0000000000000046 0000001c9a62d44f 0000000000000003 May 17 21:29:02 client-5 kernel: ffff81033ba9a100 0000000000000006 ffff81033ba9a100 ffff81010b770100 May 17 21:29:02 client-5 kernel: 0000001c9a6323ed 000000000000035d ffff81033ba9a2e8 000000033a993b80 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:29:02 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: nes_ud S ffff81000101d7a0 0 2188 123 2244 2186 (L-TLB) May 17 21:29:02 client-5 kernel: ffff81033118fe70 0000000000000046 0000001c9a680d59 0000000000000003 May 17 21:29:02 client-5 kernel: ffff81033fb5f100 0000000000000005 ffff81033fb5f100 ffff81010b770100 May 17 21:29:02 client-5 kernel: 0000001c9a6822a2 00000000000002f8 ffff81033fb5f2e8 000000033a993b80 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:29:02 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: ib_mcast S 0000000000000282 0 2244 123 2245 2188 (L-TLB) May 17 21:29:02 client-5 kernel: ffff810331209e70 0000000000000046 ffff81033cfc5a00 ffff81033cfc5a60 May 17 21:29:02 client-5 kernel: 0000000300000000 000000000000000a ffff81033bd96820 ffff81033b90e7e0 May 17 21:29:02 client-5 kernel: 00000020516862a2 0000000000003813 ffff81033bd96a08 000000008002e23f May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] :ib_sa:mcast_work_handler+0x0/0x4e5 May 17 21:29:02 client-5 kernel: [] run_workqueue+0xba/0xe4 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:29:02 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: ib_inform S 0000000000000282 0 2245 123 2246 2244 (L-TLB) May 17 21:29:02 client-5 kernel: ffff81033120be70 0000000000000046 0000001ca52755ce 0000000000000003 May 17 21:29:02 client-5 kernel: ffff81033ab1b100 0000000000000006 ffff81033ab1b100 ffff81033bf037a0 May 17 21:29:02 client-5 kernel: 0000001ca5275d8d 000000000000018e ffff81033ab1b2e8 0000000300000000 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] thread_return+0x62/0xfe May 17 21:29:02 client-5 kernel: [] recalc_sigpending_and_wake+0x9/0x1a May 17 21:29:02 client-5 kernel: [] do_sigaction+0x14d/0x199 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:29:02 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: local_sa S ffff81000101d7a0 0 2246 123 2252 2245 (L-TLB) May 17 21:29:02 client-5 kernel: ffff810331da1e70 0000000000000046 0000001ca5278521 0000000000000003 May 17 21:29:02 client-5 kernel: 0000000300000000 000000000000000a ffff81033fc127a0 ffff81010b770100 May 17 21:29:02 client-5 kernel: 0000001fab5ac812 00000000000013c7 ffff81033fc12988 000000038002e23f May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:29:02 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: ib_cm/0 S ffff81033ae7b7e0 0 2252 123 2253 2246 (L-TLB) May 17 21:29:02 client-5 kernel: ffff81033132be70 0000000000000046 ffff8103113205c0 ffffffff8850104a May 17 21:29:02 client-5 kernel: 0000000300000000 000000000000000a ffff81033aba40c0 ffff81033ae7b7e0 May 17 21:29:02 client-5 kernel: 000033688079addc 0000000000004d8b ffff81033aba42a8 000000008002e23f May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] :ib_cm:cm_rep_handler+0x3ac/0x3ec May 17 21:29:02 client-5 kernel: [] :ib_cm:cm_work_handler+0x0/0xb0f May 17 21:29:02 client-5 kernel: [] run_workqueue+0xba/0xe4 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:29:02 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: ib_cm/1 S ffff81033ae7b7e0 0 2253 123 2254 2252 (L-TLB) May 17 21:29:02 client-5 kernel: ffff81033132de70 0000000000000046 ffff81030fa07d80 ffffffff8850104a May 17 21:29:02 client-5 kernel: 0000000300000000 0000000000000009 ffff81033fbaf7a0 ffff81033ae7b7e0 May 17 21:29:02 client-5 kernel: 0000400e244fe122 0000000000006048 ffff81033fbaf988 000000018002e23f May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] :ib_cm:cm_rep_handler+0x3ac/0x3ec May 17 21:29:02 client-5 kernel: [] :ib_cm:cm_work_handler+0x0/0xb0f May 17 21:29:02 client-5 kernel: [] run_workqueue+0xba/0xe4 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:29:02 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: ib_cm/2 S ffff810001015120 0 2254 123 2255 2253 (L-TLB) May 17 21:29:02 client-5 kernel: ffff81033132fe70 0000000000000046 ffff81032349ab80 ffffffff8850104a May 17 21:29:02 client-5 kernel: 0000000300000000 000000000000000a ffff81033bb84860 ffff81010b764080 May 17 21:29:02 client-5 kernel: 0000300fdf89d3f9 00000000000046d3 ffff81033bb84a48 000000028002e23f May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] :ib_cm:cm_rep_handler+0x3ac/0x3ec May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:29:02 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: ib_cm/3 S ffff81033ae7b7e0 0 2255 123 2272 2254 (L-TLB) May 17 21:29:02 client-5 kernel: ffff810331331e70 0000000000000046 ffff81031e3278c0 ffffffff8850104a May 17 21:29:02 client-5 kernel: 0000000300000000 000000000000000a ffff81033b0407e0 ffff81033ae7b7e0 May 17 21:29:02 client-5 kernel: 0000385250d68cb5 000000000000626e ffff81033b0409c8 000000038002e23f May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] :ib_cm:cm_rep_handler+0x3ac/0x3ec May 17 21:29:02 client-5 kernel: [] :ib_cm:cm_work_handler+0x0/0xb0f May 17 21:29:02 client-5 kernel: [] run_workqueue+0xba/0xe4 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:29:02 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: ipoib S ffff81000100caa0 0 2272 123 2382 2255 (L-TLB) May 17 21:29:02 client-5 kernel: ffff8103308f1e70 0000000000000046 ffff8103308f2f68 0000000000000286 May 17 21:29:02 client-5 kernel: 0000000300000000 000000000000000a ffff81033bb84100 ffff81010b70e100 May 17 21:29:02 client-5 kernel: 00004276429d4d3d 00000000000009f2 ffff81033bb842e8 000000018002e23f May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:29:02 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: ib_addr S ffff810001004420 0 2382 123 2388 2272 (L-TLB) May 17 21:29:02 client-5 kernel: ffff81032a949e70 0000000000000046 0000000000000000 0000000000000000 May 17 21:29:02 client-5 kernel: 0000000300000000 000000000000000a ffff81033b304860 ffffffff80308b60 May 17 21:29:02 client-5 kernel: 0000426de0625de1 0000000000000791 ffff81033b304a48 000000008002e23f May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:29:02 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: iw_cm_wq S ffff81000101d7a0 0 2388 123 2393 2382 (L-TLB) May 17 21:29:02 client-5 kernel: ffff81032a931e70 0000000000000046 0000001de92c58d9 0000000000000003 May 17 21:29:02 client-5 kernel: ffff81033aca80c0 0000000000000008 ffff81033aca80c0 ffff81010b770100 May 17 21:29:02 client-5 kernel: 0000001de92c7102 00000000000004d8 ffff81033aca82a8 000000033bb63400 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:29:02 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: rdma_cm S ffff810001004420 0 2393 123 2400 2388 (L-TLB) May 17 21:29:02 client-5 kernel: ffff81032aaf5e70 0000000000000046 0000001de99294a5 0000000000000003 May 17 21:29:02 client-5 kernel: 0000000300000000 000000000000000a ffff81033fb320c0 ffffffff80308b60 May 17 21:29:02 client-5 kernel: 0000400e24324b35 0000000000005d86 ffff81033fb322a8 000000008002e23f May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:29:02 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: rx_comp_wq/0 S 0000000000000282 0 2400 123 2401 2393 (L-TLB) May 17 21:29:02 client-5 kernel: ffff81032a24fe70 0000000000000046 0000001deb854d91 0000000000000003 May 17 21:29:02 client-5 kernel: ffff81033aa460c0 0000000000000008 ffff81033aa460c0 ffff81033c5467e0 May 17 21:29:02 client-5 kernel: 0000001deb85db3a 0000000000000554 ffff81033aa462a8 0000000000000000 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] thread_return+0x62/0xfe May 17 21:29:02 client-5 kernel: [] recalc_sigpending_and_wake+0x9/0x1a May 17 21:29:02 client-5 kernel: [] do_sigaction+0x14d/0x199 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:29:02 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: rx_comp_wq/1 S ffff81000100caa0 0 2401 123 2402 2400 (L-TLB) May 17 21:29:02 client-5 kernel: ffff81032a151e70 0000000000000046 0000001deb85f0d8 0000000000000003 May 17 21:29:02 client-5 kernel: ffff81033b6f6080 0000000000000006 ffff81033b6f6080 ffff81010b70e100 May 17 21:29:02 client-5 kernel: 0000001deb86cd6a 000000000000082e ffff81033b6f6268 000000013eac60c0 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:29:02 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: rx_comp_wq/2 S ffff810001015120 0 2402 123 2403 2401 (L-TLB) May 17 21:29:02 client-5 kernel: ffff81032a153e70 0000000000000046 0000001deb86cf2f 0000000000000003 May 17 21:29:02 client-5 kernel: ffff81033ad4f100 0000000000000006 ffff81033ad4f100 ffff81010b764080 May 17 21:29:02 client-5 kernel: 0000001deb87ed98 0000000000000a5c ffff81033ad4f2e8 000000023faccbc0 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:29:02 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: rx_comp_wq/3 S ffff81000101d7a0 0 2403 123 2404 2402 (L-TLB) May 17 21:29:02 client-5 kernel: ffff81032a155e70 0000000000000046 0000001deb87f978 0000000000000003 May 17 21:29:02 client-5 kernel: ffff81033fb39040 0000000000000007 ffff81033fb39040 ffff81010b770100 May 17 21:29:02 client-5 kernel: 0000001deb8813e3 000000000000049f ffff81033fb39228 000000033bb63400 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:29:02 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: sdp_wq S ffff81000101d7a0 0 2404 123 2405 2403 (L-TLB) May 17 21:29:02 client-5 kernel: ffff81032a157e70 0000000000000046 0000001deb885379 0000000000000003 May 17 21:29:02 client-5 kernel: ffff81033fbad860 0000000000000007 ffff81033fbad860 ffff81010b770100 May 17 21:29:02 client-5 kernel: 0000001deb88667c 0000000000000323 ffff81033fbada48 000000033bb63400 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:29:02 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: ib_fmr(mlx4_0 S ffff81000101d7a0 0 2405 123 2412 2404 (L-TLB) May 17 21:29:02 client-5 kernel: ffff81032a141ed0 0000000000000046 ffff81032a141ee0 ffffffff80062ff8 May 17 21:29:02 client-5 kernel: ffff81033c46a100 0000000000000007 ffff81033c46a100 ffff81010b770100 May 17 21:29:02 client-5 kernel: 0000001deb8a82f5 000000000000040f ffff81033c46a2e8 00000003000003a2 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] thread_return+0x62/0xfe May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] :ib_core:ib_fmr_cleanup_thread+0x0/0x87 May 17 21:29:02 client-5 kernel: [] :ib_core:ib_fmr_cleanup_thread+0x6a/0x87 May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: qlgc_vnic_vip S 0000000000000282 0 2412 123 2413 2405 (L-TLB) May 17 21:29:02 client-5 kernel: ffff81032a2afe70 0000000000000046 ffff81010b7e4068 000000000000003d May 17 21:29:02 client-5 kernel: ffff81032a2ae000 0000000000000007 ffff81033fbb1080 ffff81033c7d90c0 May 17 21:29:02 client-5 kernel: 0000001ded911df2 0000000000000d87 ffff81033fbb1268 000000033f521100 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] thread_return+0x62/0xfe May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] :qlgc_vnic:viport_statemachine+0x0/0x138a May 17 21:29:02 client-5 kernel: [] :qlgc_vnic:viport_statemachine+0xc3/0x138a May 17 21:29:02 client-5 kernel: [] autoremove_wake_function+0x0/0x2e May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] :qlgc_vnic:viport_statemachine+0x0/0x138a May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: qlgc_vnic_npe S ffff81000101d7a0 0 2413 123 2471 2412 (L-TLB) May 17 21:29:02 client-5 kernel: ffff810329c0de80 0000000000000046 ffff81033c7d90c0 ffffffff8008c787 May 17 21:29:02 client-5 kernel: ffff81033c7d90c0 0000000000000007 ffff81033c7d90c0 ffff81010b770100 May 17 21:29:02 client-5 kernel: 0000001ded918289 0000000000000361 ffff81033c7d92a8 000000033f521100 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] dequeue_task+0x18/0x37 May 17 21:29:02 client-5 kernel: [] thread_return+0x62/0xfe May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] :qlgc_vnic:vnic_npevent_statemachine+0x0/0xa85 May 17 21:29:02 client-5 kernel: [] :qlgc_vnic:vnic_npevent_statemachine+0x9b/0xa85 May 17 21:29:02 client-5 kernel: [] autoremove_wake_function+0x0/0x2e May 17 21:29:02 client-5 kernel: [] :qlgc_vnic:vnic_npevent_statemachine+0x0/0xa85 May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: iscsi_eh S ffff81000101d7a0 0 2471 123 3068 2413 (L-TLB) May 17 21:29:02 client-5 kernel: ffff81032a475e70 0000000000000046 0000001df2588751 0000000000000003 May 17 21:29:02 client-5 kernel: ffff81033fbb8100 0000000000000007 ffff81033fbb8100 ffff81010b770100 May 17 21:29:02 client-5 kernel: 0000001df258da9c 000000000000064e ffff81033fbb82e8 000000033c4fd8c0 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:29:02 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: brcm_iscsiuio S ffff81033eb71098 0 2526 1 2527 692 (NOTLB) May 17 21:29:02 client-5 kernel: ffff81032a2a3a98 0000000000000086 0000000000000000 ffff8103338adf70 May 17 21:29:02 client-5 kernel: ffff810331b406c0 0000000000000001 ffff81033ab03080 ffff81033b49b820 May 17 21:29:02 client-5 kernel: 0000001df98a3de9 000000000004a2e1 ffff81033ab03268 0000000080019c08 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] schedule_timeout+0x1e/0xad May 17 21:29:02 client-5 kernel: [] prepare_to_wait_exclusive+0x38/0x65 May 17 21:29:02 client-5 kernel: [] __skb_recv_datagram+0x193/0x215 May 17 21:29:02 client-5 kernel: [] autoremove_wake_function+0x0/0x2e May 17 21:29:02 client-5 kernel: [] :jbd:journal_stop+0x1f3/0x1ff May 17 21:29:02 client-5 kernel: [] skb_recv_datagram+0x1c/0x21 May 17 21:29:02 client-5 kernel: [] :ext3:ext3_ordered_write_end+0xd7/0x116 May 17 21:29:02 client-5 kernel: [] netlink_recvmsg+0x47/0x290 May 17 21:29:02 client-5 kernel: [] generic_file_buffered_write+0x20b/0x675 May 17 21:29:02 client-5 kernel: [] zone_statistics+0x3e/0x6d May 17 21:29:02 client-5 kernel: [] sock_recvmsg+0x107/0x15f May 17 21:29:02 client-5 kernel: [] autoremove_wake_function+0x0/0x2e May 17 21:29:02 client-5 kernel: [] do_wp_page+0x87c/0x902 May 17 21:29:02 client-5 kernel: [] find_get_page+0x21/0x51 May 17 21:29:02 client-5 kernel: [] sys_recvmsg+0x15c/0x24c May 17 21:29:02 client-5 kernel: [] __up_read+0x19/0x7f May 17 21:29:02 client-5 kernel: [] do_page_fault+0x4fe/0x874 May 17 21:29:02 client-5 kernel: [] sys_newstat+0x28/0x31 May 17 21:29:02 client-5 kernel: [] system_call+0x7e/0x83 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: brcm_iscsiuio S 0000000000000000 0 2527 1 2528 2526 (NOTLB) May 17 21:29:02 client-5 kernel: ffff81032a537e58 0000000000000086 ffff81032a537e48 ffffffff8000eaaf May 17 21:29:02 client-5 kernel: ffff81033968d000 0000000000000001 ffff81033a4dd860 ffff81033fb397a0 May 17 21:29:02 client-5 kernel: 0000001df98b14a1 0000000000004214 ffff81033a4dda48 000000003968d000 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] link_path_walk+0xa6/0xb2 May 17 21:29:02 client-5 kernel: [] _atomic_dec_and_lock+0x39/0x57 May 17 21:29:02 client-5 kernel: [] mntput_no_expire+0x19/0x89 May 17 21:29:02 client-5 kernel: [] schedule_timeout+0x1e/0xad May 17 21:29:02 client-5 kernel: [] recalc_sigpending+0xe/0x25 May 17 21:29:02 client-5 kernel: [] dequeue_signal+0x47/0xcd May 17 21:29:02 client-5 kernel: [] sys_rt_sigtimedwait+0x1ce/0x2b4 May 17 21:29:02 client-5 kernel: [] sys_newstat+0x28/0x31 May 17 21:29:02 client-5 kernel: [] system_call+0x7e/0x83 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: brcm_iscsiuio S ffff810001004420 0 2528 1 2531 2527 (NOTLB) May 17 21:29:02 client-5 kernel: ffff810329c2fd78 0000000000000086 00000000ffffffff 0000000000000000 May 17 21:29:02 client-5 kernel: ffff8103a9c2fe47 0000000000000001 ffff81033fb397a0 ffffffff80308b60 May 17 21:29:02 client-5 kernel: 0000001df98b4127 0000000000002c86 ffff81033fb39988 00000000802b278f May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] schedule_timeout+0x1e/0xad May 17 21:29:02 client-5 kernel: [] prepare_to_wait_exclusive+0x38/0x65 May 17 21:29:02 client-5 kernel: [] __skb_recv_datagram+0x193/0x215 May 17 21:29:02 client-5 kernel: [] autoremove_wake_function+0x0/0x2e May 17 21:29:02 client-5 kernel: [] inotify_d_instantiate+0x12/0x3c May 17 21:29:02 client-5 kernel: [] skb_recv_datagram+0x1c/0x21 May 17 21:29:02 client-5 kernel: [] unix_accept+0x54/0x103 May 17 21:29:02 client-5 kernel: [] sys_accept+0x11c/0x1ea May 17 21:29:02 client-5 kernel: [] mntput_no_expire+0x19/0x89 May 17 21:29:02 client-5 kernel: [] filp_close+0x5c/0x64 May 17 21:29:02 client-5 kernel: [] recalc_sigpending+0xe/0x25 May 17 21:29:02 client-5 kernel: [] sigprocmask+0xb7/0xdb May 17 21:29:02 client-5 kernel: [] system_call+0x7e/0x83 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: iscsid S ffff810001015120 0 2531 1 2532 2528 (NOTLB) May 17 21:29:02 client-5 kernel: ffff81032a2ffeb8 0000000000000086 ffff81032a2ffe58 ffff810001017200 May 17 21:29:02 client-5 kernel: 000000003b9aca00 0000000000000007 ffff81033ba9a860 ffff81010b764080 May 17 21:29:02 client-5 kernel: 00004276604ec360 0000000000001418 ffff81033ba9aa48 00000002093c831f May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] enqueue_hrtimer+0x55/0x70 May 17 21:29:02 client-5 kernel: [] do_nanosleep+0x3f/0x70 May 17 21:29:02 client-5 kernel: [] hrtimer_nanosleep+0x58/0x118 May 17 21:29:02 client-5 kernel: [] hrtimer_wakeup+0x0/0x22 May 17 21:29:02 client-5 kernel: [] sigprocmask+0xb7/0xdb May 17 21:29:02 client-5 kernel: [] sys_nanosleep+0x4c/0x62 May 17 21:29:02 client-5 kernel: [] system_call+0x7e/0x83 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: iscsid S ffff81000100caa0 0 2532 1 2995 2531 (NOTLB) May 17 21:29:02 client-5 kernel: ffff81032a35db38 0000000000000086 0000000000582657 ffff81032a35db48 May 17 21:29:02 client-5 kernel: ffff81032a977830 000000000000000a ffff81033b49b820 ffff81010b70e100 May 17 21:29:02 client-5 kernel: 0000427671cd587d 0000000000001468 ffff81033b49ba08 000000012a35de34 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] unix_poll+0x0/0x99 May 17 21:29:02 client-5 kernel: [] schedule_timeout+0x8a/0xad May 17 21:29:02 client-5 kernel: [] process_timeout+0x0/0x5 May 17 21:29:02 client-5 kernel: [] do_sys_poll+0x277/0x360 May 17 21:29:02 client-5 kernel: [] __pollwait+0x0/0xe2 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] __rmqueue+0x91/0xc6 May 17 21:29:02 client-5 kernel: [] vsnprintf+0x5e7/0x62f May 17 21:29:02 client-5 kernel: [] snprintf+0x44/0x4c May 17 21:29:02 client-5 kernel: [] d_lookup+0x1e/0x42 May 17 21:29:02 client-5 kernel: [] proc_flush_task+0x4e/0x1f6 May 17 21:29:02 client-5 kernel: [] sched_exit+0xa9/0xb5 May 17 21:29:02 client-5 kernel: [] release_task+0x3a5/0x3cb May 17 21:29:02 client-5 kernel: [] remove_wait_queue+0x1c/0x2c May 17 21:29:02 client-5 kernel: [] do_wait+0xa3b/0xad7 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] sys_poll+0x2d/0x34 May 17 21:29:02 client-5 kernel: [] system_call+0x7e/0x83 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: auditd S ffff81000101d7a0 0 2995 1 2997 2996 2532 (NOTLB) May 17 21:29:02 client-5 kernel: ffff8103317d7e78 0000000000000082 ffff810300000000 ffff8103317d7c28 May 17 21:29:02 client-5 kernel: ffff8103317d7ee8 000000000000000a ffff81033aba4820 ffff81010b770100 May 17 21:29:02 client-5 kernel: 00004274a069c271 0000000000002c37 ffff81033aba4a08 00000003317d7e60 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] schedule_timeout+0x8a/0xad May 17 21:29:02 client-5 kernel: [] process_timeout+0x0/0x5 May 17 21:29:02 client-5 kernel: [] sys_epoll_wait+0x184/0x3f9 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] ktime_get_ts+0x1a/0x4e May 17 21:29:02 client-5 kernel: [] system_call+0x7e/0x83 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: auditd S ffff81000100caa0 0 2996 1 3017 2995 (NOTLB) May 17 21:29:02 client-5 kernel: ffff81032a0cfd28 0000000000000082 0000000000000000 0000000000000000 May 17 21:29:02 client-5 kernel: ffff81032a0cfd98 000000000000000a ffff81033a646820 ffff81010b70e100 May 17 21:29:02 client-5 kernel: 000040efcc157d35 0000000000007c4d ffff81033a646a08 00000001000000b6 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] schedule_timeout+0x1e/0xad May 17 21:29:02 client-5 kernel: [] wait_on_page_writeback_range+0xd6/0x12e May 17 21:29:02 client-5 kernel: [] add_wait_queue+0x24/0x34 May 17 21:29:02 client-5 kernel: [] do_futex+0x25e/0xcf3 May 17 21:29:02 client-5 kernel: [] __next_cpu+0x19/0x28 May 17 21:29:02 client-5 kernel: [] autoremove_wake_function+0x0/0x2e May 17 21:29:02 client-5 kernel: [] :ext3:ext3_statfs+0xf6/0x113 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] sys_fstatfs+0x57/0x68 May 17 21:29:02 client-5 kernel: [] sys_futex+0x10a/0x12b May 17 21:29:02 client-5 kernel: [] system_call+0x7e/0x83 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: audispd S ffff810001004420 0 2997 2995 2998 (NOTLB) May 17 21:29:02 client-5 kernel: ffff810329ad9d28 0000000000000082 ffffffff800cc235 ffff810329ad9d28 May 17 21:29:02 client-5 kernel: 0000000000000206 000000000000000a ffff81033ad91100 ffffffff80308b60 May 17 21:29:02 client-5 kernel: 000040efcc139a6c 0000000000001e5c ffff81033ad912e8 0000000000000bb5 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] zone_statistics+0x3e/0x6d May 17 21:29:02 client-5 kernel: [] schedule_timeout+0x1e/0xad May 17 21:29:02 client-5 kernel: [] add_wait_queue+0x24/0x34 May 17 21:29:02 client-5 kernel: [] do_futex+0x25e/0xcf3 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] __sched_text_start+0xf6/0xbd6 May 17 21:29:02 client-5 kernel: [] sys_futex+0x10a/0x12b May 17 21:29:02 client-5 kernel: [] system_call+0x7e/0x83 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: audispd S ffff810001004420 0 2998 2995 2997 (NOTLB) May 17 21:29:02 client-5 kernel: ffff810332333b38 0000000000000082 ffff8102fc0f2400 000000000000000a May 17 21:29:02 client-5 kernel: ffff81033fc017e0 000000000000000a ffff81033fc017e0 ffffffff80308b60 May 17 21:29:02 client-5 kernel: 000042764c9c7b35 00000000000016a0 ffff81033fc019c8 000000002c352e34 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] unix_poll+0x0/0x99 May 17 21:29:02 client-5 kernel: [] schedule_timeout+0x8a/0xad May 17 21:29:02 client-5 kernel: [] process_timeout+0x0/0x5 May 17 21:29:02 client-5 kernel: [] do_sys_poll+0x277/0x360 May 17 21:29:02 client-5 kernel: [] __pollwait+0x0/0xe2 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] try_to_wake_up+0x472/0x484 May 17 21:29:02 client-5 kernel: [] sock_readv+0xb7/0xd1 May 17 21:29:02 client-5 kernel: [] zone_statistics+0x3e/0x6d May 17 21:29:02 client-5 kernel: [] __wake_up_common+0x3e/0x68 May 17 21:29:02 client-5 kernel: [] __wake_up+0x38/0x4f May 17 21:29:02 client-5 kernel: [] wake_futex+0x39/0x43 May 17 21:29:02 client-5 kernel: [] do_futex+0x7b7/0xcf3 May 17 21:29:02 client-5 kernel: [] do_readv_writev+0x26e/0x291 May 17 21:29:02 client-5 kernel: [] sys_futex+0x10a/0x12b May 17 21:29:02 client-5 kernel: [] sys_poll+0x2d/0x34 May 17 21:29:02 client-5 kernel: [] system_call+0x7e/0x83 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: syslogd D ffff81033f5d17e0 0 3017 1 3020 2996 (NOTLB) May 17 21:29:02 client-5 kernel: ffff810329cdda18 0000000000000086 ffff810329e64910 ffffffff880312e8 May 17 21:29:02 client-5 kernel: ffff810329e64910 0000000000000009 ffff81033c46a860 ffff81010b764080 May 17 21:29:02 client-5 kernel: ffff8102e0516680 ffffffff88031cb3 ffff8102e7732570 0000000000000000 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] :jbd:__journal_file_buffer+0x13e/0x243 May 17 21:29:02 client-5 kernel: [] schedule_timeout+0x1e/0xad May 17 21:29:02 client-5 kernel: [] add_wait_queue+0x24/0x34 May 17 21:29:02 client-5 kernel: [] do_select+0x3fe/0x462 May 17 21:29:02 client-5 kernel: [] cfq_set_request+0x370/0x454 May 17 21:29:02 client-5 kernel: [] __pollwait+0x0/0xe2 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] cfq_insert_request+0x3d/0x4fc May 17 21:29:02 client-5 kernel: [] elv_insert+0xfe/0x1c0 May 17 21:29:02 client-5 kernel: [] __make_request+0x3c5/0x414 May 17 21:29:02 client-5 kernel: [] generic_make_request+0x211/0x228 May 17 21:29:02 client-5 kernel: [] enqueue_task+0x41/0x56 May 17 21:29:02 client-5 kernel: [] dequeue_task+0x18/0x37 May 17 21:29:02 client-5 kernel: [] thread_return+0x62/0xfe May 17 21:29:02 client-5 kernel: [] find_get_pages_tag+0x34/0x89 May 17 21:29:02 client-5 kernel: [] :jbd:log_wait_commit+0xa3/0xf5 May 17 21:29:02 client-5 kernel: [] autoremove_wake_function+0x0/0x2e May 17 21:29:02 client-5 kernel: [] :jbd:journal_stop+0x1cf/0x1ff May 17 21:29:02 client-5 kernel: [] find_get_pages_tag+0x34/0x89 May 17 21:29:02 client-5 kernel: [] pagevec_lookup_tag+0x1a/0x21 May 17 21:29:02 client-5 kernel: [] wait_on_page_writeback_range+0xd6/0x12e May 17 21:29:02 client-5 kernel: [] sync_inode+0x24/0x33 May 17 21:29:02 client-5 kernel: [] sys_select+0xc1/0x17c May 17 21:29:02 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: klogd R running task 0 3020 1 3083 3017 (NOTLB) May 17 21:29:02 client-5 kernel: ffff81033127be68 0000000000000086 ffff810300000000 ffff81033127bdd8 May 17 21:29:02 client-5 kernel: 0000000000000001 000000000000000a ffff81033aabe040 ffff81010b770100 May 17 21:29:02 client-5 kernel: 0000427839806061 0000000000001f6b ffff81033aabe228 000000038df05920 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] do_sync_write+0xc7/0x104 May 17 21:29:02 client-5 kernel: [] do_syslog+0x11d/0x3aa May 17 21:29:02 client-5 kernel: [] autoremove_wake_function+0x0/0x2e May 17 21:29:02 client-5 kernel: [] kmsg_read+0x3a/0x44 May 17 21:29:02 client-5 kernel: [] vfs_read+0xcb/0x171 May 17 21:29:02 client-5 kernel: [] sys_read+0x45/0x6e May 17 21:29:02 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: kondemand/0 S ffff810001004420 0 3068 123 3070 2471 (L-TLB) May 17 21:29:02 client-5 kernel: ffff810329bb7e70 0000000000000046 ffff810001007ea0 0000000000000000 May 17 21:29:02 client-5 kernel: 0000000300000000 000000000000000a ffff81033add9040 ffffffff80308b60 May 17 21:29:02 client-5 kernel: 0000427675f7f1c7 0000000000000aa5 ffff81033add9228 000000008002e23f May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:29:02 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: kondemand/1 S ffff81000100caa0 0 3070 123 3071 3068 (L-TLB) May 17 21:29:02 client-5 kernel: ffff810329bbbe70 0000000000000046 ffff810001010520 0000000000000000 May 17 21:29:02 client-5 kernel: 0000000300000000 000000000000000a ffff81033b27c080 ffff81010b70e100 May 17 21:29:02 client-5 kernel: 0000427675f80fce 0000000000000c37 ffff81033b27c268 000000018002e23f May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:29:02 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: kondemand/2 S ffff810001015120 0 3071 123 3072 3070 (L-TLB) May 17 21:29:02 client-5 kernel: ffff810329bbde70 0000000000000046 ffff810001018ba0 0000000000000000 May 17 21:29:02 client-5 kernel: 0000000300000000 000000000000000a ffff81033b0aa7e0 ffff81010b764080 May 17 21:29:02 client-5 kernel: 0000427675f813cd 0000000000000c5c ffff81033b0aa9c8 000000028002e23f May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:29:02 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: kondemand/3 S ffff81000101d7a0 0 3072 123 3128 3071 (L-TLB) May 17 21:29:02 client-5 kernel: ffff810329bbfe70 0000000000000046 ffff810001021220 0000000000000000 May 17 21:29:02 client-5 kernel: 0000000300000000 000000000000000a ffff81033f5c1100 ffff81010b770100 May 17 21:29:02 client-5 kernel: 0000427675f8124b 0000000000000d25 ffff81033f5c12e8 000000038002e23f May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:29:02 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: irqbalance S ffff81000100caa0 0 3083 1 3101 3020 (NOTLB) May 17 21:29:02 client-5 kernel: ffff810329bb5eb8 0000000000000086 ffff810329bb5e58 ffff81000100eb80 May 17 21:29:02 client-5 kernel: 00000002540279d8 0000000000000007 ffff81033a3bf860 ffff81010b70e100 May 17 21:29:02 client-5 kernel: 0000427665ce0cc9 000000000005e0e1 ffff81033a3bfa48 000000012725ca6f May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] enqueue_hrtimer+0x55/0x70 May 17 21:29:02 client-5 kernel: [] do_nanosleep+0x3f/0x70 May 17 21:29:02 client-5 kernel: [] hrtimer_nanosleep+0x58/0x118 May 17 21:29:02 client-5 kernel: [] hrtimer_wakeup+0x0/0x22 May 17 21:29:02 client-5 kernel: [] sys_nanosleep+0x4c/0x62 May 17 21:29:02 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: portmap S ffff810001004420 0 3101 1 3141 3083 (NOTLB) May 17 21:29:02 client-5 kernel: ffff810329bc7b38 0000000000000082 ffff810327d997e0 ffff810327c15780 May 17 21:29:02 client-5 kernel: ffff810327d997c0 000000000000000a ffff81033b49b0c0 ffffffff80308b60 May 17 21:29:02 client-5 kernel: 0000422b725e1132 00000000000061f3 ffff81033b49b2a8 000000008024db4e May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] schedule_timeout+0x1e/0xad May 17 21:29:02 client-5 kernel: [] add_wait_queue+0x24/0x34 May 17 21:29:02 client-5 kernel: [] tcp_poll+0x1c/0x132 May 17 21:29:02 client-5 kernel: [] do_sys_poll+0x277/0x360 May 17 21:29:02 client-5 kernel: [] __pollwait+0x0/0xe2 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] autoremove_wake_function+0x0/0x2e May 17 21:29:02 client-5 kernel: [] zone_statistics+0x3e/0x6d May 17 21:29:02 client-5 kernel: [] sys_sendmsg+0x217/0x28a May 17 21:29:02 client-5 kernel: [] audit_syscall_entry+0x180/0x1b3 May 17 21:29:02 client-5 kernel: [] sys_poll+0x2d/0x34 May 17 21:29:02 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: rpciod/0 S 0000000000000282 0 3128 123 3129 3072 (L-TLB) May 17 21:29:02 client-5 kernel: ffff8103295e5e70 0000000000000046 0000001f8250ec9a 0000000000000003 May 17 21:29:02 client-5 kernel: ffff81033fc12040 0000000000000006 ffff81033fc12040 ffff81033fbbb040 May 17 21:29:02 client-5 kernel: 0000001f8251052f 00000000000004d1 ffff81033fc12228 0000000000000000 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] thread_return+0x62/0xfe May 17 21:29:02 client-5 kernel: [] recalc_sigpending_and_wake+0x9/0x1a May 17 21:29:02 client-5 kernel: [] do_sigaction+0x14d/0x199 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:29:02 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: rpciod/1 S ffff81033b5957a0 0 3129 123 3130 3128 (L-TLB) May 17 21:29:02 client-5 kernel: ffff8103295e7e70 0000000000000046 ffff810191bda530 ffff810191bda528 May 17 21:29:02 client-5 kernel: 0000000300000000 000000000000000a ffff81033fbc1860 ffff81033b5957a0 May 17 21:29:02 client-5 kernel: 0000422b725d6e68 0000000000000d1e ffff81033fbc1a48 000000018002e23f May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] :sunrpc:xs_udp_connect_worker+0x0/0x188 May 17 21:29:02 client-5 kernel: [] run_workqueue+0xba/0xe4 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:29:02 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: rpciod/2 S ffff81031f9dc7e0 0 3130 123 3131 3129 (L-TLB) May 17 21:29:02 client-5 kernel: ffff8103295e9e70 0000000000000046 ffff810315d6e530 ffff810315d6e528 May 17 21:29:02 client-5 kernel: 0000000300000000 000000000000000a ffff81033c1a3040 ffff81031f9dc7e0 May 17 21:29:02 client-5 kernel: 00002b7d34bd07a9 0000000000001084 ffff81033c1a3228 000000028002e23f May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] :sunrpc:xs_udp_connect_worker+0x0/0x188 May 17 21:29:02 client-5 kernel: [] run_workqueue+0xba/0xe4 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:29:02 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: rpciod/3 S ffff81000101d7a0 0 3131 123 932 3130 (L-TLB) May 17 21:29:02 client-5 kernel: ffff8103295ebe70 0000000000000046 ffff8102f96f87d8 ffff8102f96f8580 May 17 21:29:02 client-5 kernel: 0000000300000000 000000000000000a ffff81033c54a7a0 ffff81010b770100 May 17 21:29:02 client-5 kernel: 00002b79f6371f67 0000000000006218 ffff81033c54a988 000000038002e23f May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:29:02 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: rpc.statd S ffff810001015120 0 3141 1 3163 3101 (NOTLB) May 17 21:29:02 client-5 kernel: ffff8103295eda18 0000000000000086 0000000000000001 ffffffff80150790 May 17 21:29:02 client-5 kernel: 00000000000280d2 0000000000000008 ffff81033ae7b080 ffff81010b764080 May 17 21:29:02 client-5 kernel: 0000001f864d63ed 0000000000002d35 ffff81033ae7b268 000000028008bae0 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] __next_cpu+0x19/0x28 May 17 21:29:02 client-5 kernel: [] schedule_timeout+0x1e/0xad May 17 21:29:02 client-5 kernel: [] add_wait_queue+0x24/0x34 May 17 21:29:02 client-5 kernel: [] tcp_poll+0x1c/0x132 May 17 21:29:02 client-5 kernel: [] do_select+0x3fe/0x462 May 17 21:29:02 client-5 kernel: [] __pollwait+0x0/0xe2 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 last message repeated 2 times May 17 21:29:02 client-5 kernel: [] autoremove_wake_function+0x0/0x2e May 17 21:29:02 client-5 kernel: [] skb_queue_tail+0x17/0x3e May 17 21:29:02 client-5 kernel: [] sock_def_readable+0x10/0x62 May 17 21:29:02 client-5 kernel: [] unix_dgram_sendmsg+0x44e/0x4e0 May 17 21:29:02 client-5 kernel: [] sock_sendmsg+0xf8/0x14a May 17 21:29:02 client-5 kernel: [] dput+0x2c/0x114 May 17 21:29:02 client-5 kernel: [] autoremove_wake_function+0x0/0x2e May 17 21:29:02 client-5 kernel: [] core_sys_select+0x1bc/0x265 May 17 21:29:02 client-5 kernel: [] _atomic_dec_and_lock+0x39/0x57 May 17 21:29:02 client-5 kernel: [] mntput_no_expire+0x19/0x89 May 17 21:29:02 client-5 kernel: [] sys_sendto+0x11c/0x14f May 17 21:29:02 client-5 kernel: [] sys_select+0xc1/0x17c May 17 21:29:02 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: rpc.idmapd S ffff81033b49b0c0 0 3163 1 3176 3141 (NOTLB) May 17 21:29:02 client-5 kernel: ffff81032946fe78 0000000000000082 ffff810329558380 ffff81032946fc38 May 17 21:29:02 client-5 kernel: ffff81032946fee8 000000000000000a ffff81033b686040 ffff81033b49b0c0 May 17 21:29:02 client-5 kernel: 0000422b72597297 000000000001b58a ffff81033b686228 0000000000000246 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] d_alloc+0x174/0x1a9 May 17 21:29:02 client-5 kernel: [] schedule_timeout+0x1e/0xad May 17 21:29:02 client-5 kernel: [] sys_epoll_wait+0x184/0x3f9 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: dbus-daemon S ffff810001004420 0 3176 1 3184 3163 (NOTLB) May 17 21:29:02 client-5 kernel: ffff810329651b38 0000000000000082 0000000000000000 0000000000000000 May 17 21:29:02 client-5 kernel: 0000000000000008 000000000000000a ffff81033fbbb040 ffffffff80308b60 May 17 21:29:02 client-5 kernel: 00004181495d616e 0000000000000c8a ffff81033fbbb228 0000000000000001 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] schedule_timeout+0x1e/0xad May 17 21:29:02 client-5 kernel: [] add_wait_queue+0x24/0x34 May 17 21:29:02 client-5 kernel: [] do_sys_poll+0x277/0x360 May 17 21:29:02 client-5 kernel: [] __pollwait+0x0/0xe2 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 last message repeated 4 times May 17 21:29:02 client-5 kernel: [] autoremove_wake_function+0x0/0x2e May 17 21:29:02 client-5 kernel: [] sock_aio_read+0x4f/0x5e May 17 21:29:02 client-5 kernel: [] mntput_no_expire+0x19/0x89 May 17 21:29:02 client-5 kernel: [] do_sync_read+0xc7/0x104 May 17 21:29:02 client-5 kernel: [] do_readv_writev+0x26e/0x291 May 17 21:29:02 client-5 kernel: [] audit_syscall_entry+0x180/0x1b3 May 17 21:29:02 client-5 kernel: [] sys_poll+0x2d/0x34 May 17 21:29:02 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: hcid S ffff81033fbbb040 0 3184 1 3190 3176 (NOTLB) May 17 21:29:02 client-5 kernel: ffff8103296b1b38 0000000000000086 0000000000000000 0000000000000000 May 17 21:29:02 client-5 kernel: ffff81033c46a860 0000000000000001 ffff81033b304100 ffff81033fbbb040 May 17 21:29:02 client-5 kernel: 0000001f966db063 0000000000024fd5 ffff81033b3042e8 0000000300000000 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] try_to_wake_up+0x472/0x484 May 17 21:29:02 client-5 kernel: [] schedule_timeout+0x1e/0xad May 17 21:29:02 client-5 kernel: [] add_wait_queue+0x24/0x34 May 17 21:29:02 client-5 kernel: [] do_sys_poll+0x277/0x360 May 17 21:29:02 client-5 kernel: [] __pollwait+0x0/0xe2 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] sock_sendmsg+0xf8/0x14a May 17 21:29:02 client-5 kernel: [] dput+0x2c/0x114 May 17 21:29:02 client-5 kernel: [] find_get_page+0x21/0x51 May 17 21:29:02 client-5 kernel: [] filemap_nopage+0x193/0x360 May 17 21:29:02 client-5 kernel: [] __handle_mm_fault+0x5f3/0x1039 May 17 21:29:02 client-5 kernel: [] _atomic_dec_and_lock+0x39/0x57 May 17 21:29:02 client-5 kernel: [] do_page_fault+0x4fe/0x874 May 17 21:29:02 client-5 kernel: [] sock_ioctl+0x1c1/0x1e5 May 17 21:29:02 client-5 kernel: [] do_ioctl+0x21/0x6b May 17 21:29:02 client-5 kernel: [] vfs_ioctl+0x457/0x4b9 May 17 21:29:02 client-5 kernel: [] audit_syscall_entry+0x180/0x1b3 May 17 21:29:02 client-5 kernel: [] sys_poll+0x2d/0x34 May 17 21:29:02 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: sdpd S ffff810001015120 0 3190 1 3221 3184 (NOTLB) May 17 21:29:02 client-5 kernel: ffff8103296db9c8 0000000000000082 ffffffff800cc235 ffff81010b70b0c0 May 17 21:29:02 client-5 kernel: 0000000000000286 0000000000000001 ffff81033b0aa080 ffff81010b764080 May 17 21:29:02 client-5 kernel: 0000001f96f336d4 0000000000003a31 ffff81033b0aa268 0000000200000001 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] zone_statistics+0x3e/0x6d May 17 21:29:02 client-5 kernel: [] schedule_timeout+0x8a/0xad May 17 21:29:02 client-5 kernel: [] process_timeout+0x0/0x5 May 17 21:29:02 client-5 kernel: [] do_select+0x3fe/0x462 May 17 21:29:02 client-5 kernel: [] __pollwait+0x0/0xe2 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] __wake_up_common+0x3e/0x68 May 17 21:29:02 client-5 kernel: [] __wake_up+0x38/0x4f May 17 21:29:02 client-5 kernel: [] sock_def_readable+0x37/0x62 May 17 21:29:02 client-5 kernel: [] unix_dgram_sendmsg+0x44e/0x4e0 May 17 21:29:02 client-5 kernel: [] touch_atime+0x67/0xaa May 17 21:29:02 client-5 kernel: [] sock_sendmsg+0xf8/0x14a May 17 21:29:02 client-5 kernel: [] core_sys_select+0x1bc/0x265 May 17 21:29:02 client-5 kernel: [] __handle_mm_fault+0x5f3/0x1039 May 17 21:29:02 client-5 kernel: [] _atomic_dec_and_lock+0x39/0x57 May 17 21:29:02 client-5 kernel: [] __up_read+0x19/0x7f May 17 21:29:02 client-5 kernel: [] do_page_fault+0x4fe/0x874 May 17 21:29:02 client-5 kernel: [] sys_pselect7+0x114/0x210 May 17 21:29:02 client-5 kernel: [] audit_syscall_entry+0x180/0x1b3 May 17 21:29:02 client-5 kernel: [] sys_pselect6+0x54/0x61 May 17 21:29:02 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: krfcommd S ffff810001004420 0 3221 1 3256 3190 (L-TLB) May 17 21:29:02 client-5 kernel: ffff8103297e7e40 0000000000000046 ffff810329cf6340 0000000000000000 May 17 21:29:02 client-5 kernel: 00000000ffffffea 0000000000000006 ffff81033b657100 ffffffff80308b60 May 17 21:29:02 client-5 kernel: 0000001f975fed7a 0000000000000f4d ffff81033b6572e8 00000000800335e5 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] :rfcomm:rfcomm_run+0x1a0/0x1207 May 17 21:29:02 client-5 kernel: [] audit_syscall_exit+0x336/0x362 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] :rfcomm:rfcomm_run+0x0/0x1207 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: pcscd S 0000000000000020 0 3256 1 3265 3221 (NOTLB) May 17 21:29:02 client-5 kernel: ffff810329bd5a18 0000000000000082 ffff81033e52e288 ffff81032a9f9d20 May 17 21:29:02 client-5 kernel: ffff81033e548000 0000000000000001 ffff81033c5167a0 ffff81033aca8820 May 17 21:29:02 client-5 kernel: 0000001fa70bab0b 0000000000057933 ffff81033c516988 000000003e52e288 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] elv_insert+0xfe/0x1c0 May 17 21:29:02 client-5 kernel: [] __make_request+0x3c5/0x414 May 17 21:29:02 client-5 kernel: [] schedule_timeout+0x1e/0xad May 17 21:29:02 client-5 kernel: [] add_wait_queue+0x24/0x34 May 17 21:29:02 client-5 kernel: [] do_select+0x3fe/0x462 May 17 21:29:02 client-5 kernel: [] __pollwait+0x0/0xe2 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] :jbd:journal_dirty_metadata+0x1b5/0x1e3 May 17 21:29:02 client-5 kernel: [] __do_page_cache_readahead+0x82/0x179 May 17 21:29:02 client-5 kernel: [] current_fs_time+0x3b/0x40 May 17 21:29:02 client-5 kernel: [] blockable_page_cache_readahead+0x53/0xb2 May 17 21:29:02 client-5 kernel: [] touch_atime+0x67/0xaa May 17 21:29:02 client-5 kernel: [] zone_statistics+0x3e/0x6d May 17 21:29:02 client-5 kernel: [] get_page_from_freelist+0x378/0x43a May 17 21:29:02 client-5 kernel: [] zone_statistics+0x3e/0x6d May 17 21:29:02 client-5 kernel: [] get_page_from_freelist+0x378/0x43a May 17 21:29:02 client-5 kernel: [] __rmqueue+0x91/0xc6 May 17 21:29:02 client-5 kernel: [] __bitmap_weight+0x65/0x76 May 17 21:29:02 client-5 kernel: [] core_sys_select+0x1bc/0x265 May 17 21:29:02 client-5 kernel: [] attach_pid+0x8c/0xb9 May 17 21:29:02 client-5 kernel: [] enqueue_task+0x41/0x56 May 17 21:29:02 client-5 kernel: [] __activate_task+0x56/0x6d May 17 21:29:02 client-5 kernel: [] wake_up_new_task+0x287/0x296 May 17 21:29:02 client-5 kernel: [] sys_select+0xc1/0x17c May 17 21:29:02 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: pcscd S ffff81000101d7a0 0 3276 1 3273 3265 (NOTLB) May 17 21:29:02 client-5 kernel: ffff810329241eb8 0000000000000082 ffff810329241e58 ffff81000101f880 May 17 21:29:02 client-5 kernel: 000000003b9aca00 0000000000000007 ffff81033aca8820 ffff81010b770100 May 17 21:29:02 client-5 kernel: 000042766c952925 0000000000048e75 ffff81033aca8a08 0000000315881477 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] enqueue_hrtimer+0x55/0x70 May 17 21:29:02 client-5 kernel: [] do_nanosleep+0x3f/0x70 May 17 21:29:02 client-5 kernel: [] hrtimer_nanosleep+0x58/0x118 May 17 21:29:02 client-5 kernel: [] hrtimer_wakeup+0x0/0x22 May 17 21:29:02 client-5 kernel: [] sys_nanosleep+0x4c/0x62 May 17 21:29:02 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: acpid S 0000000000000000 0 3265 1 3276 3256 (NOTLB) May 17 21:29:02 client-5 kernel: ffff81032940db38 0000000000000086 ffff81033eb1c800 ffffffff8804d916 May 17 21:29:02 client-5 kernel: ffff81032940dad8 0000000000000002 ffff81033add97a0 ffff81033a91a040 May 17 21:29:02 client-5 kernel: 0000001fb16e00c1 000000000000f49a ffff81033add9988 000000030b7752d8 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] :ext3:ext3_mark_iloc_dirty+0x2f8/0x360 May 17 21:29:02 client-5 kernel: [] schedule_timeout+0x1e/0xad May 17 21:29:02 client-5 kernel: [] add_wait_queue+0x24/0x34 May 17 21:29:02 client-5 kernel: [] do_sys_poll+0x277/0x360 May 17 21:29:02 client-5 kernel: [] __pollwait+0x0/0xe2 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] __generic_file_aio_write_nolock+0x369/0x3b6 May 17 21:29:02 client-5 kernel: [] generic_file_aio_write+0x70/0xc1 May 17 21:29:02 client-5 kernel: [] :ext3:ext3_file_write+0x16/0x91 May 17 21:29:02 client-5 kernel: [] do_sync_write+0xc7/0x104 May 17 21:29:02 client-5 kernel: [] sock_getsockopt+0x32b/0x34d May 17 21:29:02 client-5 kernel: [] autoremove_wake_function+0x0/0x2e May 17 21:29:02 client-5 kernel: [] sys_accept+0x1a4/0x1ea May 17 21:29:02 client-5 kernel: [] audit_syscall_entry+0x180/0x1b3 May 17 21:29:02 client-5 kernel: [] sys_poll+0x2d/0x34 May 17 21:29:02 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: hald S ffff81000101d7a0 0 3273 1 3274 3315 3276 (NOTLB) May 17 21:29:02 client-5 kernel: ffff8103290b7b38 0000000000000082 0000000000000001 0000000000000000 May 17 21:29:02 client-5 kernel: 0000000000000000 000000000000000a ffff81033c1a37a0 ffff81010b770100 May 17 21:29:02 client-5 kernel: 00004273d73c2d90 00000000000295ec ffff81033c1a3988 0000000300000002 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] get_pid_task+0x5/0xf May 17 21:29:02 client-5 kernel: [] schedule_timeout+0x8a/0xad May 17 21:29:02 client-5 kernel: [] process_timeout+0x0/0x5 May 17 21:29:02 client-5 kernel: [] do_sys_poll+0x277/0x360 May 17 21:29:02 client-5 kernel: [] __pollwait+0x0/0xe2 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 last message repeated 8 times May 17 21:29:02 client-5 kernel: [] free_pages_and_swap_cache+0x67/0x7e May 17 21:29:02 client-5 kernel: [] audit_syscall_entry+0x180/0x1b3 May 17 21:29:02 client-5 kernel: [] sys_poll+0x2d/0x34 May 17 21:29:02 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: hald-runner S ffff810001015120 0 3274 3273 3282 (NOTLB) May 17 21:29:02 client-5 kernel: ffff8103290f7b38 0000000000000082 ffff8103290f7aa8 ffffffff8008c9a2 May 17 21:29:02 client-5 kernel: 0000001fd62a30aa 0000000000000007 ffff81033acc1080 ffff81010b764080 May 17 21:29:02 client-5 kernel: 0000001fd63ab01c 00000000000060c9 ffff81033acc1268 0000000200000001 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] __activate_task+0x56/0x6d May 17 21:29:02 client-5 kernel: [] schedule_timeout+0x1e/0xad May 17 21:29:02 client-5 kernel: [] add_wait_queue+0x24/0x34 May 17 21:29:02 client-5 kernel: [] do_sys_poll+0x277/0x360 May 17 21:29:02 client-5 kernel: [] __pollwait+0x0/0xe2 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] get_page_from_freelist+0x378/0x43a May 17 21:29:02 client-5 kernel: [] do_sock_write+0xc6/0x102 May 17 21:29:02 client-5 kernel: [] sock_writev+0xb7/0xd1 May 17 21:29:02 client-5 kernel: [] flush_tlb_page+0xac/0xda May 17 21:29:02 client-5 kernel: [] do_wp_page+0x3fd/0x902 May 17 21:29:02 client-5 kernel: [] __handle_mm_fault+0xf6b/0x1039 May 17 21:29:02 client-5 kernel: [] math_state_restore+0x23/0x4c May 17 21:29:02 client-5 kernel: [] error_exit+0x0/0x84 May 17 21:29:02 client-5 kernel: [] __up_read+0x19/0x7f May 17 21:29:02 client-5 kernel: [] do_page_fault+0x4fe/0x874 May 17 21:29:02 client-5 kernel: [] thread_return+0x7a/0xfe May 17 21:29:02 client-5 kernel: [] release_task+0x3a5/0x3cb May 17 21:29:02 client-5 kernel: [] audit_syscall_entry+0x180/0x1b3 May 17 21:29:02 client-5 kernel: [] sys_poll+0x2d/0x34 May 17 21:29:02 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: hald-addon-ac S ffff81000101d7a0 0 3282 3274 3289 (NOTLB) May 17 21:29:02 client-5 kernel: ffff810328d95bd8 0000000000000086 ffff810328d95b48 0000000000000092 May 17 21:29:02 client-5 kernel: 00000000fffc9956 0000000000000001 ffff81033a91a040 ffff81010b770100 May 17 21:29:02 client-5 kernel: 0000001fb16e2d01 0000000000002c40 ffff81033a91a228 00000003802ac375 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] schedule_timeout+0x1e/0xad May 17 21:29:02 client-5 kernel: [] prepare_to_wait+0x34/0x61 May 17 21:29:02 client-5 kernel: [] unix_stream_recvmsg+0x245/0x56f May 17 21:29:02 client-5 kernel: [] autoremove_wake_function+0x0/0x2e May 17 21:29:02 client-5 kernel: [] dput+0x2c/0x114 May 17 21:29:02 client-5 kernel: [] enqueue_task+0x41/0x56 May 17 21:29:02 client-5 kernel: [] do_sock_read+0xcf/0x110 May 17 21:29:02 client-5 kernel: [] sock_aio_read+0x4f/0x5e May 17 21:29:02 client-5 kernel: [] do_sync_read+0xc7/0x104 May 17 21:29:02 client-5 kernel: [] autoremove_wake_function+0x0/0x2e May 17 21:29:02 client-5 kernel: [] do_mmap_pgoff+0x615/0x780 May 17 21:29:02 client-5 kernel: [] vfs_read+0xde/0x171 May 17 21:29:02 client-5 kernel: [] sys_read+0x45/0x6e May 17 21:29:02 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: hald-addon-ke S ffff81000100caa0 0 3289 3274 3282 (NOTLB) May 17 21:29:02 client-5 kernel: ffff810328dd7ea8 0000000000000086 0000000000000000 00002ab9a9b24000 May 17 21:29:02 client-5 kernel: 00002ab9a9925000 0000000000000001 ffff81033b724100 ffff81010b70e100 May 17 21:29:02 client-5 kernel: 0000001fc3753e10 000000000004120e ffff81033b7242e8 00000001a9b25000 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] vma_merge+0x141/0x1ee May 17 21:29:02 client-5 kernel: [] evdev_read+0x112/0x247 May 17 21:29:02 client-5 kernel: [] autoremove_wake_function+0x0/0x2e May 17 21:29:02 client-5 kernel: [] vfs_read+0xcb/0x171 May 17 21:29:02 client-5 kernel: [] sys_read+0x45/0x6e May 17 21:29:02 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: hidd S 0000000000000000 0 3315 1 3344 3273 (NOTLB) May 17 21:29:02 client-5 kernel: ffff8103298d1ad8 0000000000000082 00002b5c71e5e000 ffff81033a993b80 May 17 21:29:02 client-5 kernel: ffff810329740c78 0000000000000001 ffff81033c516040 ffff81033ad4f860 May 17 21:29:02 client-5 kernel: 0000001fdc936b05 00000000000159e1 ffff81033c516228 000000000ca5eaa0 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] find_get_page+0x21/0x51 May 17 21:29:02 client-5 kernel: [] schedule_timeout+0x1e/0xad May 17 21:29:02 client-5 kernel: [] add_wait_queue+0x24/0x34 May 17 21:29:02 client-5 kernel: [] do_sys_poll+0x277/0x360 May 17 21:29:02 client-5 kernel: [] __pollwait+0x0/0xe2 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] skb_queue_tail+0x17/0x3e May 17 21:29:02 client-5 kernel: [] sock_def_readable+0x10/0x62 May 17 21:29:02 client-5 kernel: [] unix_dgram_sendmsg+0x44e/0x4e0 May 17 21:29:02 client-5 kernel: [] touch_atime+0x67/0xaa May 17 21:29:02 client-5 kernel: [] sock_sendmsg+0xf8/0x14a May 17 21:29:02 client-5 kernel: [] dput+0x2c/0x114 May 17 21:29:02 client-5 kernel: [] find_get_page+0x21/0x51 May 17 21:29:02 client-5 kernel: [] filemap_nopage+0x193/0x360 May 17 21:29:02 client-5 kernel: [] __handle_mm_fault+0x5f3/0x1039 May 17 21:29:02 client-5 kernel: [] _atomic_dec_and_lock+0x39/0x57 May 17 21:29:02 client-5 kernel: [] __up_read+0x19/0x7f May 17 21:29:02 client-5 kernel: [] do_page_fault+0x4fe/0x874 May 17 21:29:02 client-5 kernel: [] sys_ppoll+0xf1/0x1f2 May 17 21:29:02 client-5 kernel: [] tracesys+0x71/0xe0 May 17 21:29:02 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: automount S ffffffff8006271a 0 3344 1 3345 3315 (NOTLB) May 17 21:29:02 client-5 kernel: ffff810328ebfe58 0000000000000082 0000000000000286 0000000000000286 May 17 21:29:02 client-5 kernel: 0000000000010800 0000000000000005 ffff81033a97d100 ffff81033fc57860 May 17 21:29:02 client-5 kernel: 000000202b7ff6e0 0000000000003e3b ffff81033a97d2e8 0000000300000000 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] schedule_timeout+0x1e/0xad May 17 21:29:02 client-5 kernel: [] recalc_sigpending+0xe/0x25 May 17 21:29:02 client-5 kernel: [] dequeue_signal+0x47/0xcd May 17 21:29:02 client-5 kernel: [] sys_rt_sigtimedwait+0x1ce/0x2b4 May 17 21:29:02 client-5 kernel: [] audit_syscall_entry+0x180/0x1b3 May 17 21:29:02 client-5 kernel: [] tracesys+0x71/0xe0 May 17 21:29:02 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: automount S 0000000000000000 0 3345 1 3346 3344 (NOTLB) May 17 21:29:02 client-5 kernel: ffff810328edfd28 0000000000000082 000000013bb6e088 0000000000000001 May 17 21:29:02 client-5 kernel: 000000013e91c040 000000000000000a ffff81033aace040 ffff8102c8efa7e0 May 17 21:29:02 client-5 kernel: 000042712ccbe5f5 0000000000000c4b ffff81033aace228 0000000100000092 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] lock_timer_base+0x1b/0x3c May 17 21:29:02 client-5 kernel: [] __mod_timer+0x100/0x10f May 17 21:29:02 client-5 kernel: [] schedule_timeout+0x8a/0xad May 17 21:29:02 client-5 kernel: [] process_timeout+0x0/0x5 May 17 21:29:02 client-5 kernel: [] do_futex+0x25e/0xcf3 May 17 21:29:02 client-5 kernel: [] __handle_mm_fault+0xf6b/0x1039 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] sys_futex+0x10a/0x12b May 17 21:29:02 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: automount S ffff81000100caa0 0 3346 1 3349 3345 (NOTLB) May 17 21:29:02 client-5 kernel: ffff810328ee1d28 0000000000000082 0000000000000000 0000000000000000 May 17 21:29:02 client-5 kernel: ffff8102fb7f1df0 000000000000000a ffff81033bb430c0 ffff81010b70e100 May 17 21:29:02 client-5 kernel: 000042764c7e3532 0000000000001d1c ffff81033bb432a8 0000000100000000 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] schedule_timeout+0x8a/0xad May 17 21:29:02 client-5 kernel: [] process_timeout+0x0/0x5 May 17 21:29:02 client-5 kernel: [] do_futex+0x25e/0xcf3 May 17 21:29:02 client-5 kernel: [] __alloc_pages+0x78/0x308 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] wake_up_new_task+0x287/0x296 May 17 21:29:02 client-5 kernel: [] do_fork+0x148/0x1c1 May 17 21:29:02 client-5 kernel: [] sys_futex+0x10a/0x12b May 17 21:29:02 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: automount S ffff810001004420 0 3349 1 3352 3346 (NOTLB) May 17 21:29:02 client-5 kernel: ffff810328835b38 0000000000000082 0000000000000000 0000000000000000 May 17 21:29:02 client-5 kernel: 0000000000000008 0000000000000002 ffff81033acc17e0 ffffffff80308b60 May 17 21:29:02 client-5 kernel: 0000002028e6e22b 00000000000009bd ffff81033acc19c8 0000000000000001 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] schedule_timeout+0x1e/0xad May 17 21:29:02 client-5 kernel: [] add_wait_queue+0x24/0x34 May 17 21:29:02 client-5 kernel: [] pipe_poll+0x2d/0x90 May 17 21:29:02 client-5 kernel: [] do_sys_poll+0x277/0x360 May 17 21:29:02 client-5 kernel: [] __pollwait+0x0/0xe2 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 last message repeated 2 times May 17 21:29:02 client-5 kernel: [] __wake_up_common+0x3e/0x68 May 17 21:29:02 client-5 kernel: [] __wake_up+0x38/0x4f May 17 21:29:02 client-5 kernel: [] wake_futex+0x39/0x43 May 17 21:29:02 client-5 kernel: [] futex_wake+0x95/0xd4 May 17 21:29:02 client-5 kernel: [] do_futex+0x2c2/0xcf3 May 17 21:29:02 client-5 kernel: [] __handle_mm_fault+0xf6b/0x1039 May 17 21:29:02 client-5 kernel: [] do_page_fault+0x4fe/0x874 May 17 21:29:02 client-5 kernel: [] thread_return+0x62/0xfe May 17 21:29:02 client-5 kernel: [] do_ioctl+0x21/0x6b May 17 21:29:02 client-5 kernel: [] audit_syscall_entry+0x180/0x1b3 May 17 21:29:02 client-5 kernel: [] sys_poll+0x2d/0x34 May 17 21:29:02 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: automount S 0000000000000000 0 3352 1 3353 3349 (NOTLB) May 17 21:29:02 client-5 kernel: ffff810328837b38 0000000000000082 0000000000000000 0000000000000000 May 17 21:29:02 client-5 kernel: 0000000000000008 0000000000000004 ffff81033c3277e0 ffff81033aa46820 May 17 21:29:02 client-5 kernel: 0000002028fd07f6 00000000000008d0 ffff81033c3279c8 0000000000000001 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] match_number+0x81/0x8f May 17 21:29:02 client-5 kernel: [] schedule_timeout+0x1e/0xad May 17 21:29:02 client-5 kernel: [] add_wait_queue+0x24/0x34 May 17 21:29:02 client-5 kernel: [] pipe_poll+0x2d/0x90 May 17 21:29:02 client-5 kernel: [] do_sys_poll+0x277/0x360 May 17 21:29:02 client-5 kernel: [] __pollwait+0x0/0xe2 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 last message repeated 2 times May 17 21:29:02 client-5 kernel: [] __wake_up_common+0x3e/0x68 May 17 21:29:02 client-5 kernel: [] __wake_up+0x38/0x4f May 17 21:29:02 client-5 kernel: [] wake_futex+0x39/0x43 May 17 21:29:02 client-5 kernel: [] futex_wake+0x95/0xd4 May 17 21:29:02 client-5 kernel: [] do_futex+0x2c2/0xcf3 May 17 21:29:02 client-5 kernel: [] :autofs4:autofs_dev_ioctl_openmount+0xae/0x12d May 17 21:29:02 client-5 kernel: [] :autofs4:autofs_dev_ioctl_timeout+0x0/0x2c May 17 21:29:02 client-5 kernel: [] :autofs4:autofs_dev_ioctl+0x2dc/0x33f May 17 21:29:02 client-5 kernel: [] thread_return+0x62/0xfe May 17 21:29:02 client-5 kernel: [] do_ioctl+0x21/0x6b May 17 21:29:02 client-5 kernel: [] audit_syscall_entry+0x180/0x1b3 May 17 21:29:02 client-5 kernel: [] sys_poll+0x2d/0x34 May 17 21:29:02 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: automount S ffff810001015120 0 3353 1 3354 3352 (NOTLB) May 17 21:29:02 client-5 kernel: ffff810328857b38 0000000000000082 0000000000000000 0000000000000000 May 17 21:29:02 client-5 kernel: ffff810001003200 0000000000000007 ffff81033aa46820 ffff81010b764080 May 17 21:29:02 client-5 kernel: 0000422b6e4080a7 0000000000001b3e ffff81033aa46a08 0000000228857b08 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] schedule_timeout+0x1e/0xad May 17 21:29:02 client-5 kernel: [] add_wait_queue+0x24/0x34 May 17 21:29:02 client-5 kernel: [] pipe_poll+0x2d/0x90 May 17 21:29:02 client-5 kernel: [] do_sys_poll+0x277/0x360 May 17 21:29:02 client-5 kernel: [] __pollwait+0x0/0xe2 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 last message repeated 2 times May 17 21:29:02 client-5 kernel: [] futex_wake+0x50/0xd4 May 17 21:29:02 client-5 kernel: [] do_futex+0x2c2/0xcf3 May 17 21:29:02 client-5 kernel: [] __handle_mm_fault+0xf6b/0x1039 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] do_fork+0x148/0x1c1 May 17 21:29:02 client-5 kernel: [] audit_syscall_entry+0x180/0x1b3 May 17 21:29:02 client-5 kernel: [] sys_poll+0x2d/0x34 May 17 21:29:02 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: automount S ffff81000101d7a0 0 3354 1 3365 3353 (NOTLB) May 17 21:29:02 client-5 kernel: ffff810328897b38 0000000000000082 0000000000000000 0000000000000000 May 17 21:29:02 client-5 kernel: 0000000000000008 0000000000000004 ffff81033fc57860 ffff81010b770100 May 17 21:29:02 client-5 kernel: 000000202b801433 0000000000001d53 ffff81033fc57a48 0000000300000001 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] schedule_timeout+0x1e/0xad May 17 21:29:02 client-5 kernel: [] add_wait_queue+0x24/0x34 May 17 21:29:02 client-5 kernel: [] pipe_poll+0x2d/0x90 May 17 21:29:02 client-5 kernel: [] do_sys_poll+0x277/0x360 May 17 21:29:02 client-5 kernel: [] __pollwait+0x0/0xe2 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 last message repeated 2 times May 17 21:29:02 client-5 kernel: [] __wake_up_common+0x3e/0x68 May 17 21:29:02 client-5 kernel: [] __wake_up+0x38/0x4f May 17 21:29:02 client-5 kernel: [] wake_futex+0x39/0x43 May 17 21:29:02 client-5 kernel: [] futex_wake+0x95/0xd4 May 17 21:29:02 client-5 kernel: [] do_futex+0x2c2/0xcf3 May 17 21:29:02 client-5 kernel: [] :autofs4:autofs_dev_ioctl_openmount+0xae/0x12d May 17 21:29:02 client-5 kernel: [] :autofs4:autofs_dev_ioctl_timeout+0x0/0x2c May 17 21:29:02 client-5 kernel: [] :autofs4:autofs_dev_ioctl+0x2dc/0x33f May 17 21:29:02 client-5 kernel: [] thread_return+0x62/0xfe May 17 21:29:02 client-5 kernel: [] do_ioctl+0x21/0x6b May 17 21:29:02 client-5 kernel: [] audit_syscall_entry+0x180/0x1b3 May 17 21:29:02 client-5 kernel: [] sys_poll+0x2d/0x34 May 17 21:29:02 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: sssd S ffff810001004420 0 3365 1 3373 3379 3354 (NOTLB) May 17 21:29:02 client-5 kernel: ffff810328ac9e78 0000000000000086 0000000000000001 0000000000000000 May 17 21:29:02 client-5 kernel: 0000000000000000 000000000000000a ffff81033ad4f860 ffffffff80308b60 May 17 21:29:02 client-5 kernel: 0000427665c83798 00000000000029c6 ffff81033ad4fa48 000000008000ceb5 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] schedule_timeout+0x8a/0xad May 17 21:29:02 client-5 kernel: [] process_timeout+0x0/0x5 May 17 21:29:02 client-5 kernel: [] sys_epoll_wait+0x184/0x3f9 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: sssd_be S ffff810001015120 0 3373 3365 3376 (NOTLB) May 17 21:29:02 client-5 kernel: ffff810328ba1e78 0000000000000082 ffff810328ba1e18 ffff81033f757ac0 May 17 21:29:02 client-5 kernel: 000000000000001c 000000000000000a ffff81033a91a7a0 ffff81010b764080 May 17 21:29:02 client-5 kernel: 000042762a4bd8ab 0000000000000ebe ffff81033a91a988 00000002800182c3 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] kmem_cache_alloc+0x6e/0x76 May 17 21:29:02 client-5 kernel: [] schedule_timeout+0x8a/0xad May 17 21:29:02 client-5 kernel: [] process_timeout+0x0/0x5 May 17 21:29:02 client-5 kernel: [] sys_epoll_wait+0x184/0x3f9 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: sssd_nss S ffff81033fcc0100 0 3376 3365 3377 3373 (NOTLB) May 17 21:29:02 client-5 kernel: ffff81032851fe78 0000000000000082 ffff81032851fe18 0000000000000000 May 17 21:29:02 client-5 kernel: 0000000000000000 000000000000000a ffff81033a97d860 ffff81033fcc0100 May 17 21:29:02 client-5 kernel: 000042762a4bd469 000000000000052b ffff81033a97da48 000000008000ceb5 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] lock_timer_base+0x1b/0x3c May 17 21:29:02 client-5 kernel: [] __mod_timer+0x100/0x10f May 17 21:29:02 client-5 kernel: [] schedule_timeout+0x8a/0xad May 17 21:29:02 client-5 kernel: [] process_timeout+0x0/0x5 May 17 21:29:02 client-5 kernel: [] sys_epoll_wait+0x184/0x3f9 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: sssd_pam S ffff810001004420 0 3377 3365 3376 (NOTLB) May 17 21:29:02 client-5 kernel: ffff810328521e78 0000000000000086 ffff810328521e18 0000000000000000 May 17 21:29:02 client-5 kernel: 0000000000000000 000000000000000a ffff81033fcc0100 ffffffff80308b60 May 17 21:29:02 client-5 kernel: 000042762a4bd863 00000000000003fa ffff81033fcc02e8 000000008000ceb5 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] schedule_timeout+0x8a/0xad May 17 21:29:02 client-5 kernel: [] process_timeout+0x0/0x5 May 17 21:29:02 client-5 kernel: [] sys_epoll_wait+0x184/0x3f9 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: sshd S ffff810001015120 0 3379 1 3391 3365 (NOTLB) May 17 21:29:02 client-5 kernel: ffff81032871fa18 0000000000000082 0000000000000000 ffff810000021c10 May 17 21:29:02 client-5 kernel: 00000000000280d2 0000000000000006 ffff81033fc0f100 ffff81010b764080 May 17 21:29:02 client-5 kernel: 0000015917bf9cd4 00000000000085df ffff81033fc0f2e8 00000002000280d2 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] schedule_timeout+0x1e/0xad May 17 21:29:02 client-5 kernel: [] add_wait_queue+0x24/0x34 May 17 21:29:02 client-5 kernel: [] tcp_poll+0x1c/0x132 May 17 21:29:02 client-5 kernel: [] do_select+0x3fe/0x462 May 17 21:29:02 client-5 kernel: [] __pollwait+0x0/0xe2 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 last message repeated 2 times May 17 21:29:02 client-5 kernel: [] __alloc_skb+0x77/0x12d May 17 21:29:02 client-5 kernel: [] sock_alloc_send_pskb+0x7d/0x282 May 17 21:29:02 client-5 kernel: [] memcpy_fromiovec+0x36/0x66 May 17 21:29:02 client-5 kernel: [] skb_queue_tail+0x17/0x3e May 17 21:29:02 client-5 kernel: [] sock_def_readable+0x10/0x62 May 17 21:29:02 client-5 kernel: [] unix_stream_sendmsg+0x281/0x346 May 17 21:29:02 client-5 kernel: [] flush_tlb_page+0xac/0xda May 17 21:29:02 client-5 kernel: [] vsnprintf+0x5e7/0x62f May 17 21:29:02 client-5 kernel: [] core_sys_select+0x1bc/0x265 May 17 21:29:02 client-5 kernel: [] math_state_restore+0x23/0x4c May 17 21:29:02 client-5 kernel: [] error_exit+0x0/0x84 May 17 21:29:02 client-5 kernel: [] sys_rt_sigreturn+0x283/0x356 May 17 21:29:02 client-5 kernel: [] sys_rt_sigreturn+0x323/0x356 May 17 21:29:02 client-5 kernel: [] sys_select+0xc1/0x17c May 17 21:29:02 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: xinetd S 0000000000000040 0 3391 1 11742 3402 3379 (NOTLB) May 17 21:29:02 client-5 kernel: ffff8103287d1a18 0000000000000082 ffff81033b464880 ffff81033b464880 May 17 21:29:02 client-5 kernel: 0000000010008040 0000000000000009 ffff81033a4dd100 ffff81017de92080 May 17 21:29:02 client-5 kernel: 0000401690fd3a6f 000000000000cc2d ffff81033a4dd2e8 0000000300000009 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] schedule_timeout+0x1e/0xad May 17 21:29:02 client-5 kernel: [] add_wait_queue+0x24/0x34 May 17 21:29:02 client-5 kernel: [] tcp_poll+0x1c/0x132 May 17 21:29:02 client-5 kernel: [] do_select+0x3fe/0x462 May 17 21:29:02 client-5 kernel: [] __pollwait+0x0/0xe2 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 last message repeated 2 times May 17 21:29:02 client-5 kernel: [] sock_def_readable+0x37/0x62 May 17 21:29:02 client-5 kernel: [] unix_dgram_sendmsg+0x44e/0x4e0 May 17 21:29:02 client-5 kernel: [] flush_tlb_page+0xac/0xda May 17 21:29:02 client-5 kernel: [] do_wp_page+0x3fd/0x902 May 17 21:29:02 client-5 kernel: [] autoremove_wake_function+0x0/0x2e May 17 21:29:02 client-5 kernel: [] core_sys_select+0x1bc/0x265 May 17 21:29:02 client-5 kernel: [] _atomic_dec_and_lock+0x39/0x57 May 17 21:29:02 client-5 kernel: [] mntput_no_expire+0x19/0x89 May 17 21:29:02 client-5 kernel: [] __up_read+0x19/0x7f May 17 21:29:02 client-5 kernel: [] do_page_fault+0x4fe/0x874 May 17 21:29:02 client-5 kernel: [] sys_select+0xc1/0x17c May 17 21:29:02 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: ntpd S ffff81000100caa0 0 3402 1 3420 3391 (NOTLB) May 17 21:29:02 client-5 kernel: ffff810328105a18 0000000000000086 ffff81033c0ca000 ffffffff80064bbd May 17 21:29:02 client-5 kernel: ffff81033c0ca000 000000000000000a ffff81033c5467e0 ffff81010b70e100 May 17 21:29:02 client-5 kernel: 000042766da3559e 000000000000342a ffff81033c5469c8 0000000128105ac8 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] _write_lock_bh+0x9/0x1c May 17 21:29:02 client-5 kernel: [] schedule_timeout+0x1e/0xad May 17 21:29:02 client-5 kernel: [] datagram_poll+0x18/0xcb May 17 21:29:02 client-5 kernel: [] udp_poll+0x15/0xf3 May 17 21:29:02 client-5 kernel: [] do_select+0x3fe/0x462 May 17 21:29:02 client-5 kernel: [] __pollwait+0x0/0xe2 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 last message repeated 7 times May 17 21:29:02 client-5 kernel: [] mntput_no_expire+0x19/0x89 May 17 21:29:02 client-5 kernel: [] core_sys_select+0x1bc/0x265 May 17 21:29:02 client-5 kernel: [] math_state_restore+0x23/0x4c May 17 21:29:02 client-5 kernel: [] error_exit+0x0/0x84 May 17 21:29:02 client-5 kernel: [] sys_rt_sigreturn+0x283/0x356 May 17 21:29:02 client-5 kernel: [] sys_rt_sigreturn+0x323/0x356 May 17 21:29:02 client-5 kernel: [] sys_select+0xc1/0x17c May 17 21:29:02 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: sendmail S ffff810001015120 0 3420 1 3428 3402 (NOTLB) May 17 21:29:02 client-5 kernel: ffff8103283a5a18 0000000000000086 ffff81033b9dc3c0 ffff81033b9dc3c0 May 17 21:29:02 client-5 kernel: 0000000010008040 000000000000000a ffff81033b591820 ffff81010b764080 May 17 21:29:02 client-5 kernel: 000042763eed8c5a 000000000000b13a ffff81033b591a08 000000020000000a May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] tcp_poll+0x0/0x132 May 17 21:29:02 client-5 kernel: [] schedule_timeout+0x8a/0xad May 17 21:29:02 client-5 kernel: [] process_timeout+0x0/0x5 May 17 21:29:02 client-5 kernel: [] do_select+0x3fe/0x462 May 17 21:29:02 client-5 kernel: [] __pollwait+0x0/0xe2 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] cache_alloc_refill+0x106/0x186 May 17 21:29:02 client-5 kernel: [] proc_alloc_inode+0x40/0x66 May 17 21:29:02 client-5 kernel: [] alloc_inode+0xeb/0x192 May 17 21:29:02 client-5 kernel: [] iget_locked+0xc5/0x149 May 17 21:29:02 client-5 kernel: [] wake_up_bit+0x11/0x22 May 17 21:29:02 client-5 kernel: [] inotify_d_instantiate+0x12/0x3c May 17 21:29:02 client-5 kernel: [] d_instantiate+0x4a/0x82 May 17 21:29:02 client-5 kernel: [] d_rehash+0x21/0x34 May 17 21:29:02 client-5 kernel: [] vsnprintf+0x5e7/0x62f May 17 21:29:02 client-5 kernel: [] dput+0x2c/0x114 May 17 21:29:02 client-5 kernel: [] sprintf+0x51/0x59 May 17 21:29:02 client-5 kernel: [] core_sys_select+0x1bc/0x265 May 17 21:29:02 client-5 kernel: [] loadavg_read_proc+0xae/0xda May 17 21:29:02 client-5 kernel: [] lock_kernel+0x1b/0x32 May 17 21:29:02 client-5 kernel: [] de_put+0xe/0x63 May 17 21:29:02 client-5 kernel: [] sys_select+0x153/0x17c May 17 21:29:02 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: sendmail S ffff81000100caa0 0 3428 1 3437 3420 (NOTLB) May 17 21:29:02 client-5 kernel: ffff810327d6ff68 0000000000000086 0000000000000000 ffffffff80043b90 May 17 21:29:02 client-5 kernel: ffff81033b2520c0 0000000000000007 ffff81033b2520c0 ffff81010b70e100 May 17 21:29:02 client-5 kernel: 0000418088fb323c 00000000000043db ffff81033b2522a8 00000001800b7649 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] sys_rt_sigreturn+0x323/0x356 May 17 21:29:02 client-5 kernel: [] sys_pause+0x19/0x22 May 17 21:29:02 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: gpm S 0000000000000020 0 3437 1 3453 3428 (NOTLB) May 17 21:29:02 client-5 kernel: ffff8103281e3a18 0000000000000086 0000000000000008 0000000000000000 May 17 21:29:02 client-5 kernel: 0000000000000000 0000000000000001 ffff81033f5c1860 ffff81033b90e7e0 May 17 21:29:02 client-5 kernel: 000000204fbf296d 0000000000028bb8 ffff81033f5c1a48 0000000000000000 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] vsnprintf+0x5e7/0x62f May 17 21:29:02 client-5 kernel: [] lock_timer_base+0x1b/0x3c May 17 21:29:02 client-5 kernel: [] __mod_timer+0x100/0x10f May 17 21:29:02 client-5 kernel: [] schedule_timeout+0x8a/0xad May 17 21:29:02 client-5 kernel: [] process_timeout+0x0/0x5 May 17 21:29:02 client-5 kernel: [] do_select+0x3fe/0x462 May 17 21:29:02 client-5 kernel: [] __pollwait+0x0/0xe2 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] vsnprintf+0x5e7/0x62f May 17 21:29:02 client-5 kernel: [] vsnprintf+0x400/0x62f May 17 21:29:02 client-5 kernel: [] sprintf+0x51/0x59 May 17 21:29:02 client-5 kernel: [] __wake_up+0x38/0x4f May 17 21:29:02 client-5 kernel: [] __next_cpu+0x19/0x28 May 17 21:29:02 client-5 kernel: [] smp_send_reschedule+0x4e/0x53 May 17 21:29:02 client-5 kernel: [] find_get_page+0x21/0x51 May 17 21:29:02 client-5 kernel: [] enqueue_task+0x41/0x56 May 17 21:29:02 client-5 kernel: [] filemap_nopage+0x193/0x360 May 17 21:29:02 client-5 kernel: [] core_sys_select+0x1bc/0x265 May 17 21:29:02 client-5 kernel: [] __wake_up_common+0x3e/0x68 May 17 21:29:02 client-5 kernel: [] __up_read+0x19/0x7f May 17 21:29:02 client-5 kernel: [] do_page_fault+0x4fe/0x874 May 17 21:29:02 client-5 kernel: [] __up_wakeup+0x35/0x67 May 17 21:29:02 client-5 kernel: [] class_device_create_release+0x0/0x5 May 17 21:29:02 client-5 kernel: [] mousedev_read+0x153/0x188 May 17 21:29:02 client-5 kernel: [] sys_select+0x153/0x17c May 17 21:29:02 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: crond S ffff810001004420 0 3453 1 3472 3437 (NOTLB) May 17 21:29:02 client-5 kernel: ffff810327d65eb8 0000000000000086 ffff810327d65e58 ffff810001006500 May 17 21:29:02 client-5 kernel: 0000000df8475800 000000000000000a ffff81033fbaf040 ffffffff80308b60 May 17 21:29:02 client-5 kernel: 0000427650019355 000000000000d9d8 ffff81033fbaf228 00000000b595035f May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] enqueue_hrtimer+0x55/0x70 May 17 21:29:02 client-5 kernel: [] do_nanosleep+0x3f/0x70 May 17 21:29:02 client-5 kernel: [] hrtimer_nanosleep+0x58/0x118 May 17 21:29:02 client-5 kernel: [] hrtimer_wakeup+0x0/0x22 May 17 21:29:02 client-5 kernel: [] sys_nanosleep+0x4c/0x62 May 17 21:29:02 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: atd S ffff810001004420 0 3472 1 3509 3453 (NOTLB) May 17 21:29:02 client-5 kernel: ffff810327e99eb8 0000000000000086 ffff810327e99e58 ffff810001006500 May 17 21:29:02 client-5 kernel: 00000045d964b800 0000000000000007 ffff81033b740100 ffffffff80308b60 May 17 21:29:02 client-5 kernel: 00004251ce6e3b2f 000000000000a5b7 ffff81033b7402e8 0000000005b80c7f May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] enqueue_hrtimer+0x55/0x70 May 17 21:29:02 client-5 kernel: [] do_nanosleep+0x3f/0x70 May 17 21:29:02 client-5 kernel: [] hrtimer_nanosleep+0x58/0x118 May 17 21:29:02 client-5 kernel: [] hrtimer_wakeup+0x0/0x22 May 17 21:29:02 client-5 kernel: [] sys_nanosleep+0x4c/0x62 May 17 21:29:02 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: smartd S ffff810001015120 0 3509 1 3513 3472 (NOTLB) May 17 21:29:02 client-5 kernel: ffff810327a3deb8 0000000000000082 ffff810327a3de58 ffff810001017200 May 17 21:29:02 client-5 kernel: 000001a3185c5000 0000000000000007 ffff810327a21820 ffff81010b764080 May 17 21:29:02 client-5 kernel: 00004180abb582fa 0000000000004989 ffff810327a21a08 00000002c9baf4b7 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] enqueue_hrtimer+0x55/0x70 May 17 21:29:02 client-5 kernel: [] do_nanosleep+0x3f/0x70 May 17 21:29:02 client-5 kernel: [] hrtimer_nanosleep+0x58/0x118 May 17 21:29:02 client-5 kernel: [] hrtimer_wakeup+0x0/0x22 May 17 21:29:02 client-5 kernel: [] sys_nanosleep+0x4c/0x62 May 17 21:29:02 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: mingetty S ffff81000100caa0 0 3513 1 3514 3509 (NOTLB) May 17 21:29:02 client-5 kernel: ffff810327a1bdb8 0000000000000086 0000000000000000 000000000000000c May 17 21:29:02 client-5 kernel: 0000000000000001 0000000000000007 ffff81033fc57100 ffff81010b70e100 May 17 21:29:02 client-5 kernel: 000000206e60dadf 000000000000182b ffff81033fc572e8 0000000180063054 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] schedule_timeout+0x1e/0xad May 17 21:29:02 client-5 kernel: [] add_wait_queue+0x24/0x34 May 17 21:29:02 client-5 kernel: [] read_chan+0x3d4/0x694 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] tty_ldisc_ref_wait+0xe/0x9b May 17 21:29:02 client-5 kernel: [] tty_read+0x72/0xc8 May 17 21:29:02 client-5 kernel: [] vfs_read+0xcb/0x171 May 17 21:29:02 client-5 kernel: [] sys_read+0x45/0x6e May 17 21:29:02 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: mingetty S ffff81033ea9c158 0 3514 1 3515 3513 (NOTLB) May 17 21:29:02 client-5 kernel: ffff810327e5ddb8 0000000000000086 0000000000000000 000000000000000c May 17 21:29:02 client-5 kernel: 0000000000000001 0000000000000007 ffff810332cd7860 ffff810327f437e0 May 17 21:29:02 client-5 kernel: 000000206e7fbb84 0000000000009756 ffff810332cd7a48 00000002800172da May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] do_con_write+0x1692/0x16c6 May 17 21:29:02 client-5 kernel: [] schedule_timeout+0x1e/0xad May 17 21:29:02 client-5 kernel: [] add_wait_queue+0x24/0x34 May 17 21:29:02 client-5 kernel: [] read_chan+0x3d4/0x694 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] tty_ldisc_ref_wait+0xe/0x9b May 17 21:29:02 client-5 kernel: [] tty_read+0x72/0xc8 May 17 21:29:02 client-5 kernel: [] vfs_read+0xcb/0x171 May 17 21:29:02 client-5 kernel: [] sys_read+0x45/0x6e May 17 21:29:02 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: mingetty S ffff810001004420 0 3515 1 3516 3514 (NOTLB) May 17 21:29:02 client-5 kernel: ffff810327eeddb8 0000000000000082 0000000000000000 000000000000000c May 17 21:29:02 client-5 kernel: 0000000000000001 0000000000000006 ffff810332cd7100 ffffffff80308b60 May 17 21:29:02 client-5 kernel: 000000206e88a46c 000000000009558d ffff810332cd72e8 00000000800172da May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] schedule_timeout+0x1e/0xad May 17 21:29:02 client-5 kernel: [] add_wait_queue+0x24/0x34 May 17 21:29:02 client-5 kernel: [] read_chan+0x3d4/0x694 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] tty_ldisc_ref_wait+0xe/0x9b May 17 21:29:02 client-5 kernel: [] tty_read+0x72/0xc8 May 17 21:29:02 client-5 kernel: [] vfs_read+0xcb/0x171 May 17 21:29:02 client-5 kernel: [] sys_read+0x45/0x6e May 17 21:29:02 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: mingetty S ffff81033ea5ea18 0 3516 1 3519 3515 (NOTLB) May 17 21:29:02 client-5 kernel: ffff810327fcddb8 0000000000000086 0000000000000000 000000000000000c May 17 21:29:02 client-5 kernel: 0000000000000001 0000000000000005 ffff810333be57a0 ffff81010b6fd7a0 May 17 21:29:02 client-5 kernel: 000000206e6dd636 00000000000dbeff ffff810333be5988 00000000800172da May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] do_con_write+0x1692/0x16c6 May 17 21:29:02 client-5 kernel: [] schedule_timeout+0x1e/0xad May 17 21:29:02 client-5 kernel: [] add_wait_queue+0x24/0x34 May 17 21:29:02 client-5 kernel: [] read_chan+0x3d4/0x694 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] tty_ldisc_ref_wait+0xe/0x9b May 17 21:29:02 client-5 kernel: [] tty_read+0x72/0xc8 May 17 21:29:02 client-5 kernel: [] vfs_read+0xcb/0x171 May 17 21:29:02 client-5 kernel: [] sys_read+0x45/0x6e May 17 21:29:02 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: mingetty S ffff81033ea5e718 0 3519 1 3520 3516 (NOTLB) May 17 21:29:02 client-5 kernel: ffff810327f2bdb8 0000000000000082 0000000000000000 000000000000000c May 17 21:29:02 client-5 kernel: 0000000000000001 0000000000000007 ffff810333be5040 ffff810332cd7100 May 17 21:29:02 client-5 kernel: 000000206e7cfc61 0000000000020a0b ffff810333be5228 00000000800172da May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] do_con_write+0x1692/0x16c6 May 17 21:29:02 client-5 kernel: [] schedule_timeout+0x1e/0xad May 17 21:29:02 client-5 kernel: [] add_wait_queue+0x24/0x34 May 17 21:29:02 client-5 kernel: [] read_chan+0x3d4/0x694 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] tty_ldisc_ref_wait+0xe/0x9b May 17 21:29:02 client-5 kernel: [] tty_read+0x72/0xc8 May 17 21:29:02 client-5 kernel: [] vfs_read+0xcb/0x171 May 17 21:29:02 client-5 kernel: [] sys_read+0x45/0x6e May 17 21:29:02 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: mingetty S ffff81033ea5e418 0 3520 1 3582 3519 (NOTLB) May 17 21:29:02 client-5 kernel: ffff810327c0bdb8 0000000000000082 0000000000000000 000000000000000c May 17 21:29:02 client-5 kernel: 0000000000000001 0000000000000002 ffff81033b040080 ffff810327f53820 May 17 21:29:02 client-5 kernel: 000000206e94b0be 000000000001be48 ffff81033b040268 00000000800172da May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] do_con_write+0x1692/0x16c6 May 17 21:29:02 client-5 kernel: [] schedule_timeout+0x1e/0xad May 17 21:29:02 client-5 kernel: [] add_wait_queue+0x24/0x34 May 17 21:29:02 client-5 kernel: [] read_chan+0x3d4/0x694 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] tty_ldisc_ref_wait+0xe/0x9b May 17 21:29:02 client-5 kernel: [] tty_read+0x72/0xc8 May 17 21:29:02 client-5 kernel: [] vfs_read+0xcb/0x171 May 17 21:29:02 client-5 kernel: [] sys_read+0x45/0x6e May 17 21:29:02 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: yum-updatesd S ffff810001004420 0 3582 1 3584 3520 (NOTLB) May 17 21:29:02 client-5 kernel: ffff8103267bdb38 0000000000000082 000041814507b2cf ffff810001004420 May 17 21:29:02 client-5 kernel: ffff8103267bdab8 000000000000000a ffff810327ad0080 ffffffff80308b60 May 17 21:29:02 client-5 kernel: 00004181531c9c93 000000000001981c ffff810327ad0268 0000000080046edb May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] schedule_timeout+0x8a/0xad May 17 21:29:02 client-5 kernel: [] process_timeout+0x0/0x5 May 17 21:29:02 client-5 kernel: [] do_sys_poll+0x277/0x360 May 17 21:29:02 client-5 kernel: [] __pollwait+0x0/0xe2 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] flush_tlb_page+0x57/0xda May 17 21:29:02 client-5 kernel: [] flush_tlb_page+0xac/0xda May 17 21:29:02 client-5 kernel: [] do_wp_page+0x3fd/0x902 May 17 21:29:02 client-5 kernel: [] __handle_mm_fault+0xf6b/0x1039 May 17 21:29:02 client-5 kernel: [] do_page_fault+0x4fe/0x874 May 17 21:29:02 client-5 kernel: [] release_task+0x3a5/0x3cb May 17 21:29:02 client-5 kernel: [] remove_wait_queue+0x1c/0x2c May 17 21:29:02 client-5 kernel: [] audit_syscall_entry+0x180/0x1b3 May 17 21:29:02 client-5 kernel: [] sys_poll+0x2d/0x34 May 17 21:29:02 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: gam_server S ffff810001004420 0 3584 1 3630 3582 (NOTLB) May 17 21:29:02 client-5 kernel: ffff810325b91b38 0000000000000086 ffff810000020c00 00000002000204d0 May 17 21:29:02 client-5 kernel: 0000000100000000 000000000000000a ffff810327b88100 ffffffff80308b60 May 17 21:29:02 client-5 kernel: 000042763c11c035 0000000000003489 ffff810327b882e8 0000000000000000 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] schedule_timeout+0x8a/0xad May 17 21:29:02 client-5 kernel: [] process_timeout+0x0/0x5 May 17 21:29:02 client-5 kernel: [] do_sys_poll+0x277/0x360 May 17 21:29:02 client-5 kernel: [] __pollwait+0x0/0xe2 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 last message repeated 4 times May 17 21:29:02 client-5 kernel: [] do_sock_write+0xc6/0x102 May 17 21:29:02 client-5 kernel: [] sock_aio_write+0x4f/0x5e May 17 21:29:02 client-5 kernel: [] _atomic_dec_and_lock+0x39/0x57 May 17 21:29:02 client-5 kernel: [] do_sync_write+0xc7/0x104 May 17 21:29:02 client-5 kernel: [] prepare_to_wait+0x34/0x61 May 17 21:29:02 client-5 kernel: [] autoremove_wake_function+0x0/0x2e May 17 21:29:02 client-5 kernel: [] autoremove_wake_function+0x0/0x2e May 17 21:29:02 client-5 kernel: [] audit_syscall_entry+0x180/0x1b3 May 17 21:29:02 client-5 kernel: [] sys_poll+0x2d/0x34 May 17 21:29:02 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: login S ffff810001015120 0 3630 1 3631 28792 3584 (NOTLB) May 17 21:29:02 client-5 kernel: ffff810325f7dee8 0000000000000086 0000000000000292 ffffffff80022205 May 17 21:29:02 client-5 kernel: ffff81033c448500 0000000000000008 ffff81033aabe7a0 ffff81010b764080 May 17 21:29:02 client-5 kernel: 0000002c1049f6f9 0000000000004bf2 ffff81033aabe988 000000023aabe7a0 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] __up_read+0x19/0x7f May 17 21:29:02 client-5 kernel: [] do_wait+0xa03/0xad7 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: bash S 0000000000000001 0 3631 3630 3677 (NOTLB) May 17 21:29:02 client-5 kernel: ffff810327fb9ee8 0000000000000082 ffff81033f7d1800 0000000000000000 May 17 21:29:02 client-5 kernel: ffff810327fb9e88 0000000000000009 ffff81033b547100 ffff81033b6f67e0 May 17 21:29:02 client-5 kernel: 00000036f5d18ca9 000000000000ea52 ffff81033b5472e8 0000000200000001 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] do_wait+0xa03/0xad7 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: sh S 0000000000000001 0 3677 3631 29246 (NOTLB) May 17 21:29:02 client-5 kernel: ffff810325803ee8 0000000000000082 0000000000000292 ffffffff80022205 May 17 21:29:02 client-5 kernel: ffff81033c4fdc40 0000000000000002 ffff81033b6f67e0 ffff810321d6c0c0 May 17 21:29:02 client-5 kernel: 00003ffe05ba4b64 000000000000dfcf ffff81033b6f69c8 000000023b6f67e0 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] __up_read+0x19/0x7f May 17 21:29:02 client-5 kernel: [] do_wait+0xa03/0xad7 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: nfsiod S 0000000000000282 0 932 123 3131 (L-TLB) May 17 21:29:02 client-5 kernel: ffff810310adfe70 0000000000000046 00002b320234ac06 0000000000000001 May 17 21:29:02 client-5 kernel: ffff81030be69860 0000000000000006 ffff81030be69860 ffff81033fb790c0 May 17 21:29:02 client-5 kernel: 00002b32023b9049 000000000000064a ffff81030be69a48 0000000132d65540 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] thread_return+0x89/0xfe May 17 21:29:02 client-5 kernel: [] recalc_sigpending_and_wake+0x9/0x1a May 17 21:29:02 client-5 kernel: [] do_sigaction+0x14d/0x199 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] worker_thread+0x0/0x122 May 17 21:29:02 client-5 kernel: [] worker_thread+0xc0/0x122 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0xfe/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] keventd_create_kthread+0x0/0xc4 May 17 21:29:02 client-5 kernel: [] kthread+0x0/0x132 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: in.rlogind S ffff810001015120 0 11742 3391 11743 (NOTLB) May 17 21:29:02 client-5 kernel: ffff81031936da18 0000000000000086 ffff81033c0ca500 00000009000002a4 May 17 21:29:02 client-5 kernel: 000000c300c31a08 0000000000000009 ffff81033fc69820 ffff81010b764080 May 17 21:29:02 client-5 kernel: 000042767653ea8d 00000000000039f2 ffff81033fc69a08 000000020b7e4038 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] schedule_timeout+0x1e/0xad May 17 21:29:02 client-5 kernel: [] tty_poll+0x5f/0x6d May 17 21:29:02 client-5 kernel: [] do_select+0x3fe/0x462 May 17 21:29:02 client-5 kernel: [] __pollwait+0x0/0xe2 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 last message repeated 2 times May 17 21:29:02 client-5 kernel: [] lock_timer_base+0x1b/0x3c May 17 21:29:02 client-5 kernel: [] __mod_timer+0x100/0x10f May 17 21:29:02 client-5 kernel: [] sk_reset_timer+0xf/0x19 May 17 21:29:02 client-5 kernel: [] __tcp_push_pending_frames+0x782/0x85c May 17 21:29:02 client-5 kernel: [] __alloc_skb+0x77/0x12d May 17 21:29:02 client-5 kernel: [] _spin_lock_bh+0x9/0x14 May 17 21:29:02 client-5 kernel: [] release_sock+0x13/0xaa May 17 21:29:02 client-5 kernel: [] tcp_sendmsg+0xa07/0xb0b May 17 21:29:02 client-5 kernel: [] do_sock_write+0xc6/0x102 May 17 21:29:02 client-5 kernel: [] core_sys_select+0x1bc/0x265 May 17 21:29:02 client-5 kernel: [] remove_wait_queue+0x1c/0x2c May 17 21:29:02 client-5 kernel: [] do_sync_write+0xc7/0x104 May 17 21:29:02 client-5 kernel: [] read_chan+0x626/0x694 May 17 21:29:02 client-5 kernel: [] autoremove_wake_function+0x0/0x2e May 17 21:29:02 client-5 kernel: [] tty_ldisc_deref+0x68/0x7b May 17 21:29:02 client-5 kernel: [] sys_select+0xc1/0x17c May 17 21:29:02 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: login S ffff810001004420 0 11743 11742 11744 (NOTLB) May 17 21:29:02 client-5 kernel: ffff8103119a9ee8 0000000000000082 0000000000000292 ffffffff80022205 May 17 21:29:02 client-5 kernel: ffff81033cb4bb40 0000000000000007 ffff8102d94087e0 ffffffff80308b60 May 17 21:29:02 client-5 kernel: 00003de93ce0f0aa 0000000000005961 ffff8102d94089c8 00000000d94087e0 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] __up_read+0x19/0x7f May 17 21:29:02 client-5 kernel: [] do_wait+0xa03/0xad7 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: bash R running task 0 11744 11743 (NOTLB) May 17 21:29:02 client-5 kernel: 0000000000000008 ffffffff800767ae 0000000000000008 0000000000000000 May 17 21:29:02 client-5 kernel: 0000000000000000 0000000000000000 ffff81033aabe040 ffffffff8008c937 May 17 21:29:02 client-5 kernel: 0000000000000008 0000000000000000 0000000000000000 0000000000000000 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] enqueue_task+0x41/0x56 May 17 21:29:02 client-5 kernel: [] __wake_up+0x38/0x4f May 17 21:29:02 client-5 kernel: [] serial8250_console_putchar+0x3f/0xa5 May 17 21:29:02 client-5 last message repeated 2 times May 17 21:29:02 client-5 kernel: [] printk+0x52/0xbd May 17 21:29:02 client-5 kernel: [] __wake_up_common+0x3e/0x68 May 17 21:29:02 client-5 kernel: [] __wake_up_common+0x3e/0x68 May 17 21:29:02 client-5 kernel: [] __wake_up+0x38/0x4f May 17 21:29:02 client-5 kernel: [] kallsyms_lookup+0xe6/0x1ae May 17 21:29:02 client-5 last message repeated 3 times May 17 21:29:02 client-5 kernel: [] printk_address+0x9f/0xab May 17 21:29:02 client-5 kernel: [] gart_map_sg+0x1cf/0x401 May 17 21:29:02 client-5 kernel: [] printk+0x52/0xbd May 17 21:29:02 client-5 kernel: [] module_text_address+0x33/0x3c May 17 21:29:02 client-5 kernel: [] kernel_text_address+0x1a/0x26 May 17 21:29:02 client-5 kernel: [] dump_trace+0x206/0x22f May 17 21:29:02 client-5 kernel: [] show_trace+0x34/0x47 May 17 21:29:02 client-5 kernel: [] _show_stack+0xdb/0xea May 17 21:29:02 client-5 kernel: [] show_state+0x48/0x86 May 17 21:29:02 client-5 kernel: [] __handle_sysrq+0x90/0x121 May 17 21:29:02 client-5 kernel: [] write_sysrq_trigger+0x2a/0x32 May 17 21:29:02 client-5 kernel: [] vfs_write+0xce/0x174 May 17 21:29:02 client-5 kernel: [] sys_write+0x45/0x6e May 17 21:29:02 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: cfs_wi_sd000 S 0000000000000000 0 28792 1 28793 3630 (L-TLB) May 17 21:29:02 client-5 kernel: ffff8103192f1ec0 0000000000000046 00000000ffffffff 0000000000000001 May 17 21:29:02 client-5 kernel: 0000000000000003 0000000000000001 ffff81027425b0c0 ffff810309801860 May 17 21:29:02 client-5 kernel: 00003ffddde6c79b 0000000000001469 ffff81027425b2a8 00000000192f1f18 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] :libcfs:cfs_wi_scheduler+0x239/0x2a0 May 17 21:29:02 client-5 kernel: [] autoremove_wake_function+0x0/0x2e May 17 21:29:02 client-5 kernel: [] audit_syscall_exit+0x336/0x362 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] :libcfs:cfs_wi_scheduler+0x0/0x2a0 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: cfs_wi_sd001 S 0000000000000000 0 28793 1 28794 28792 (L-TLB) May 17 21:29:02 client-5 kernel: ffff81030269dec0 0000000000000046 00000000ffffffff 0000000000000001 May 17 21:29:02 client-5 kernel: 0000000000000003 0000000000000001 ffff810309801860 ffff8102f44587e0 May 17 21:29:02 client-5 kernel: 00003ffddde6cfcf 0000000000000834 ffff810309801a48 000000000269df18 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] :libcfs:cfs_wi_scheduler+0x239/0x2a0 May 17 21:29:02 client-5 kernel: [] autoremove_wake_function+0x0/0x2e May 17 21:29:02 client-5 kernel: [] audit_syscall_exit+0x336/0x362 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] :libcfs:cfs_wi_scheduler+0x0/0x2a0 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: cfs_wi_sd002 S 0000000000000000 0 28794 1 28795 28793 (L-TLB) May 17 21:29:02 client-5 kernel: ffff810314b51ec0 0000000000000046 00000000ffffffff 0000000000000001 May 17 21:29:02 client-5 kernel: 0000000000000003 0000000000000001 ffff8102f44587e0 ffff81027425b820 May 17 21:29:02 client-5 kernel: 00003ffddde6d84b 000000000000087c ffff8102f44589c8 0000000014b51f18 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] :libcfs:cfs_wi_scheduler+0x239/0x2a0 May 17 21:29:02 client-5 kernel: [] autoremove_wake_function+0x0/0x2e May 17 21:29:02 client-5 kernel: [] audit_syscall_exit+0x336/0x362 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] :libcfs:cfs_wi_scheduler+0x0/0x2a0 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: cfs_wi_sd003 S 0000000000000000 0 28795 1 28796 28794 (L-TLB) May 17 21:29:02 client-5 kernel: ffff81031bc73ec0 0000000000000046 00000000ffffffff 0000000000000001 May 17 21:29:02 client-5 kernel: 0000000000000003 0000000000000001 ffff81027425b820 ffff81031a8f10c0 May 17 21:29:02 client-5 kernel: 00003ffddde6dfc5 000000000000077a ffff81027425ba08 0000000000000030 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] :libcfs:cfs_wi_scheduler+0x239/0x2a0 May 17 21:29:02 client-5 kernel: [] autoremove_wake_function+0x0/0x2e May 17 21:29:02 client-5 kernel: [] audit_syscall_exit+0x336/0x362 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] :libcfs:cfs_wi_scheduler+0x0/0x2a0 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: wi_serial_sd S 0000000000000000 0 28796 1 28859 28795 (L-TLB) May 17 21:29:02 client-5 kernel: ffff810326cf1ec0 0000000000000046 ffff81031d5ee878 000000000000003f May 17 21:29:02 client-5 kernel: ffffffff80359d38 0000000000000001 ffff81031a8f10c0 ffff81031526e820 May 17 21:29:02 client-5 kernel: 00003ffddde6e706 0000000000000741 ffff81031a8f12a8 0000000000000000 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] :libcfs:cfs_wi_scheduler+0x239/0x2a0 May 17 21:29:02 client-5 kernel: [] autoremove_wake_function+0x0/0x2e May 17 21:29:02 client-5 kernel: [] audit_syscall_exit+0x336/0x362 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] :libcfs:cfs_wi_scheduler+0x0/0x2a0 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: obd_zombid S 00002ad503746010 0 28859 1 28878 28796 (L-TLB) May 17 21:29:02 client-5 kernel: ffff81031f049ec0 0000000000000046 0000000000061a04 000000000036c752 May 17 21:29:02 client-5 kernel: 0000000100000001 0000000000000006 ffff81031ec2e860 ffff81031931b0c0 May 17 21:29:02 client-5 kernel: 00003ffde5e89e0c 00000000000020bf ffff81031ec2ea48 0000000000000000 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] __wake_up_common+0x3e/0x68 May 17 21:29:02 client-5 kernel: [] :obdclass:obd_zombie_impexp_check+0x11/0xc0 May 17 21:29:02 client-5 kernel: [] :obdclass:obd_zombie_impexp_thread+0x18d/0x2a0 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] :obdclass:obd_zombie_impexp_thread+0x0/0x2a0 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: ptlrpc_hr_0 S ffff8102e80ad820 0 28878 1 28879 28859 (L-TLB) May 17 21:29:02 client-5 kernel: ffff8102c611be40 0000000000000046 ffff8102c611bd90 0000000000000086 May 17 21:29:02 client-5 kernel: 0000000000445299 0000000000000001 ffff81031931b0c0 ffff8102e80ad820 May 17 21:29:02 client-5 kernel: 00003ffdea1bd3b5 0000000000002e3c ffff81031931b2a8 000000008008b3c7 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] __wake_up+0x38/0x4f May 17 21:29:02 client-5 kernel: [] :ptlrpc:ptlrpc_hr_main+0x19f/0x650 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] audit_syscall_exit+0x336/0x362 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] :ptlrpc:ptlrpc_hr_main+0x0/0x650 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: ptlrpc_hr_1 S ffff8102e80ad820 0 28879 1 28880 28878 (L-TLB) May 17 21:29:02 client-5 kernel: ffff8103174b9e40 0000000000000046 ffff8103174b9d90 0000000000000086 May 17 21:29:02 client-5 kernel: 000000000023201d 0000000000000001 ffff81033fb647e0 ffff8102e80ad820 May 17 21:29:02 client-5 kernel: 00003ffdea1bfbee 000000000000119b ffff81033fb649c8 000000008008b3c7 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] __wake_up+0x38/0x4f May 17 21:29:02 client-5 kernel: [] :ptlrpc:ptlrpc_hr_main+0x19f/0x650 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] audit_syscall_exit+0x336/0x362 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] :ptlrpc:ptlrpc_hr_main+0x0/0x650 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: ptlrpc_hr_2 S ffff8102e80ad820 0 28880 1 28881 28879 (L-TLB) May 17 21:29:02 client-5 kernel: ffff81030f55be40 0000000000000046 ffff81030f55bd90 0000000000000086 May 17 21:29:02 client-5 kernel: 000000180000000a 0000000000000001 ffff81033b27c7e0 ffff8102e80ad820 May 17 21:29:02 client-5 kernel: 00003ffdea1c2521 0000000000000bd1 ffff81033b27c9c8 000000008008b3c7 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] __wake_up+0x38/0x4f May 17 21:29:02 client-5 kernel: [] :ptlrpc:ptlrpc_hr_main+0x19f/0x650 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] audit_syscall_exit+0x336/0x362 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] :ptlrpc:ptlrpc_hr_main+0x0/0x650 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: ptlrpc_hr_3 S ffff8102e80ad820 0 28881 1 28889 28880 (L-TLB) May 17 21:29:02 client-5 kernel: ffff81032304fe40 0000000000000046 ffff81032304fd90 0000000000000086 May 17 21:29:02 client-5 kernel: 0000000000440eca 0000000000000001 ffff81033b158820 ffff8102e80ad820 May 17 21:29:02 client-5 kernel: 00003ffdea1dd9aa 0000000000019756 ffff81033b158a08 000000008008b3c7 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] __wake_up+0x38/0x4f May 17 21:29:02 client-5 kernel: [] :ptlrpc:ptlrpc_hr_main+0x19f/0x650 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] audit_syscall_exit+0x336/0x362 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] :ptlrpc:ptlrpc_hr_main+0x0/0x650 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: kiblnd_sd_00 S ffff81031997a040 0 28889 1 28890 28881 (L-TLB) May 17 21:29:02 client-5 kernel: ffff810321e97e50 0000000000000046 ffff8101f577ee00 000000000000001f May 17 21:29:02 client-5 kernel: 0000000000000282 000000000000000a ffff810309801100 ffff81031997a040 May 17 21:29:02 client-5 kernel: 0000427556ffa0f4 000000000000022d ffff8103098012e8 0000000300000001 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] :ko2iblnd:kiblnd_scheduler+0x66c/0x6e0 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] audit_syscall_exit+0x336/0x362 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] :ko2iblnd:kiblnd_scheduler+0x0/0x6e0 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: kiblnd_sd_01 S ffff81000101d7a0 0 28890 1 28891 28889 (L-TLB) May 17 21:29:02 client-5 kernel: ffff810310681e50 0000000000000046 ffff8102fffb6ec0 000000c0000000c0 May 17 21:29:02 client-5 kernel: 0000001200000000 000000000000000a ffff81031997a040 ffff81010b770100 May 17 21:29:02 client-5 kernel: 0000427556ffa339 0000000000000245 ffff81031997a228 0000000388a0ed9b May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] :ko2iblnd:kiblnd_scheduler+0x66c/0x6e0 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] audit_syscall_exit+0x336/0x362 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] :ko2iblnd:kiblnd_scheduler+0x0/0x6e0 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: kiblnd_sd_02 S ffff81026c179080 0 28891 1 28892 28890 (L-TLB) May 17 21:29:02 client-5 kernel: ffff81032f0b5e50 0000000000000046 ffff810252c21ec0 0000000000000082 May 17 21:29:02 client-5 kernel: 0000001200000000 000000000000000a ffff81031977e080 ffff81026c179080 May 17 21:29:02 client-5 kernel: 0000427556ff9b06 0000000000007047 ffff81031977e268 0000000388a0e2bf May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] :mlx4_ib:mlx4_ib_poll_cq+0x57e/0x5b9 May 17 21:29:02 client-5 kernel: [] :ko2iblnd:kiblnd_scheduler+0x66c/0x6e0 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] audit_syscall_exit+0x336/0x362 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] :ko2iblnd:kiblnd_scheduler+0x0/0x6e0 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: kiblnd_sd_03 S ffff810309801100 0 28892 1 28893 28891 (L-TLB) May 17 21:29:02 client-5 kernel: ffff8102fa2b5e50 0000000000000046 ffff8101f577ee00 0000000000000014 May 17 21:29:02 client-5 kernel: 0000000000000282 000000000000000a ffff81026c179080 ffff810309801100 May 17 21:29:02 client-5 kernel: 0000427556ff9ec7 00000000000003c1 ffff81026c179268 0000000300000001 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] :ko2iblnd:kiblnd_scheduler+0x66c/0x6e0 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] audit_syscall_exit+0x336/0x362 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] :ko2iblnd:kiblnd_scheduler+0x0/0x6e0 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: kiblnd_connd S ffff81000101d7a0 0 28893 1 28896 28892 (L-TLB) May 17 21:29:02 client-5 kernel: ffff8102fc199e50 0000000000000046 ffff81010b66bc80 ffffffff800939b4 May 17 21:29:02 client-5 kernel: ffff8102fc199e60 000000000000000a ffff8102cc559860 ffff81010b770100 May 17 21:29:02 client-5 kernel: 00004276641ddb8a 0000000000000920 ffff8102cc559a48 0000000388a0d4e6 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] daemonize+0x2fa/0x304 May 17 21:29:02 client-5 kernel: [] schedule_timeout+0x8a/0xad May 17 21:29:02 client-5 kernel: [] process_timeout+0x0/0x5 May 17 21:29:02 client-5 kernel: [] :ko2iblnd:kiblnd_connd+0x2d3/0x360 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] audit_syscall_exit+0x336/0x362 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] :ko2iblnd:kiblnd_connd+0x0/0x360 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: socknal_sd00 S 0000000000000000 0 28894 1 28898 28896 (L-TLB) May 17 21:29:02 client-5 kernel: ffff810310f53e60 0000000000000046 00000000ffffffff 0000000000000001 May 17 21:29:02 client-5 kernel: 0000000000000002 0000000000000001 ffff8102cab51080 ffff81033c54a040 May 17 21:29:02 client-5 kernel: 00003ffdeafd8e11 0000000000001606 ffff8102cab51268 00000000800646f9 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] :ksocklnd:ksocknal_scheduler+0x58b/0x640 May 17 21:29:02 client-5 kernel: [] autoremove_wake_function+0x0/0x2e May 17 21:29:02 client-5 kernel: [] audit_syscall_exit+0x336/0x362 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] :ksocklnd:ksocknal_scheduler+0x0/0x640 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: socknal_sd01 S 0000000000000000 0 28895 1 28897 28903 (L-TLB) May 17 21:29:02 client-5 kernel: ffff8102fc389e60 0000000000000046 00000000ffffffff 0000000000000001 May 17 21:29:02 client-5 kernel: 0000000000000002 0000000000000001 ffff81033fb790c0 ffff8102e80ad820 May 17 21:29:02 client-5 kernel: 00003ffdeafe41fb 0000000000000bb1 ffff81033fb792a8 00000000800646f9 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] :ksocklnd:ksocknal_scheduler+0x58b/0x640 May 17 21:29:02 client-5 kernel: [] autoremove_wake_function+0x0/0x2e May 17 21:29:02 client-5 kernel: [] audit_syscall_exit+0x336/0x362 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] :ksocklnd:ksocknal_scheduler+0x0/0x640 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: socknal_sd02 S ffff810001015120 0 28896 1 28894 28893 (L-TLB) May 17 21:29:02 client-5 kernel: ffff810318cd3e60 0000000000000046 ffffffff8032bc28 0000000000000000 May 17 21:29:02 client-5 kernel: 0000000000000001 0000000000000001 ffff81033ab1b860 ffff81010b764080 May 17 21:29:02 client-5 kernel: 00003ffdeafc4a09 0000000000004854 ffff81033ab1ba48 00000002112320b0 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] :ksocklnd:ksocknal_scheduler+0x58b/0x640 May 17 21:29:02 client-5 kernel: [] autoremove_wake_function+0x0/0x2e May 17 21:29:02 client-5 kernel: [] audit_syscall_exit+0x336/0x362 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] :ksocklnd:ksocknal_scheduler+0x0/0x640 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: socknal_sd03 S 0000000000000000 0 28897 1 28904 28895 (L-TLB) May 17 21:29:02 client-5 kernel: ffff8102e863de60 0000000000000046 00000000ffffffff 0000000000000001 May 17 21:29:02 client-5 kernel: 0000000000000002 0000000000000001 ffff8102efe8d820 ffff81030318b820 May 17 21:29:02 client-5 kernel: 00003ffdeaffad68 0000000000000fb0 ffff8102efe8da08 00000000800646f9 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] :ksocklnd:ksocknal_scheduler+0x58b/0x640 May 17 21:29:02 client-5 kernel: [] autoremove_wake_function+0x0/0x2e May 17 21:29:02 client-5 kernel: [] audit_syscall_exit+0x336/0x362 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] :ksocklnd:ksocknal_scheduler+0x0/0x640 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: socknal_cd00 S ffff810330eebee0 0 28898 1 28899 28894 (L-TLB) May 17 21:29:02 client-5 kernel: ffff810330eebe40 0000000000000046 ffff81010b66bc80 ffffffff800939b4 May 17 21:29:02 client-5 kernel: 0000003000000008 0000000000000001 ffff81033c54a040 ffff8102cc559100 May 17 21:29:02 client-5 kernel: 00003ffdeafd9dfd 0000000000000fec ffff81033c54a228 0000000030eebf20 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] daemonize+0x2fa/0x304 May 17 21:29:02 client-5 kernel: [] schedule_timeout+0x1e/0xad May 17 21:29:02 client-5 kernel: [] add_wait_queue_exclusive+0x28/0x38 May 17 21:29:02 client-5 kernel: [] :ksocklnd:ksocknal_connd+0x415/0x4b0 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] :ksocklnd:ksocknal_connd+0x0/0x4b0 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: socknal_cd01 S ffff810327ba5ee0 0 28899 1 28900 28898 (L-TLB) May 17 21:29:02 client-5 kernel: ffff810327ba5e40 0000000000000046 ffff81010b66bc80 ffffffff800939b4 May 17 21:29:02 client-5 kernel: 0000003000000008 0000000000000001 ffff8102cc559100 ffff81031931b820 May 17 21:29:02 client-5 kernel: 00003ffdeafda8b9 0000000000000abc ffff8102cc5592e8 0000000027ba5f20 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] daemonize+0x2fa/0x304 May 17 21:29:02 client-5 kernel: [] schedule_timeout+0x1e/0xad May 17 21:29:02 client-5 kernel: [] add_wait_queue_exclusive+0x28/0x38 May 17 21:29:02 client-5 kernel: [] :ksocklnd:ksocknal_connd+0x415/0x4b0 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] :ksocklnd:ksocknal_connd+0x0/0x4b0 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: socknal_cd02 S ffff81030a269ee0 0 28900 1 28901 28899 (L-TLB) May 17 21:29:02 client-5 kernel: ffff81030a269e40 0000000000000046 ffff81010b66bc80 ffffffff800939b4 May 17 21:29:02 client-5 kernel: 0000003000000008 0000000000000001 ffff81031931b820 ffff810327ad07e0 May 17 21:29:02 client-5 kernel: 00003ffdeafdb279 00000000000009c0 ffff81031931ba08 000000000a269f20 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] daemonize+0x2fa/0x304 May 17 21:29:02 client-5 kernel: [] schedule_timeout+0x1e/0xad May 17 21:29:02 client-5 kernel: [] add_wait_queue_exclusive+0x28/0x38 May 17 21:29:02 client-5 kernel: [] :ksocklnd:ksocknal_connd+0x415/0x4b0 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] :ksocklnd:ksocknal_connd+0x0/0x4b0 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: socknal_cd03 S ffff810323945ee0 0 28901 1 28902 28900 (L-TLB) May 17 21:29:02 client-5 kernel: ffff810323945e40 0000000000000046 ffff81010b66bc80 ffffffff800939b4 May 17 21:29:02 client-5 kernel: 0000003000000008 0000000000000001 ffff810327ad07e0 ffff810321d6c820 May 17 21:29:02 client-5 kernel: 00003ffdeafdbbc0 0000000000000947 ffff810327ad09c8 0000000023945f20 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] daemonize+0x2fa/0x304 May 17 21:29:02 client-5 kernel: [] schedule_timeout+0x1e/0xad May 17 21:29:02 client-5 kernel: [] add_wait_queue_exclusive+0x28/0x38 May 17 21:29:02 client-5 kernel: [] :ksocklnd:ksocknal_connd+0x415/0x4b0 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] :ksocklnd:ksocknal_connd+0x0/0x4b0 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: socknal_reape S ffff81033c327080 0 28902 1 28903 28901 (L-TLB) May 17 21:29:02 client-5 kernel: ffff810303bf7e60 0000000000000046 ffff81010b66bc80 ffffffff800939b4 May 17 21:29:02 client-5 kernel: 0000003000000008 000000000000000a ffff810321d6c820 ffff81033c327080 May 17 21:29:02 client-5 kernel: 00004276645aa02e 0000000000000aca ffff810321d6ca08 0000000000000000 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] daemonize+0x2fa/0x304 May 17 21:29:02 client-5 kernel: [] :ksocklnd:ksocknal_check_peer_timeouts+0x2b/0x410 May 17 21:29:02 client-5 kernel: [] lock_timer_base+0x1b/0x3c May 17 21:29:02 client-5 kernel: [] __mod_timer+0x100/0x10f May 17 21:29:02 client-5 kernel: [] schedule_timeout+0x8a/0xad May 17 21:29:02 client-5 kernel: [] process_timeout+0x0/0x5 May 17 21:29:02 client-5 kernel: [] :ksocklnd:ksocknal_reaper+0x2ec/0x370 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] audit_syscall_exit+0x336/0x362 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] :ksocklnd:ksocknal_reaper+0x0/0x370 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: acceptor_7988 S ffff810289b09f18 0 28903 1 28895 28902 (L-TLB) May 17 21:29:02 client-5 kernel: ffff810289b09e30 0000000000000046 00000000000000d0 ffffffff80064b29 May 17 21:29:02 client-5 kernel: ffff810306b2d940 0000000000000001 ffff81033c546080 ffff81033fb790c0 May 17 21:29:02 client-5 kernel: 00003ffdeafe364a 0000000000006ac3 ffff81033c546268 00000000803199d0 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] _spin_lock_bh+0x9/0x14 May 17 21:29:02 client-5 kernel: [] new_inode+0x4e/0x7f May 17 21:29:02 client-5 kernel: [] :libcfs:libcfs_sock_accept+0xf9/0x170 May 17 21:29:02 client-5 kernel: [] __wake_up_common+0x3e/0x68 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] :lnet:lnet_acceptor+0x208/0x4a0 May 17 21:29:02 client-5 kernel: [] audit_syscall_exit+0x336/0x362 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] :lnet:lnet_acceptor+0x0/0x4a0 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: ptlrpcd-brw S ffff810001004420 0 28904 1 28905 28897 (L-TLB) May 17 21:29:02 client-5 kernel: ffff8102c2907e00 0000000000000046 0004dd3406d00f76 00000000000004cb May 17 21:29:02 client-5 kernel: 000000004dd34477 000000000000000a ffff81030318b820 ffffffff80308b60 May 17 21:29:02 client-5 kernel: 0000420de9bf873a 0000000000000efd ffff81030318ba08 0000000017269768 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] schedule_timeout+0x8a/0xad May 17 21:29:02 client-5 kernel: [] process_timeout+0x0/0x5 May 17 21:29:02 client-5 kernel: [] :ptlrpc:ptlrpcd+0x294/0x382 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] audit_syscall_exit+0x336/0x362 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] :ptlrpc:ptlrpcd+0x0/0x382 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: ptlrpcd-brw-r S ffff810001004420 0 28905 1 28906 28904 (L-TLB) May 17 21:29:02 client-5 kernel: ffff81032790fe00 0000000000000046 0000000000000086 ffffffff80046edb May 17 21:29:02 client-5 kernel: 00000000ffffffff 000000000000000a ffff81033c327080 ffffffff80308b60 May 17 21:29:02 client-5 kernel: 00004276645aabb7 0000000000000b89 ffff81033c327268 00000000887524b8 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] try_to_wake_up+0x472/0x484 May 17 21:29:02 client-5 kernel: [] schedule_timeout+0x8a/0xad May 17 21:29:02 client-5 kernel: [] process_timeout+0x0/0x5 May 17 21:29:02 client-5 kernel: [] :ptlrpc:ptlrpcd+0x294/0x382 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] audit_syscall_exit+0x336/0x362 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] :ptlrpc:ptlrpcd+0x0/0x382 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: ptlrpcd S ffff810001004420 0 28906 1 28907 28905 (L-TLB) May 17 21:29:02 client-5 kernel: ffff8102e5593e00 0000000000000046 ffff810291ba8338 00000000000070ea May 17 21:29:02 client-5 kernel: ffff810191bda4d8 000000000000000a ffff810136568080 ffffffff80308b60 May 17 21:29:02 client-5 kernel: 0000427644fe9d08 0000000000000faa ffff810136568268 0000000000000000 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] schedule_timeout+0x8a/0xad May 17 21:29:02 client-5 kernel: [] process_timeout+0x0/0x5 May 17 21:29:02 client-5 kernel: [] :ptlrpc:ptlrpcd+0x294/0x382 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] audit_syscall_exit+0x336/0x362 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] :ptlrpc:ptlrpcd+0x0/0x382 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: ptlrpcd-rcv S ffff81000101d7a0 0 28907 1 28908 28906 (L-TLB) May 17 21:29:02 client-5 kernel: ffff8102d563fe00 0000000000000046 ffff8102ff3fc238 ffffffff000070eb May 17 21:29:02 client-5 kernel: ffff81032faa24d8 000000000000000a ffff810327b88860 ffff81010b770100 May 17 21:29:02 client-5 kernel: 00004276476f50f8 000000000000112b ffff810327b88a48 0000000300000000 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] schedule_timeout+0x8a/0xad May 17 21:29:02 client-5 kernel: [] process_timeout+0x0/0x5 May 17 21:29:02 client-5 kernel: [] :ptlrpc:ptlrpcd+0x294/0x382 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] audit_syscall_exit+0x336/0x362 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] :ptlrpc:ptlrpcd+0x0/0x382 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: ll_ping S ffff810001015120 0 28908 1 28909 28907 (L-TLB) May 17 21:29:02 client-5 kernel: ffff81033fae5e10 0000000000000046 0000000000000001 0000000000000009 May 17 21:29:02 client-5 kernel: 0000000000000286 000000000000000a ffff81033fbc00c0 ffff81010b764080 May 17 21:29:02 client-5 kernel: 0000427556f9dc08 00000000000074b0 ffff81033fbc02a8 0000000200000190 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] :ptlrpc:lustre_msg_size+0x11a/0x130 May 17 21:29:02 client-5 kernel: [] schedule_timeout+0x8a/0xad May 17 21:29:02 client-5 kernel: [] process_timeout+0x0/0x5 May 17 21:29:02 client-5 kernel: [] :ptlrpc:ptlrpc_pinger_main+0x68a/0x820 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] audit_syscall_exit+0x336/0x362 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] :ptlrpc:ptlrpc_pinger_main+0x0/0x820 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: sptlrpc_gc S ffff810001004420 0 28909 1 29054 28908 (L-TLB) May 17 21:29:02 client-5 kernel: ffff81030f423e60 0000000000000046 000000000f423dc0 0000000000000003 May 17 21:29:02 client-5 kernel: 00000000ffffffff 000000000000000a ffff81033c7d9820 ffffffff80308b60 May 17 21:29:02 client-5 kernel: 000041a050353016 0000000000000d8c ffff81033c7d9a08 0000000000000000 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] schedule_timeout+0x8a/0xad May 17 21:29:02 client-5 kernel: [] process_timeout+0x0/0x5 May 17 21:29:02 client-5 kernel: [] :ptlrpc:sec_gc_main+0x2cf/0x360 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] :ptlrpc:sec_gc_main+0x0/0x360 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: ll_capa S 00002b1c55064010 0 29054 1 30445 28909 (L-TLB) May 17 21:29:02 client-5 kernel: ffff81031a459e30 0000000000000046 0000000000000000 ffff81031d0a3e98 May 17 21:29:02 client-5 kernel: 0000000000000001 0000000000000001 ffff81025b211040 ffff81031977e7e0 May 17 21:29:02 client-5 kernel: 00003ffdfacacf98 0000000000001998 ffff81025b211228 000000028008b3c7 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] __wake_up+0x38/0x4f May 17 21:29:02 client-5 kernel: [] :lustre:capa_thread_main+0x4d2/0x15a0 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] audit_syscall_exit+0x336/0x362 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] :lustre:capa_thread_main+0x0/0x15a0 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: bash S 0000000000000001 0 29246 3677 31816 (NOTLB) May 17 21:29:02 client-5 kernel: ffff81032e5c7ee8 0000000000000082 0000000000000292 ffffffff80022205 May 17 21:29:02 client-5 kernel: ffff81033b5f8040 0000000000000001 ffff810321d6c0c0 ffff81017de92080 May 17 21:29:02 client-5 kernel: 0000401193a6c079 000000000000a280 ffff810321d6c2a8 0000000321d6c0c0 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] __up_read+0x19/0x7f May 17 21:29:02 client-5 kernel: [] do_wait+0xa03/0xad7 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: ldlm_bl_00 S ffff810001015120 0 30445 1 30446 29054 (L-TLB) May 17 21:29:02 client-5 kernel: ffff8102f4023ea0 0000000000000046 00000000ffffffff ffff81031585ab40 May 17 21:29:02 client-5 kernel: ffffc200112dd000 000000000000000a ffff81031977e7e0 ffff81010b764080 May 17 21:29:02 client-5 kernel: 0000400fd78c90c8 0000000000007a63 ffff81031977e9c8 0000000200000000 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] :ptlrpc:ldlm_bl_thread_main+0x18e/0x420 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] audit_syscall_exit+0x336/0x362 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] :ptlrpc:ldlm_bl_thread_main+0x0/0x420 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: ldlm_bl_01 S ffff810001004420 0 30446 1 30447 30445 (L-TLB) May 17 21:29:02 client-5 kernel: ffff81026a87dea0 0000000000000046 ffff81031585a900 0000000000000286 May 17 21:29:02 client-5 kernel: ffff81031585a900 0000000000000009 ffff81030318b0c0 ffffffff80308b60 May 17 21:29:02 client-5 kernel: 000040156e73dea6 000000000000b1de ffff81030318b2a8 0000000000000000 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] __wake_up_common+0x3e/0x68 May 17 21:29:02 client-5 kernel: [] :ptlrpc:ldlm_bl_thread_main+0x18e/0x420 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] audit_syscall_exit+0x336/0x362 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] :ptlrpc:ldlm_bl_thread_main+0x0/0x420 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: ldlm_cn_00 S ffff810331e7f860 0 30447 1 30448 30446 (L-TLB) May 17 21:29:02 client-5 kernel: ffff810319fc7df0 0000000000000046 ffffffff88784a68 0000000000000000 May 17 21:29:02 client-5 kernel: ffff810319fc7d80 0000000000000001 ffff81033a3bf100 ffff810331e7f860 May 17 21:29:02 client-5 kernel: 0000400cf73e6235 0000000000000b54 ffff81033a3bf2e8 0000000200000003 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] __wake_up+0x38/0x4f May 17 21:29:02 client-5 kernel: [] _spin_lock_bh+0x9/0x14 May 17 21:29:02 client-5 kernel: [] :libcfs:lc_watchdog_del_pending+0x16/0x70 May 17 21:29:02 client-5 kernel: [] :ptlrpc:ptlrpc_wait_event+0x1b9/0x3e0 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] __wake_up+0x38/0x4f May 17 21:29:02 client-5 kernel: [] :ptlrpc:ptlrpc_main+0x150c/0x1560 May 17 21:29:02 client-5 kernel: [] audit_syscall_exit+0x336/0x362 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] :ptlrpc:ptlrpc_main+0x0/0x1560 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: lc_watchdogd S ffff8103048011c8 0 30448 1 30449 30447 (L-TLB) May 17 21:29:02 client-5 kernel: ffff810302a7de60 0000000000000046 0000000000000000 0000000000000000 May 17 21:29:02 client-5 kernel: ffff810304801380 0000000000000001 ffff8103024350c0 ffff8101365687e0 May 17 21:29:02 client-5 kernel: 0000400cf73e33d7 0000000000000cf7 ffff8103024352a8 0000000200000000 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] __wake_up_common+0x3e/0x68 May 17 21:29:02 client-5 kernel: [] :libcfs:lcw_dispatch_main+0x166/0x890 May 17 21:29:02 client-5 kernel: [] autoremove_wake_function+0x0/0x2e May 17 21:29:02 client-5 kernel: [] audit_syscall_exit+0x336/0x362 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] :libcfs:lcw_dispatch_main+0x0/0x890 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: ldlm_cn_01 S 0000000000000000 0 30449 1 30450 30448 (L-TLB) May 17 21:29:02 client-5 kernel: ffff8103155eddf0 0000000000000046 0000000300000000 ffffffff88784a60 May 17 21:29:02 client-5 kernel: 0000000000000246 0000000000000001 ffff8101365687e0 ffff81033b5910c0 May 17 21:29:02 client-5 kernel: 0000400cf73e88ee 00000000000003ff ffff8101365689c8 00000002365687e0 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] _spin_lock_bh+0x9/0x14 May 17 21:29:02 client-5 kernel: [] :libcfs:lc_watchdog_del_pending+0x16/0x70 May 17 21:29:02 client-5 kernel: [] :ptlrpc:ptlrpc_wait_event+0x1b9/0x3e0 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] __wake_up+0x38/0x4f May 17 21:29:02 client-5 kernel: [] :ptlrpc:ptlrpc_main+0x150c/0x1560 May 17 21:29:02 client-5 kernel: [] audit_syscall_exit+0x336/0x362 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] :ptlrpc:ptlrpc_main+0x0/0x1560 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: ldlm_cb_00 S ffff810001015120 0 30450 1 30451 30449 (L-TLB) May 17 21:29:02 client-5 kernel: ffff8102e368fdf0 0000000000000046 0000400cf73ea378 0000000000000002 May 17 21:29:02 client-5 kernel: ffff810331e7f860 000000000000000a ffff81033b5910c0 ffff81010b764080 May 17 21:29:02 client-5 kernel: 0000400ff3022e9b 00000000000004de ffff81033b5912a8 0000000200000001 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] :ptlrpc:ptlrpc_wait_event+0x1b9/0x3e0 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] __wake_up+0x38/0x4f May 17 21:29:02 client-5 kernel: [] :ptlrpc:ptlrpc_main+0x150c/0x1560 May 17 21:29:02 client-5 kernel: [] audit_syscall_exit+0x336/0x362 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] :ptlrpc:ptlrpc_main+0x0/0x1560 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: ldlm_cb_01 S ffff810001015120 0 30451 1 30452 30450 (L-TLB) May 17 21:29:02 client-5 kernel: ffff810313a2bdf0 0000000000000046 0000000000000000 0000000000000000 May 17 21:29:02 client-5 kernel: ffffffff8898b478 000000000000000a ffff81025b2117a0 ffff81010b764080 May 17 21:29:02 client-5 kernel: 0000401158087ed0 0000000000001a03 ffff81025b211988 00000002d94c5648 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] __mod_timer+0x100/0x10f May 17 21:29:02 client-5 kernel: [] :ptlrpc:ptlrpc_wait_event+0x1b9/0x3e0 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] :ptlrpc:ptlrpc_main+0x150c/0x1560 May 17 21:29:02 client-5 kernel: [] audit_syscall_exit+0x336/0x362 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] :ptlrpc:ptlrpc_main+0x0/0x1560 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: ldlm_elt S ffff8102c4623ef0 0 30452 1 30453 30451 (L-TLB) May 17 21:29:02 client-5 kernel: ffff8102c4623eb0 0000000000000046 0000000000000500 ffff81013e897588 May 17 21:29:02 client-5 kernel: 0000000000000001 0000000000000001 ffff8102c8efa080 ffff810331e7f860 May 17 21:29:02 client-5 kernel: 0000400cf73f09f8 0000000000000c25 ffff8102c8efa268 000000028008b3c7 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] __wake_up+0x38/0x4f May 17 21:29:02 client-5 kernel: [] :ptlrpc:expired_lock_main+0x311/0x8b0 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] :ptlrpc:expired_lock_main+0x0/0x8b0 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: ldlm_poold S ffff81000101d7a0 0 30453 1 30456 30452 (L-TLB) May 17 21:29:02 client-5 kernel: ffff81031ffe7e60 0000000000000046 ffffffff889f9d20 ffffffff8891c2c8 May 17 21:29:02 client-5 kernel: ffff8103191d90a0 000000000000000a ffff8102d9408080 ffff81010b770100 May 17 21:29:02 client-5 kernel: 000042763eed01c7 00000000000023e3 ffff8102d9408268 000000030b7e4068 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] :ptlrpc:ldlm_cli_pool_pop_slv+0x68/0x80 May 17 21:29:02 client-5 kernel: [] schedule_timeout+0x8a/0xad May 17 21:29:02 client-5 kernel: [] process_timeout+0x0/0x5 May 17 21:29:02 client-5 kernel: [] :ptlrpc:ldlm_pools_thread_main+0x1c4/0x2b3 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] :ptlrpc:ldlm_pools_thread_main+0x0/0x2b3 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: ll_close S ffff81031a0a93f0 0 30456 1 31115 30453 (L-TLB) May 17 21:29:02 client-5 kernel: ffff810173109e80 0000000000000046 ffffffff88c483d2 ffff810173109db0 May 17 21:29:02 client-5 kernel: 00000000ffffffff 0000000000000006 ffff810318e3b820 ffff810331e7f860 May 17 21:29:02 client-5 kernel: 0000400e24274c37 0000000000001de2 ffff810318e3ba08 0000000200000000 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] __wake_up_common+0x3e/0x68 May 17 21:29:02 client-5 kernel: [] :lustre:ll_close_thread+0x169/0xfb0 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] audit_syscall_exit+0x336/0x362 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] :lustre:ll_close_thread+0x0/0xfb0 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: ldlm_bl_02 S ffff810001015120 0 31115 1 30456 (L-TLB) May 17 21:29:02 client-5 kernel: ffff8102c2b97ea0 0000000000000046 00000000ffffffff ffff8103229826c0 May 17 21:29:02 client-5 kernel: ffffc200112dd000 0000000000000009 ffff81033fb6f820 ffff81010b764080 May 17 21:29:02 client-5 kernel: 0000400fd78a40c7 0000000000006f3a ffff81033fb6fa08 0000000200000000 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] :ptlrpc:ldlm_bl_thread_main+0x18e/0x420 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] audit_syscall_exit+0x336/0x362 May 17 21:29:02 client-5 kernel: [] child_rip+0xa/0x11 May 17 21:29:02 client-5 kernel: [] :ptlrpc:ldlm_bl_thread_main+0x0/0x420 May 17 21:29:02 client-5 kernel: [] child_rip+0x0/0x11 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: bash S ffff810001004420 0 31816 29246 32745 31817 (NOTLB) May 17 21:29:02 client-5 kernel: ffff8102ff5f1ee8 0000000000000082 0000000000000292 ffffffff80022205 May 17 21:29:02 client-5 kernel: ffff8102f2791b80 0000000000000002 ffff81033b657860 ffffffff80308b60 May 17 21:29:02 client-5 kernel: 0000401690fde012 000000000000da35 ffff81033b657a48 000000003b657860 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] __up_read+0x19/0x7f May 17 21:29:02 client-5 kernel: [] do_wait+0xa03/0xad7 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: tee S ffff81000101d7a0 0 31817 29246 31816 (NOTLB) May 17 21:29:02 client-5 kernel: ffff81026c17fdf8 0000000000000086 0000000000000000 ffff81031ec2e100 May 17 21:29:02 client-5 kernel: ffffffff8008c937 0000000000000007 ffff81017de92080 ffff81010b770100 May 17 21:29:02 client-5 kernel: 0000401690fd9f42 00000000000064d3 ffff81017de92268 0000000311c6ae40 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] enqueue_task+0x41/0x56 May 17 21:29:02 client-5 kernel: [] generic_file_aio_write+0x70/0xc1 May 17 21:29:02 client-5 kernel: [] pipe_wait+0x69/0x93 May 17 21:29:02 client-5 kernel: [] autoremove_wake_function+0x0/0x2e May 17 21:29:02 client-5 kernel: [] mutex_lock+0xd/0x1d May 17 21:29:02 client-5 kernel: [] pipe_readv+0x324/0x3a9 May 17 21:29:02 client-5 kernel: [] pipe_read+0x1a/0x1f May 17 21:29:02 client-5 kernel: [] vfs_read+0xcb/0x171 May 17 21:29:02 client-5 kernel: [] sys_read+0x45/0x6e May 17 21:29:02 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:29:02 client-5 kernel: May 17 21:29:02 client-5 kernel: dd S ffff81000100caa0 0 32745 31816 (NOTLB) May 17 21:29:02 client-5 kernel: ffff8102fb8df558 0000000000000086 ffff810319f1c6c0 ffff8102ff3fc6a8 May 17 21:29:02 client-5 kernel: ffff810319f1c710 0000000000000001 ffff8101f9b0e040 ffff81010b70e100 May 17 21:29:02 client-5 kernel: 000040169dd977a9 0000000000200a54 ffff8101f9b0e228 0000000188b0dc18 May 17 21:29:02 client-5 kernel: Call Trace: May 17 21:29:02 client-5 kernel: [] :osc:osc_queue_async_io+0xa23/0xed0 May 17 21:29:02 client-5 kernel: [] radix_tree_insert+0xb3/0x190 May 17 21:29:02 client-5 kernel: [] :libcfs:libcfs_debug_vmsg2+0x75b/0x9f0 May 17 21:29:02 client-5 kernel: [] default_wake_function+0x0/0xe May 17 21:29:02 client-5 kernel: [] :osc:osc_page_cache_add+0x120/0x1d0 May 17 21:29:02 client-5 kernel: [] :obdclass:cl_page_invoke+0xb8/0x170 May 17 21:29:02 client-5 kernel: [] radix_tree_node_alloc+0x18/0x57 May 17 21:29:02 client-5 kernel: [] :obdclass:cl_page_cache_add+0x90/0x1d0 May 17 21:29:02 client-5 kernel: [] :lustre:vvp_write_pending+0x8c/0x130 May 17 21:29:02 client-5 kernel: [] :lustre:vvp_io_commit_write+0x26f/0x500 May 17 21:29:02 client-5 kernel: [] :lov:lov_io_prepare_write+0x12c/0x1c0 May 17 21:29:02 client-5 kernel: [] :obdclass:cl_page_at_trusted+0x67/0x1c0 May 17 21:29:02 client-5 kernel: [] :obdclass:cl_io_commit_write+0x125/0x1c0 May 17 21:29:02 client-5 kernel: [] :obdclass:cl_env_get+0x25/0x300 May 17 21:29:02 client-5 kernel: [] :lustre:ll_commit_write+0x1b4/0x280 May 17 21:29:02 client-5 kernel: [] :lustre:ll_prepare_write+0xfb/0x1b0 May 17 21:29:02 client-5 kernel: [] generic_file_buffered_write+0x4df/0x675 May 17 21:29:02 client-5 kernel: [] __generic_file_aio_write_nolock+0x369/0x3b6 May 17 21:29:02 client-5 kernel: [] __generic_file_write_nolock+0x8f/0xa8 May 17 21:29:02 client-5 kernel: [] :obdclass:cl_lock_trace0+0x13e/0x180 May 17 21:29:02 client-5 kernel: [] autoremove_wake_function+0x0/0x2e May 17 21:29:02 client-5 kernel: [] mutex_lock+0xd/0x1d May 17 21:29:02 client-5 kernel: [] generic_file_writev+0x48/0xa3 May 17 21:29:02 client-5 kernel: [] :lustre:vvp_io_write_start+0x157/0x240 May 17 21:29:02 client-5 kernel: [] :obdclass:cl_io_start+0xbc/0x160 May 17 21:29:02 client-5 kernel: [] :obdclass:cl_io_loop+0xad/0x1a0 May 17 21:29:02 client-5 kernel: [] :lustre:ll_file_io_generic+0x459/0x4a0 May 17 21:29:02 client-5 kernel: [] thread_return+0x62/0xfe May 17 21:29:02 client-5 kernel: [] :obdclass:cl_env_get+0x25/0x300 May 17 21:29:02 client-5 kernel: [] :lustre:ll_file_writev+0x1e0/0x280 May 17 21:29:02 client-5 kernel: [] :obdclass:cl_env_get+0x162/0x300 May 17 21:29:02 client-5 kernel: [] :obdclass:cl_env_put+0x279/0x2d0 May 17 21:29:02 client-5 kernel: [] :lustre:ll_file_write+0x164/0x210 May 17 21:29:02 client-5 kernel: [] read_zero+0x1cd/0x226 May 17 21:29:02 client-5 kernel: [] vfs_write+0xce/0x174 May 17 21:29:02 client-5 kernel: [] sys_write+0x45/0x6e May 17 21:29:02 client-5 kernel: [] tracesys+0xd5/0xe0 May 17 21:29:02 client-5 kernel: May 17 21:33:00 client-5 xinetd[3391]: EXIT: login status=0 pid=11742 duration=5251(sec) May 17 21:33:13 client-5 kernel: Lustre: 28904:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369115429179253 sent from lustre-OST0001-osc-ffff81031957b000 to NID 192.168.4.129@o2ib has timed out for slow reply: [sent 1305692492] [real_sent 1305692492] [current 1305693193] [deadline 701s] [delay 0s] req@ffff8102eb50ec00 x1369115429179253/t0(0) o-1->lustre-OST0001_UUID@192.168.4.129@o2ib:6/4 lens 456/416 e 6 to 1 dl 1305693193 ref 2 fl Rpc:X/ffffffff/ffffffff rc 0/-1 May 17 21:33:13 client-5 kernel: Lustre: 28904:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 1 previous similar message May 17 21:33:13 client-5 kernel: Lustre: lustre-OST0001-osc-ffff81031957b000: Connection to service lustre-OST0001 via nid 192.168.4.129@o2ib was lost; in progress operations using this service will wait for recovery to complete. May 17 21:33:13 client-5 kernel: Lustre: 28904:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1369115429179254 sent from lustre-OST0001-osc-ffff81031957b000 to NID 192.168.4.129@o2ib has timed out for slow reply: [sent 1305692492] [real_sent 1305692492] [current 1305693193] [deadline 701s] [delay 0s] req@ffff8102eb50e800 x1369115429179254/t0(0) o-1->lustre-OST0001_UUID@192.168.4.129@o2ib:6/4 lens 456/416 e 6 to 1 dl 1305693193 ref 2 fl Rpc:X/ffffffff/ffffffff rc 0/-1 May 17 21:33:13 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.129@o2ib. The ost_connect operation failed with -16 May 17 21:33:13 client-5 xinetd[3391]: START: login pid=494 from=10.10.0.1 May 17 21:33:18 client-5 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.129@o2ib. The ost_connect operation failed with -16 May 17 21:33:23 client-5 kernel: Lustre: lustre-OST0001-osc-ffff81031957b000: Connection restored to service lustre-OST0001 using nid 192.168.4.129@o2ib.