Initializing cgroup subsys cpuset
Initializing cgroup subsys cpu
Linux version 2.6.32-279.2.1.el6.dfb4e73 (root@h221f) (gcc version 4.4.6 20120305 (Red Hat 4.4.6-4) (GCC) ) #3 SMP Tue Sep 11 17:33:28 CST 2012
Command line: ro root=/dev/mapper/vg_h221f-lv_root rd_NO_LUKS LANG=en_US.UTF-8 rd_LVM_LV=vg_h221f/lv_swap rd_NO_MD quiet SYSFONT=latarcyrheb-sun16 rhgb crashkernel=auto rd_LVM_LV=vg_h221f/lv_root  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM divider=10 console=ttyS0
KERNEL supported cpus:
  Intel GenuineIntel
  AMD AuthenticAMD
  Centaur CentaurHauls
Disabled fast string operations
BIOS-provided physical RAM map:
 BIOS-e820: 0000000000000000 - 000000000009fc00 (usable)
 BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved)
 BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved)
 BIOS-e820: 0000000000100000 - 000000003fff0000 (usable)
 BIOS-e820: 000000003fff0000 - 0000000040000000 (ACPI data)
 BIOS-e820: 00000000fffc0000 - 0000000100000000 (reserved)
DMI 2.5 present.
SMBIOS version 2.5 @ 0xFFF30
DMI: innotek GmbH VirtualBox, BIOS VirtualBox 12/01/2006
e820 update range: 0000000000000000 - 0000000000001000 (usable) ==> (reserved)
e820 remove range: 00000000000a0000 - 0000000000100000 (usable)
last_pfn = 0x3fff0 max_arch_pfn = 0x400000000
MTRR default type: uncachable
MTRR variable ranges disabled:
x86 PAT enabled: cpu 0, old 0x7040600070406, new 0x7010600070106
CPU MTRRs all blank - virtualized system.
initial memory mapped : 0 - 20000000
init_memory_mapping: 0000000000000000-000000003fff0000
 0000000000 - 003fe00000 page 2M
 003fe00000 - 003fff0000 page 4k
kernel direct mapping tables up to 3fff0000 @ 8000-b000
RAMDISK: 374dc000 - 37fef985
ACPI: RSDP 00000000000e0000 00024 (v02 VBOX  )
ACPI: XSDT 000000003fff0030 0003C (v01 VBOX   VBOXXSDT 00000001 ASL  00000061)
ACPI: FACP 000000003fff00f0 000F4 (v04 VBOX   VBOXFACP 00000001 ASL  00000061)
ACPI: DSDT 000000003fff0470 01A63 (v01 VBOX   VBOXBIOS 00000002 INTL 20100528)
ACPI: FACS 000000003fff0200 00040
ACPI: APIC 000000003fff0240 00054 (v02 VBOX   VBOXAPIC 00000001 ASL  00000061)
ACPI: SSDT 000000003fff02a0 001CC (v01 VBOX   VBOXCPUT 00000002 INTL 20100528)
ACPI: Local APIC address 0xfee00000
No NUMA configuration found
Faking a node at 0000000000000000-000000003fff0000
Bootmem setup node 0 0000000000000000-000000003fff0000
  NODE_DATA [0000000000009000 - 000000000003cfff]
  bootmap [000000000003d000 -  0000000000044fff] pages 8
(7 early reservations) ==> bootmem [0000000000 - 003fff0000]
  #0 [0000000000 - 0000001000]   BIOS data page ==> [0000000000 - 0000001000]
  #1 [0000006000 - 0000008000]       TRAMPOLINE ==> [0000006000 - 0000008000]
  #2 [0001000000 - 0002012024]    TEXT DATA BSS ==> [0001000000 - 0002012024]
  #3 [00374dc000 - 0037fef985]          RAMDISK ==> [00374dc000 - 0037fef985]
  #4 [000009fc00 - 0000100000]    BIOS reserved ==> [000009fc00 - 0000100000]
  #5 [0002013000 - 0002013079]              BRK ==> [0002013000 - 0002013079]
  #6 [0000008000 - 0000009000]          PGTABLE ==> [0000008000 - 0000009000]
found SMP MP-table at [ffff88000009fff0] 9fff0
 [ffffea0000000000-ffffea0000dfffff] PMD -> [ffff880002600000-ffff8800033fffff] on node 0
Zone PFN ranges:
  DMA      0x00000001 -> 0x00001000
  DMA32    0x00001000 -> 0x00100000
  Normal   0x00100000 -> 0x00100000
Movable zone start PFN for each node
early_node_map[2] active PFN ranges
    0: 0x00000001 -> 0x0000009f
    0: 0x00000100 -> 0x0003fff0
On node 0 totalpages: 262030
  DMA zone: 56 pages used for memmap
  DMA zone: 100 pages reserved
  DMA zone: 3842 pages, LIFO batch:0
  DMA32 zone: 3528 pages used for memmap
  DMA32 zone: 254504 pages, LIFO batch:31
ACPI: PM-Timer IO Port: 0x4008
ACPI: Local APIC address 0xfee00000
ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
ACPI: IOAPIC (id[0x01] address[0xfec00000] gsi_base[0])
IOAPIC[0]: apic_id 1, version 17, address 0xfec00000, GSI 0-23
ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
ACPI: IRQ0 used by override.
ACPI: IRQ2 used by override.
ACPI: IRQ9 used by override.
Using ACPI (MADT) for SMP configuration information
SMP: Allowing 1 CPUs, 0 hotplug CPUs
nr_irqs_gsi: 24
PM: Registered nosave memory: 000000000009f000 - 00000000000a0000
PM: Registered nosave memory: 00000000000a0000 - 00000000000f0000
PM: Registered nosave memory: 00000000000f0000 - 0000000000100000
Allocating PCI resources starting at 40000000 (gap: 40000000:bffc0000)
Booting paravirtualized kernel on bare hardware
NR_CPUS:4096 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1
PERCPU: Embedded 31 pages/cpu @ffff880002200000 s94424 r8192 d24360 u2097152
pcpu-alloc: s94424 r8192 d24360 u2097152 alloc=1*2097152
pcpu-alloc: [0] 0 
Built 1 zonelists in Node order, mobility grouping on.  Total pages: 258346
Policy zone: DMA32
Kernel command line: ro root=/dev/mapper/vg_h221f-lv_root rd_NO_LUKS LANG=en_US.UTF-8 rd_LVM_LV=vg_h221f/lv_swap rd_NO_MD quiet SYSFONT=latarcyrheb-sun16 rhgb  rd_LVM_LV=vg_h221f/lv_root  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM divider=10 console=ttyS0
PID hash table entries: 4096 (order: 3, 32768 bytes)
Checking aperture...
No AGP bridge found
Memory: 1005352k/1048512k available (5152k kernel code, 392k absent, 42768k reserved, 7165k data, 1260k init)
Hierarchical RCU implementation.
NR_IRQS:33024 nr_irqs:256
Console: colour VGA+ 80x25
console [ttyS0] enabled
allocated 8388608 bytes of page_cgroup
please try 'cgroup_disable=memory' option if you don't want memory cgroups
Fast TSC calibration using PIT
Detected 3227.496 MHz processor.
Calibrating delay loop (skipped), value calculated using timer frequency.. 6454.99 BogoMIPS (lpj=3227496)
pid_max: default: 32768 minimum: 301
Security Framework initialized
SELinux:  Initializing.
SELinux:  Starting in permissive mode
Dentry cache hash table entries: 131072 (order: 8, 1048576 bytes)
Inode-cache hash table entries: 65536 (order: 7, 524288 bytes)
Mount-cache hash table entries: 256
Initializing cgroup subsys ns
Initializing cgroup subsys cpuacct
Initializing cgroup subsys memory
Initializing cgroup subsys devices
Initializing cgroup subsys freezer
Initializing cgroup subsys net_cls
Initializing cgroup subsys blkio
Initializing cgroup subsys perf_event
Initializing cgroup subsys net_prio
Disabled fast string operations
mce: CPU supports 0 MCE banks
using mwait in idle threads.
SMP alternatives: switching to UP code
Freeing SMP alternatives: 34k freed
ACPI: Core revision 20090903
ftrace: converting mcount calls to 0f 1f 44 00 00
ftrace: allocating 21018 entries in 83 pages
Setting APIC routing to flat
..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
..MP-BIOS bug: 8254 timer not connected to IO-APIC
...trying to set up timer (IRQ0) through the 8259A ...
..... (found apic 0 pin 2) ...
....... works.
CPU0: Intel(R) Core(TM) i3 CPU         550  @ 3.20GHz stepping 05
APIC calibration not consistent with PM-Timer: 145ms instead of 100ms
APIC delta adjusted to PM-Timer: 6241373 (9112077)
Performance Events: unsupported p6 CPU model 37 no PMU driver, software events only.
NMI watchdog disabled (cpu0): hardware events not enabled
Brought up 1 CPUs
Total of 1 processors activated (6454.99 BogoMIPS).
sizeof(vma)=200 bytes
sizeof(page)=56 bytes
sizeof(inode)=592 bytes
sizeof(dentry)=192 bytes
sizeof(ext3inode)=800 bytes
sizeof(buffer_head)=104 bytes
sizeof(skbuff)=232 bytes
sizeof(task_struct)=2648 bytes
devtmpfs: initialized
regulator: core version 0.5
NET: Registered protocol family 16
ACPI: bus type pci registered
PCI: Using configuration type 1 for base access
bio: create slab <bio-0> at 0
ACPI: EC: Look up EC in DSDT
ACPI: Executed 1 blocks of module-level executable AML code
ACPI: Interpreter enabled
ACPI: (supports S0 S5)
ACPI: Using IOAPIC for interrupt routing
ACPI: No dock devices found.
HEST: Table not found.
PCI: Ignoring host bridge windows from ACPI; if necessary, use "pci=use_crs" and report a bug
ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])
pci_root PNP0A03:00: host bridge window [io  0x0000-0x0cf7] (ignored)
pci_root PNP0A03:00: host bridge window [io  0x0d00-0xffff] (ignored)
pci_root PNP0A03:00: host bridge window [mem 0x000a0000-0x000bffff] (ignored)
pci_root PNP0A03:00: host bridge window [mem 0x40000000-0xffdfffff] (ignored)
pci 0000:00:01.1: reg 20: [io  0xd000-0xd00f]
pci 0000:00:02.0: reg 10: [mem 0xe0000000-0xe0ffffff pref]
pci 0000:00:03.0: reg 10: [mem 0xf0000000-0xf001ffff]
pci 0000:00:03.0: reg 18: [io  0xd010-0xd017]
pci 0000:00:04.0: reg 10: [io  0xd020-0xd03f]
pci 0000:00:04.0: reg 14: [mem 0xf0400000-0xf07fffff]
pci 0000:00:04.0: reg 18: [mem 0xf0800000-0xf0803fff pref]
pci 0000:00:05.0: reg 10: [io  0xd100-0xd1ff]
pci 0000:00:05.0: reg 14: [io  0xd200-0xd23f]
pci 0000:00:06.0: reg 10: [mem 0xf0804000-0xf0804fff]
pci 0000:00:08.0: reg 10: [mem 0xf0820000-0xf083ffff]
pci 0000:00:08.0: reg 18: [io  0xd240-0xd247]
pci 0000:00:0d.0: reg 10: [io  0xd248-0xd24f]
pci 0000:00:0d.0: reg 18: [io  0xd258-0xd25f]
pci 0000:00:0d.0: reg 20: [io  0xd270-0xd27f]
pci 0000:00:0d.0: reg 24: [mem 0xf0840000-0xf0841fff]
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0._PRT]
ACPI: PCI Interrupt Link [LNKA] (IRQs *5 9 10 11)
ACPI: PCI Interrupt Link [LNKB] (IRQs 5 9 10 *11)
ACPI: PCI Interrupt Link [LNKC] (IRQs 5 9 *10 11)
ACPI: PCI Interrupt Link [LNKD] (IRQs 5 *9 10 11)
vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none
vgaarb: loaded
vgaarb: bridge control possible 0000:00:02.0
SCSI subsystem initialized
libata version 3.00 loaded.
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
PCI: Using ACPI for IRQ routing
PCI: old code would have set cacheline size to 32 bytes, but clflush_size = 64
PCI: pci_cache_line_size set to 64 bytes
NetLabel: Initializing
NetLabel:  domain hash size = 128
NetLabel:  protocols = UNLABELED CIPSOv4
NetLabel:  unlabeled traffic allowed by default
Switching to clocksource jiffies
pnp: PnP ACPI init
ACPI: bus type pnp registered
pnp 00:00: [io  0x0cf8-0x0cff]
pnp 00:00: Plug and Play ACPI device, IDs PNP0a03 (active)
pnp 00:01: [io  0x0060]
pnp 00:01: [io  0x0064]
pnp 00:01: [irq 1]
pnp 00:01: Plug and Play ACPI device, IDs PNP0303 (active)
pnp 00:02: [io  0x0000-0x000f]
pnp 00:02: [io  0x0080-0x008f]
pnp 00:02: [io  0x00c0-0x00df]
pnp 00:02: [dma 4]
pnp 00:02: Plug and Play ACPI device, IDs PNP0200 (active)
pnp 00:03: [irq 12]
pnp 00:03: Plug and Play ACPI device, IDs PNP0f03 (active)
pnp 00:04: [io  0x0378-0x037f]
pnp 00:04: [io  0x0778-0x077f]
pnp 00:04: [irq 7]
pnp 00:04: Plug and Play ACPI device, IDs PNP0400 (active)
pnp: PnP ACPI: found 5 devices
ACPI: ACPI bus type pnp unregistered
Switching to clocksource acpi_pm
PCI: max bus depth: 0 pci_try_num: 1
pci_bus 0000:00: resource 0 [io  0x0000-0xffff]
pci_bus 0000:00: resource 1 [mem 0x00000000-0xfffffffff]
NET: Registered protocol family 2
IP route cache hash table entries: 32768 (order: 6, 262144 bytes)
TCP established hash table entries: 131072 (order: 9, 2097152 bytes)
TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
TCP: Hash tables configured (established 131072 bind 65536)
TCP reno registered
NET: Registered protocol family 1
pci 0000:00:00.0: Limiting direct PCI/PCI transfers
pci 0000:00:01.0: Activating ISA DMA hang workarounds
pci 0000:00:02.0: Boot video device
  alloc irq_desc for 22 on node -1
  alloc kstat_irqs on node -1
pci 0000:00:06.0: PCI INT A -> GSI 22 (level, low) -> IRQ 22
pci 0000:00:06.0: PCI INT A disabled
Trying to unpack rootfs image as initramfs...
hrtimer: interrupt took 3703056 ns
Freeing initrd memory: 11342k freed
platform rtc_cmos: registered platform RTC device (no PNP device found)
audit: initializing netlink socket (disabled)
type=2000 audit(1350264638.329:1): initialized
HugeTLB registered 2 MB page size, pre-allocated 0 pages
VFS: Disk quotas dquot_6.5.2
Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
msgmni has been set to 1985
SELinux:  Registering netfilter hooks
alg: No test for stdrng (krng)
ksign: Installing public key data
Loading keyring
- Added public key BAEB3EA7672E51F8
- User ID: CentOS (Kernel Module GPG key)
Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252)
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered (default)
io scheduler cfq registered
pci_hotplug: PCI Hot Plug PCI Core version: 0.5
pciehp: PCI Express Hot Plug Controller Driver version: 0.4
acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
ACPI: AC Adapter [AC] (on-line)
input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0
ACPI: Power Button [PWRF]
input: Sleep Button as /devices/LNXSYSTM:00/LNXSLPBN:00/input/input1
ACPI: Sleep Button [SLPF]
ACPI: acpi_idle registered with cpuidle
ERST: Table is not found!
GHES: HEST is not enabled!
Non-volatile memory driver v1.3
Linux agpgart interface v0.103
crash memory driver: version 1.1
Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
brd: module loaded
loop: module loaded
input: Macintosh mouse button emulation as /devices/virtual/input/input2
Fixed MDIO Bus: probed
ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
ohci_hcd 0000:00:06.0: PCI INT A -> GSI 22 (level, low) -> IRQ 22
ohci_hcd 0000:00:06.0: setting latency timer to 64
ohci_hcd 0000:00:06.0: OHCI Host Controller
ohci_hcd 0000:00:06.0: new USB bus registered, assigned bus number 1
ohci_hcd 0000:00:06.0: irq 22, io mem 0xf0804000
usb usb1: New USB device found, idVendor=1d6b, idProduct=0001
usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
usb usb1: Product: OHCI Host Controller
usb usb1: Manufacturer: Linux 2.6.32-279.2.1.el6.dfb4e73 ohci_hcd
usb usb1: SerialNumber: 0000:00:06.0
usb usb1: configuration #1 chosen from 1 choice
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 8 ports detected
uhci_hcd: USB Universal Host Controller Interface driver
PNP: PS/2 Controller [PNP0303:PS2K,PNP0f03:PS2M] at 0x60,0x64 irq 1,12
serio: i8042 KBD port at 0x60,0x64 irq 1
serio: i8042 AUX port at 0x60,0x64 irq 12
mice: PS/2 mouse device common for all mice
input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input3
rtc_cmos rtc_cmos: rtc core: registered rtc_cmos as rtc0
rtc0: alarms up to one day, 114 bytes nvram
cpuidle: using governor ladder
cpuidle: using governor menu
EFI Variables Facility v0.08 2004-May-17
input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input4
usbcore: registered new interface driver hiddev
usbcore: registered new interface driver usbhid
usbhid: v2.6:USB HID core driver
TCP cubic registered
Initializing XFRM netlink socket
NET: Registered protocol family 17
registered taskstats version 1
rtc_cmos rtc_cmos: setting system clock to 2012-10-15 01:30:39 UTC (1350264639)
Initalizing network drop monitor service
Freeing unused kernel memory: 1260k freed
Write protecting the kernel read-only data: 10240k
Freeing unused kernel memory: 972k freed
Freeing unused kernel memory: 1732k freed
dracut: dracut-004-256.el6
device-mapper: uevent: version 1.0.3
device-mapper: ioctl: 4.22.6-ioctl (2011-10-19) initialised: dm-devel@redhat.com
udev: starting version 147
dracut: Starting plymouth daemon
ata_piix 0000:00:01.1: version 2.13
ata_piix 0000:00:01.1: setting latency timer to 64
scsi0 : ata_piix
scsi1 : ata_piix
ata1: PATA max UDMA/33 cmd 0x1f0 ctl 0x3f6 bmdma 0xd000 irq 14
ata2: PATA max UDMA/33 cmd 0x170 ctl 0x376 bmdma 0xd008 irq 15
ata2.00: ATAPI: VBOX CD-ROM, 1.0, max UDMA/133
ata2.00: configured for UDMA/33
scsi 1:0:0:0: CD-ROM            VBOX     CD-ROM           1.0  PQ: 0 ANSI: 5
ahci 0000:00:0d.0: version 3.0
  alloc irq_desc for 21 on node -1
  alloc kstat_irqs on node -1
ahci 0000:00:0d.0: PCI INT A -> GSI 21 (level, low) -> IRQ 21
ahci: SSS flag set, parallel bus scan disabled
ahci 0000:00:0d.0: AHCI 0001.0100 32 slots 1 ports 3 Gbps 0x1 impl SATA mode
ahci 0000:00:0d.0: flags: 64bit ncq stag only ccc 
ahci 0000:00:0d.0: setting latency timer to 64
scsi2 : ahci
ata3: SATA max UDMA/133 abar m8192@0xf0840000 port 0xf0840100 irq 21
ata3: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
ata3.00: ATA-6: VBOX HARDDISK, 1.0, max UDMA/133
ata3.00: 83886080 sectors, multi 128: LBA48 NCQ (depth 31/32)
ata3.00: configured for UDMA/133
scsi 2:0:0:0: Direct-Access     ATA      VBOX HARDDISK    1.0  PQ: 0 ANSI: 5
sr0: scsi3-mmc drive: 32x/32x xa/form2 tray
Uniform CD-ROM driver Revision: 3.20
sr 1:0:0:0: Attached scsi CD-ROM sr0
sd 2:0:0:0: [sda] 83886080 512-byte logical blocks: (42.9 GB/40.0 GiB)
sd 2:0:0:0: [sda] Write Protect is off
sd 2:0:0:0: [sda] Mode Sense: 00 3a 00 00
sd 2:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
 sda: sda1 sda2
sd 2:0:0:0: [sda] Attached SCSI disk
dracut: Scanning devices sda2  for LVM logical volumes vg_h221f/lv_swap vg_h221f/lv_root 
dracut: inactive '/dev/vg_h221f/lv_root' [37.54 GiB] inherit
dracut: inactive '/dev/vg_h221f/lv_swap' [1.97 GiB] inherit
EXT4-fs (dm-0): INFO: recovery required on readonly filesystem
EXT4-fs (dm-0): write access will be enabled during recovery
Refined TSC clocksource calibration: 3226.590 MHz.
Switching to clocksource tsc
EXT4-fs (dm-0): recovery complete
EXT4-fs (dm-0): mounted filesystem with ordered data mode. Opts: 
dracut: Mounted root filesystem /dev/mapper/vg_h221f-lv_root
SELinux:  Disabled at runtime.
SELinux:  Unregistering netfilter hooks
type=1404 audit(1350264642.972:2): selinux=0 auid=4294967295 ses=4294967295
dracut: 
dracut: Switching root
udev: starting version 147
sr 1:0:0:0: Attached scsi generic sg0 type 5
sd 2:0:0:0: Attached scsi generic sg1 type 0
e1000: Intel(R) PRO/1000 Network Driver - version 7.3.21-k8-NAPI
e1000: Copyright (c) 1999-2006 Intel Corporation.
  alloc irq_desc for 19 on node -1
  alloc kstat_irqs on node -1
e1000 0000:00:03.0: PCI INT A -> GSI 19 (level, low) -> IRQ 19
e1000 0000:00:03.0: setting latency timer to 64
e1000 0000:00:03.0: eth0: (PCI:33MHz:32-bit) 08:00:27:ea:57:a0
e1000 0000:00:03.0: eth0: Intel(R) PRO/1000 Network Connection
  alloc irq_desc for 16 on node -1
  alloc kstat_irqs on node -1
e1000 0000:00:08.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
e1000 0000:00:08.0: setting latency timer to 64
e1000 0000:00:08.0: eth1: (PCI:33MHz:32-bit) 08:00:27:26:28:5d
e1000 0000:00:08.0: eth1: Intel(R) PRO/1000 Network Connection
Intel ICH 0000:00:05.0: PCI INT A -> GSI 21 (level, low) -> IRQ 21
Intel ICH 0000:00:05.0: setting latency timer to 64
intel8x0_measure_ac97_clock: measured 50264 usecs (3900 samples)
intel8x0: measured clock 77590 rejected
intel8x0_measure_ac97_clock: measured 50861 usecs (3899 samples)
intel8x0: measured clock 76659 rejected
intel8x0_measure_ac97_clock: measured 50596 usecs (3900 samples)
intel8x0: measured clock 77081 rejected
intel8x0: clocking to 48000
piix4_smbus 0000:00:07.0: SMBus base address uninitialized - upgrade BIOS or use force_addr=0xaddr
parport_pc 00:04: reported by Plug and Play ACPI
ppdev: user-space parallel port driver
EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: 
Adding 2064376k swap on /dev/mapper/vg_h221f-lv_swap.  Priority:-1 extents:1 across:2064376k 
NET: Registered protocol family 10
lo: Disabled Privacy Extensions
ADDRCONF(NETDEV_UP): eth1: link is not ready
e1000: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
RPC: Registered named UNIX socket transport module.
RPC: Registered udp transport module.
RPC: Registered tcp transport module.
RPC: Registered tcp NFSv4.1 backchannel transport module.
Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
NFSD: starting 90-second grace period
eth1: no IPv6 routers present
LNet: HW CPU cores: 1, npartitions: 1
alg: No test for crc32 (crc32-table)
alg: No test for adler32 (adler32-zlib)
padlock: VIA PadLock Hash Engine not detected.
Lustre: Lustre: Build Version: 2.3.53-gde2a0a6-CHANGED-2.6.32-279.2.1.el6.dfb4e73
LNet: Added LNI 192.168.56.4@tcp [8/256/0/180]
LNet: Accept secure, port 988
Lustre: Echo OBD driver; http://www.lustre.org/
LDISKFS-fs (loop0): mounted filesystem with ordered data mode. quota=on. Opts: 
LDISKFS-fs (loop0): mounted filesystem with ordered data mode. quota=on. Opts: 
LDISKFS-fs (loop0): mounted filesystem with ordered data mode. quota=on. Opts: 
LDISKFS-fs (loop0): mounted filesystem with ordered data mode. quota=on. Opts: 
Lustre: MGC192.168.56.4@tcp: Reactivating import
Lustre: Setting parameter lustre-MDT0000-mdtlov.lov.stripesize in log lustre-MDT0000
LustreError: 27370:0:(mgc_request.c:248:do_config_log_add()) failed processing sptlrpc log: -2
cannot allocate a tage (14)
cannot allocate a tage (14)
cannot allocate a tage (14)
cannot allocate a tage (14)
cannot allocate a tage (14)
cannot allocate a tage (14)
cannot allocate a tage (14)
cannot allocate a tage (14)
cannot allocate a tage (14)
cannot allocate a tage (14)
Lustre: lustre-MDT0000: new disk, initializing
LustreError: 27412:0:(sec_config.c:1024:sptlrpc_target_local_copy_conf()) missing llog context
Lustre: 27412:0:(mdt_lproc.c:418:lprocfs_wr_identity_upcall()) lustre-MDT0000: identity upcall set to /root/lustre-master/lustre/tests/../utils/l_getidentity
Lustre: lustre-MDT0000: Temporarily refusing client connection from 0@lo
LustreError: 11-0: an error occurred while communicating with 0@lo. The mds_connect operation failed with -11
Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space
LDISKFS-fs (loop1): mounted filesystem with ordered data mode. quota=on. Opts: 
Lustre: lustre-OST0000: new disk, initializing
Lustre: 27549:0:(ofd_fs.c:271:ofd_groups_init()) lustre-OST0000: 1 groups initialized
LustreError: 11-0: an error occurred while communicating with 0@lo. The mds_connect operation failed with -11
LDISKFS-fs (loop2): mounted filesystem with ordered data mode. quota=on. Opts: 
Lustre: lustre-OST0001: new disk, initializing
Lustre: 27649:0:(ofd_fs.c:271:ofd_groups_init()) lustre-OST0001: 1 groups initialized
__ratelimit: 109 callbacks suppressed
cannot allocate a tage (78)
cannot allocate a tage (78)
cannot allocate a tage (78)
cannot allocate a tage (78)
cannot allocate a tage (78)
cannot allocate a tage (78)
cannot allocate a tage (101)
cannot allocate a tage (101)
cannot allocate a tage (101)
cannot allocate a tage (101)
Lustre: Mounted lustre-client
Lustre: DEBUG MARKER: Using TIMEOUT=20
Lustre: DEBUG MARKER: only running test 5
Lustre: DEBUG MARKER: excepting tests: 24 52 60
Lustre: DEBUG MARKER: skipping tests SLOW=no: 17 26a 26b 50 51 57
Lustre: DEBUG MARKER: Using TIMEOUT=20
Lustre: DEBUG MARKER: == recovery-small test 5: rename: drop req, drop rep == 10:05:50 (1350266750)
Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400):0:mdt
Lustre: *** cfs_fail_loc=123, val=2147483648***
Lustre: 28609:0:(client.c:1909:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1350266750/real 1350266750]  req@ffff88003bc73c00 x1415856378611452/t0(0) o101->lustre-MDT0000-mdc-ffff880028456000@0@lo:12/10 lens 600/1048 e 0 to 1 dl 1350266757 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Lustre: lustre-MDT0000-mdc-ffff880028456000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: lustre-MDT0000: Client c6a640e9-ad98-3383-eca0-a8bdd97206d6 (at 0@lo) reconnecting
Lustre: lustre-MDT0000-mdc-ffff880028456000: Connection restored to lustre-MDT0000 (at 0@lo)
Lustre: *** cfs_fail_loc=119, val=2147483648***
LustreError: 27416:0:(ldlm_lib.c:2404:target_send_reply_msg()) @@@ dropping reply  req@ffff88000ab95050 x1415856378611471/t4294967300(0) o36->c6a640e9-ad98-3383-eca0-a8bdd97206d6@0@lo:0/0 lens 560/456 e 0 to 0 dl 1350266763 ref 1 fl Interpret:/0/0 rc 0/0
Lustre: 28631:0:(client.c:1909:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1350266757/real 1350266757]  req@ffff880027fe0400 x1415856378611471/t0(0) o36->lustre-MDT0000-mdc-ffff880028456000@0@lo:12/10 lens 560/888 e 0 to 1 dl 1350266764 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Lustre: lustre-MDT0000-mdc-ffff880028456000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: lustre-MDT0000: Client c6a640e9-ad98-3383-eca0-a8bdd97206d6 (at 0@lo) reconnecting
Lustre: lustre-MDT0000-mdc-ffff880028456000: Connection restored to lustre-MDT0000 (at 0@lo)
Lustre: DEBUG MARKER: == recovery-small recovery-small.sh test complete, duration 15 sec == 10:06:04 (1350266764)
Lustre: DEBUG MARKER: only running test 1
Lustre: DEBUG MARKER: excepting tests: 24 52 60
Lustre: DEBUG MARKER: skipping tests SLOW=no: 17 26a 26b 50 51 57
Lustre: DEBUG MARKER: Using TIMEOUT=20
Lustre: DEBUG MARKER: == recovery-small test 1: create, chmod, stat: drop req, drop rep == 10:07:49 (1350266869)
Lustre: *** cfs_fail_loc=123, val=2147483648***
Lustre: 29534:0:(client.c:1909:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1350266869/real 1350266869]  req@ffff88003c5c3800 x1415856378611722/t0(0) o36->lustre-MDT0000-mdc-ffff880028456000@0@lo:12/10 lens 504/568 e 0 to 1 dl 1350266876 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Lustre: lustre-MDT0000-mdc-ffff880028456000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: lustre-MDT0000: Client c6a640e9-ad98-3383-eca0-a8bdd97206d6 (at 0@lo) reconnecting
Lustre: lustre-MDT0000-mdc-ffff880028456000: Connection restored to lustre-MDT0000 (at 0@lo)
Lustre: *** cfs_fail_loc=119, val=2147483648***
LustreError: 27415:0:(ldlm_lib.c:2404:target_send_reply_msg()) @@@ dropping reply  req@ffff88000b837050 x1415856378611734/t4294967303(0) o36->c6a640e9-ad98-3383-eca0-a8bdd97206d6@0@lo:0/0 lens 504/448 e 0 to 0 dl 1350266882 ref 1 fl Interpret:/0/0 rc 0/0
Lustre: 29554:0:(client.c:1909:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1350266876/real 1350266876]  req@ffff88003c455400 x1415856378611734/t0(0) o36->lustre-MDT0000-mdc-ffff880028456000@0@lo:12/10 lens 504/568 e 0 to 1 dl 1350266883 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Lustre: lustre-MDT0000-mdc-ffff880028456000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: lustre-MDT0000: Client c6a640e9-ad98-3383-eca0-a8bdd97206d6 (at 0@lo) reconnecting
Lustre: lustre-MDT0000-mdc-ffff880028456000: Connection restored to lustre-MDT0000 (at 0@lo)
Lustre: *** cfs_fail_loc=123, val=2147483648***
Lustre: 29581:0:(client.c:1909:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1350266883/real 1350266883]  req@ffff88002a7bb800 x1415856378611774/t0(0) o101->lustre-MDT0000-mdc-ffff880028456000@0@lo:12/10 lens 592/1080 e 0 to 1 dl 1350266890 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Lustre: lustre-MDT0000-mdc-ffff880028456000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: lustre-MDT0000: Client c6a640e9-ad98-3383-eca0-a8bdd97206d6 (at 0@lo) reconnecting
Lustre: lustre-MDT0000-mdc-ffff880028456000: Connection restored to lustre-MDT0000 (at 0@lo)
Lustre: *** cfs_fail_loc=119, val=2147483648***
LustreError: 27417:0:(ldlm_lib.c:2404:target_send_reply_msg()) @@@ dropping reply  req@ffff880009175850 x1415856378611788/t4294967305(0) o36->c6a640e9-ad98-3383-eca0-a8bdd97206d6@0@lo:0/0 lens 488/456 e 0 to 0 dl 1350266896 ref 1 fl Interpret:/0/0 rc 0/0
Lustre: 29601:0:(client.c:1909:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1350266890/real 1350266890]  req@ffff88002a7bb800 x1415856378611788/t0(0) o36->lustre-MDT0000-mdc-ffff880028456000@0@lo:12/10 lens 488/1016 e 0 to 1 dl 1350266897 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Lustre: lustre-MDT0000-mdc-ffff880028456000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: lustre-MDT0000: Client c6a640e9-ad98-3383-eca0-a8bdd97206d6 (at 0@lo) reconnecting
Lustre: lustre-MDT0000-mdc-ffff880028456000: Connection restored to lustre-MDT0000 (at 0@lo)
Lustre: *** cfs_fail_loc=123, val=2147483648***
Lustre: *** cfs_fail_loc=122, val=2147483648***
LustreError: 27417:0:(ldlm_lib.c:2404:target_send_reply_msg()) @@@ dropping reply  req@ffff880015169850 x1415856378611808/t0(0) o34->c6a640e9-ad98-3383-eca0-a8bdd97206d6@0@lo:0/0 lens 472/456 e 0 to 0 dl 1350266910 ref 1 fl Interpret:/0/0 rc 0/0
Lustre: 29641:0:(client.c:1909:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1350266904/real 1350266904]  req@ffff88003be02800 x1415856378611808/t0(0) o34->lustre-MDT0000-mdc-ffff880028456000@0@lo:12/10 lens 472/960 e 0 to 1 dl 1350266911 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Lustre: 29641:0:(client.c:1909:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Lustre: lustre-MDT0000-mdc-ffff880028456000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: Skipped 1 previous similar message
Lustre: lustre-MDT0000: Client c6a640e9-ad98-3383-eca0-a8bdd97206d6 (at 0@lo) reconnecting
Lustre: Skipped 1 previous similar message
Lustre: lustre-MDT0000-mdc-ffff880028456000: Connection restored to lustre-MDT0000 (at 0@lo)
Lustre: Skipped 1 previous similar message
Lustre: DEBUG MARKER: == recovery-small recovery-small.sh test complete, duration 44 sec == 10:08:32 (1350266912)
Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request
Lustre: Unmounted lustre-client
Lustre: server umount lustre-MDT0000 complete
Lustre: 26408:0:(client.c:1909:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1350267491/real 1350267491]  req@ffff88002a7bc000 x1415856378612862/t0(0) o400->lustre-MDT0000-osp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1350267498 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Lustre: lustre-MDT0000-osp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: 29902:0:(osp_dev.c:217:osp_disconnect()) lustre-MDT0000-osp-OST0000: can't disconnect: rc = -110
LustreError: 29902:0:(obd_mount.c:2110:server_put_super()) lustre-OST0000: Fail to disconnect osp-on-ost!
LustreError: 166-1: MGC192.168.56.4@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
Lustre: server umount lustre-OST0000 complete
Lustre: server umount lustre-OST0001 complete
LNet: Removed LNI 192.168.56.4@tcp
LNet: HW CPU cores: 1, npartitions: 1
alg: No test for crc32 (crc32-table)
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.3.53-gde2a0a6-CHANGED-2.6.32-279.2.1.el6.dfb4e73
LNet: Added LNI 192.168.56.4@tcp [8/256/0/180]
LNet: Accept secure, port 988
Lustre: Echo OBD driver; http://www.lustre.org/
LDISKFS-fs (loop0): mounted filesystem with ordered data mode. quota=on. Opts: 
LDISKFS-fs (loop0): mounted filesystem with ordered data mode. quota=on. Opts: 
LDISKFS-fs (loop0): mounted filesystem with ordered data mode. quota=on. Opts: 
LDISKFS-fs (loop0): mounted filesystem with ordered data mode. quota=on. Opts: 
Lustre: MGC192.168.56.4@tcp: Reactivating import
Lustre: Setting parameter lustre-MDT0000-mdtlov.lov.stripesize in log lustre-MDT0000
Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000
Lustre: Skipped 3 previous similar messages
LustreError: 1483:0:(mgc_request.c:248:do_config_log_add()) failed processing sptlrpc log: -2
Lustre: lustre-MDT0000: new disk, initializing
LustreError: 1525:0:(sec_config.c:1024:sptlrpc_target_local_copy_conf()) missing llog context
Lustre: 1525:0:(mdt_lproc.c:418:lprocfs_wr_identity_upcall()) lustre-MDT0000: identity upcall set to /root/lustre-master/lustre/tests/../utils/l_getidentity
Lustre: lustre-MDT0000: Temporarily refusing client connection from 0@lo
LustreError: 11-0: an error occurred while communicating with 0@lo. The mds_connect operation failed with -11
Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space
LDISKFS-fs (loop1): mounted filesystem with ordered data mode. quota=on. Opts: 
Lustre: lustre-OST0000: new disk, initializing
Lustre: 1662:0:(ofd_fs.c:271:ofd_groups_init()) lustre-OST0000: 1 groups initialized
LustreError: 11-0: an error occurred while communicating with 0@lo. The mds_connect operation failed with -11
LDISKFS-fs (loop2): mounted filesystem with ordered data mode. quota=on. Opts: 
Lustre: lustre-OST0001: new disk, initializing
Lustre: 1762:0:(ofd_fs.c:271:ofd_groups_init()) lustre-OST0001: 1 groups initialized
LustreError: 11-0: an error occurred while communicating with 0@lo. The mds_connect operation failed with -11
LustreError: Skipped 2 previous similar messages
Lustre: Mounted lustre-client
Lustre: DEBUG MARKER: Using TIMEOUT=20
Lustre: DEBUG MARKER: only running test 5
Lustre: DEBUG MARKER: excepting tests: 24 52 60
Lustre: DEBUG MARKER: skipping tests SLOW=no: 17 26a 26b 50 51 57
Lustre: DEBUG MARKER: Using TIMEOUT=20
Lustre: DEBUG MARKER: == recovery-small test 5: rename: drop req, drop rep == 10:21:12 (1350267672)
Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400):0:mdt
Lustre: *** cfs_fail_loc=123, val=2147483648***
Lustre: 2740:0:(client.c:1909:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1350267672/real 1350267672]  req@ffff880006445800 x1415858224103663/t0(0) o101->lustre-MDT0000-mdc-ffff880026ba2800@0@lo:12/10 lens 600/1048 e 0 to 1 dl 1350267679 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Lustre: lustre-MDT0000-mdc-ffff880026ba2800: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: lustre-MDT0000: Client 63df8595-230c-309b-17ff-539c4db45d8b (at 0@lo) reconnecting
Lustre: lustre-MDT0000-mdc-ffff880026ba2800: Connection restored to lustre-MDT0000 (at 0@lo)
Lustre: *** cfs_fail_loc=119, val=2147483648***
LustreError: 1529:0:(ldlm_lib.c:2404:target_send_reply_msg()) @@@ dropping reply  req@ffff880003d34850 x1415858224103713/t4294967300(0) o36->63df8595-230c-309b-17ff-539c4db45d8b@0@lo:0/0 lens 560/456 e 0 to 0 dl 1350267685 ref 1 fl Interpret:/0/0 rc 0/0
Lustre: 2768:0:(client.c:1909:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1350267679/real 1350267679]  req@ffff88003c118400 x1415858224103713/t0(0) o36->lustre-MDT0000-mdc-ffff880026ba2800@0@lo:12/10 lens 560/888 e 0 to 1 dl 1350267686 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Lustre: lustre-MDT0000-mdc-ffff880026ba2800: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: lustre-MDT0000: Client 63df8595-230c-309b-17ff-539c4db45d8b (at 0@lo) reconnecting
Lustre: lustre-MDT0000-mdc-ffff880026ba2800: Connection restored to lustre-MDT0000 (at 0@lo)
Lustre: DEBUG MARKER: == recovery-small recovery-small.sh test complete, duration 15 sec == 10:21:26 (1350267686)
Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
Lustre: Unmounted lustre-client
Lustre: server umount lustre-MDT0000 complete
Lustre: 474:0:(client.c:1909:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1350288356/real 1350288356]  req@ffff880018ec8400 x1415858224140931/t0(0) o400->lustre-MDT0000-osp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1350288363 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Lustre: lustre-MDT0000-osp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: 3257:0:(client.c:1909:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1350288362/real 1350288362]  req@ffff880018ec8000 x1415858224140939/t0(0) o39->lustre-MDT0000-osp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1350288368 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
LustreError: 3257:0:(osp_dev.c:217:osp_disconnect()) lustre-MDT0000-osp-OST0000: can't disconnect: rc = -110
LustreError: 3257:0:(obd_mount.c:2110:server_put_super()) lustre-OST0000: Fail to disconnect osp-on-ost!
Lustre: 474:0:(client.c:1909:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1350288361/real 1350288361]  req@ffff880018ec8c00 x1415858224140936/t0(0) o400->lustre-MDT0000-osp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1350288368 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Lustre: server umount lustre-OST0000 complete
Lustre: 474:0:(client.c:1909:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1350288368/real 1350288368]  req@ffff8800053e7c00 x1415858224140941/t0(0) o400->MGC192.168.56.4@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1350288375 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Lustre: 474:0:(client.c:1909:ptlrpc_expire_one_request()) Skipped 1 previous similar message
LustreError: 166-1: MGC192.168.56.4@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
Lustre: 472:0:(client.c:1909:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1350288375/real 1350288375]  req@ffff8800053e7c00 x1415858224140944/t0(0) o250->MGC192.168.56.4@tcp@0@lo:26/25 lens 400/544 e 0 to 1 dl 1350288381 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Lustre: server umount lustre-OST0001 complete
LNet: Removed LNI 192.168.56.4@tcp
LNet: HW CPU cores: 1, npartitions: 1
alg: No test for crc32 (crc32-table)
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.3.53-gde2a0a6-CHANGED-2.6.32-279.2.1.el6.dfb4e73
__ratelimit: 12 callbacks suppressed
cannot allocate a tage (2)
LNet: Added LNI 192.168.56.4@tcp [8/256/0/180]
LNet: Accept secure, port 988
cannot allocate a tage (5)
Lustre: Echo OBD driver; http://www.lustre.org/
LDISKFS-fs (loop0): mounted filesystem with ordered data mode. quota=on. Opts: 
LDISKFS-fs (loop0): mounted filesystem with ordered data mode. quota=on. Opts: 
LDISKFS-fs (loop0): mounted filesystem with ordered data mode. quota=on. Opts: 
LDISKFS-fs (loop0): mounted filesystem with ordered data mode. quota=on. Opts: 
Lustre: MGC192.168.56.4@tcp: Reactivating import
Lustre: Setting parameter lustre-MDT0000-mdtlov.lov.stripesize in log lustre-MDT0000
Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000
Lustre: Skipped 3 previous similar messages
LustreError: 7207:0:(mgc_request.c:248:do_config_log_add()) failed processing sptlrpc log: -2
Lustre: lustre-MDT0000: new disk, initializing
LustreError: 7249:0:(sec_config.c:1024:sptlrpc_target_local_copy_conf()) missing llog context
cannot allocate a tage (434)
cannot allocate a tage (434)
cannot allocate a tage (434)
cannot allocate a tage (434)
cannot allocate a tage (434)
cannot allocate a tage (434)
cannot allocate a tage (434)
cannot allocate a tage (434)
Lustre: 7249:0:(mdt_lproc.c:418:lprocfs_wr_identity_upcall()) lustre-MDT0000: identity upcall set to /root/lustre-master/lustre/tests/../utils/l_getidentity
Lustre: lustre-MDT0000: Temporarily refusing client connection from 0@lo
LustreError: 11-0: an error occurred while communicating with 0@lo. The mds_connect operation failed with -11
Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space
LDISKFS-fs (loop1): mounted filesystem with ordered data mode. quota=on. Opts: 
Lustre: lustre-OST0000: new disk, initializing
Lustre: 7386:0:(ofd_fs.c:271:ofd_groups_init()) lustre-OST0000: 1 groups initialized
LustreError: 11-0: an error occurred while communicating with 0@lo. The mds_connect operation failed with -11
LDISKFS-fs (loop2): mounted filesystem with ordered data mode. quota=on. Opts: 
Lustre: lustre-OST0001: new disk, initializing
Lustre: 7486:0:(ofd_fs.c:271:ofd_groups_init()) lustre-OST0001: 1 groups initialized
LustreError: 11-0: an error occurred while communicating with 0@lo. The mds_connect operation failed with -11
LustreError: Skipped 2 previous similar messages
Lustre: Mounted lustre-client
Lustre: DEBUG MARKER: Using TIMEOUT=20
Lustre: DEBUG MARKER: excepting tests: 24 52 60
Lustre: DEBUG MARKER: Using TIMEOUT=20
Lustre: DEBUG MARKER: == recovery-small test 1: create, chmod, stat: drop req, drop rep == 16:08:25 (1350288505)
Lustre: *** cfs_fail_loc=123, val=2147483648***
Lustre: 8379:0:(client.c:1909:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1350288505/real 1350288505]  req@ffff88001072ec00 x1415880068038879/t0(0) o700->lustre-MDT0000-mdc-ffff8800110d8800@0@lo:30/10 lens 264/248 e 0 to 1 dl 1350288517 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Lustre: lustre-MDT0000-mdc-ffff8800110d8800: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: lustre-MDT0000: Client 4782b42e-2e27-d7f7-10c2-d93bd62d8964 (at 0@lo) reconnecting
Lustre: lustre-MDT0000-mdc-ffff8800110d8800: Connection restored to lustre-MDT0000 (at 0@lo)
Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400):0:mdt
Lustre: *** cfs_fail_loc=119, val=2147483648***
LustreError: 7252:0:(ldlm_lib.c:2404:target_send_reply_msg()) @@@ dropping reply  req@ffff88001dbb0450 x1415880068038929/t4294967298(0) o36->4782b42e-2e27-d7f7-10c2-d93bd62d8964@0@lo:0/0 lens 504/448 e 0 to 0 dl 1350288523 ref 1 fl Interpret:/0/0 rc 0/0
Lustre: 8407:0:(client.c:1909:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1350288517/real 1350288517]  req@ffff880014615c00 x1415880068038929/t0(0) o36->lustre-MDT0000-mdc-ffff8800110d8800@0@lo:12/10 lens 504/568 e 0 to 1 dl 1350288524 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Lustre: lustre-MDT0000-mdc-ffff8800110d8800: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: lustre-MDT0000: Client 4782b42e-2e27-d7f7-10c2-d93bd62d8964 (at 0@lo) reconnecting
Lustre: lustre-MDT0000-mdc-ffff8800110d8800: Connection restored to lustre-MDT0000 (at 0@lo)
Lustre: *** cfs_fail_loc=123, val=2147483648***
Lustre: 8427:0:(client.c:1909:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1350288524/real 1350288524]  req@ffff880014615c00 x1415880068038941/t0(0) o101->lustre-MDT0000-mdc-ffff8800110d8800@0@lo:12/10 lens 576/1048 e 0 to 1 dl 1350288531 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Lustre: lustre-MDT0000-mdc-ffff8800110d8800: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: lustre-MDT0000: Client 4782b42e-2e27-d7f7-10c2-d93bd62d8964 (at 0@lo) reconnecting
Lustre: lustre-MDT0000-mdc-ffff8800110d8800: Connection restored to lustre-MDT0000 (at 0@lo)
Lustre: *** cfs_fail_loc=119, val=2147483648***
LustreError: 7252:0:(ldlm_lib.c:2404:target_send_reply_msg()) @@@ dropping reply  req@ffff88002f42e050 x1415880068038954/t4294967300(0) o36->4782b42e-2e27-d7f7-10c2-d93bd62d8964@0@lo:0/0 lens 488/456 e 0 to 0 dl 1350288537 ref 1 fl Interpret:/0/0 rc 0/0
Lustre: 8447:0:(client.c:1909:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1350288531/real 1350288531]  req@ffff880014615c00 x1415880068038954/t0(0) o36->lustre-MDT0000-mdc-ffff8800110d8800@0@lo:12/10 lens 488/1016 e 0 to 1 dl 1350288538 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Lustre: lustre-MDT0000-mdc-ffff8800110d8800: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: lustre-MDT0000: Client 4782b42e-2e27-d7f7-10c2-d93bd62d8964 (at 0@lo) reconnecting
Lustre: lustre-MDT0000-mdc-ffff8800110d8800: Connection restored to lustre-MDT0000 (at 0@lo)
Lustre: *** cfs_fail_loc=123, val=2147483648***
Lustre: 8467:0:(client.c:1909:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1350288538/real 1350288538]  req@ffff88000f6da000 x1415880068038966/t0(0) o34->lustre-MDT0000-mdc-ffff8800110d8800@0@lo:12/10 lens 472/960 e 0 to 1 dl 1350288545 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Lustre: lustre-MDT0000-mdc-ffff8800110d8800: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: lustre-MDT0000: Client 4782b42e-2e27-d7f7-10c2-d93bd62d8964 (at 0@lo) reconnecting
Lustre: lustre-MDT0000-mdc-ffff8800110d8800: Connection restored to lustre-MDT0000 (at 0@lo)
Lustre: *** cfs_fail_loc=122, val=2147483648***
LustreError: 7254:0:(ldlm_lib.c:2404:target_send_reply_msg()) @@@ dropping reply  req@ffff88003c9be050 x1415880068038976/t0(0) o34->4782b42e-2e27-d7f7-10c2-d93bd62d8964@0@lo:0/0 lens 472/456 e 0 to 0 dl 1350288551 ref 1 fl Interpret:/0/0 rc 0/0
Lustre: 8487:0:(client.c:1909:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1350288545/real 1350288545]  req@ffff88000f6da000 x1415880068038976/t0(0) o34->lustre-MDT0000-mdc-ffff8800110d8800@0@lo:12/10 lens 472/960 e 0 to 1 dl 1350288552 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Lustre: lustre-MDT0000-mdc-ffff8800110d8800: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: lustre-MDT0000: Client 4782b42e-2e27-d7f7-10c2-d93bd62d8964 (at 0@lo) reconnecting
Lustre: lustre-MDT0000-mdc-ffff8800110d8800: Connection restored to lustre-MDT0000 (at 0@lo)
Lustre: DEBUG MARKER: == recovery-small test 4: open: drop req, drop rep == 16:09:12 (1350288552)
Lustre: *** cfs_fail_loc=123, val=2147483648***
Lustre: *** cfs_fail_loc=122, val=2147483648***
LustreError: 7256:0:(ldlm_lib.c:2404:target_send_reply_msg()) @@@ dropping reply  req@ffff880000c21c50 x1415880068039005/t4294967306(0) o35->4782b42e-2e27-d7f7-10c2-d93bd62d8964@0@lo:0/0 lens 392/456 e 0 to 0 dl 1350288565 ref 1 fl Interpret:/0/0 rc 0/0
Lustre: 8601:0:(client.c:1909:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1350288559/real 1350288559]  req@ffff880014615c00 x1415880068039005/t0(0) o35->lustre-MDT0000-mdc-ffff8800110d8800@0@lo:23/10 lens 392/888 e 0 to 1 dl 1350288566 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Lustre: 8601:0:(client.c:1909:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Lustre: lustre-MDT0000-mdc-ffff8800110d8800: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: Skipped 1 previous similar message
Lustre: lustre-MDT0000: Client 4782b42e-2e27-d7f7-10c2-d93bd62d8964 (at 0@lo) reconnecting
Lustre: Skipped 1 previous similar message
Lustre: lustre-MDT0000-mdc-ffff8800110d8800: Connection restored to lustre-MDT0000 (at 0@lo)
Lustre: Skipped 1 previous similar message
Lustre: DEBUG MARKER: == recovery-small test 5: rename: drop req, drop rep == 16:09:27 (1350288567)
Lustre: *** cfs_fail_loc=123, val=2147483648***
Lustre: *** cfs_fail_loc=119, val=2147483648***
LustreError: 7253:0:(ldlm_lib.c:2404:target_send_reply_msg()) @@@ dropping reply  req@ffff88001606ac50 x1415880068039039/t4294967310(0) o36->4782b42e-2e27-d7f7-10c2-d93bd62d8964@0@lo:0/0 lens 560/456 e 0 to 0 dl 1350288580 ref 1 fl Interpret:/0/0 rc 0/0
Lustre: DEBUG MARKER: == recovery-small test 6: link, unlink: drop req, drop rep == 16:09:41 (1350288581)
Lustre: *** cfs_fail_loc=123, val=2147483648***
Lustre: 8817:0:(client.c:1909:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1350288581/real 1350288581]  req@ffff88000e0fc000 x1415880068039054/t0(0) o101->lustre-MDT0000-mdc-ffff8800110d8800@0@lo:12/10 lens 592/1080 e 0 to 1 dl 1350288588 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Lustre: 8817:0:(client.c:1909:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Lustre: lustre-MDT0000-mdc-ffff8800110d8800: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: Skipped 2 previous similar messages
Lustre: lustre-MDT0000: Client 4782b42e-2e27-d7f7-10c2-d93bd62d8964 (at 0@lo) reconnecting
Lustre: Skipped 2 previous similar messages
Lustre: lustre-MDT0000-mdc-ffff8800110d8800: Connection restored to lustre-MDT0000 (at 0@lo)
Lustre: Skipped 2 previous similar messages
Lustre: *** cfs_fail_loc=119, val=2147483648***
LustreError: 7252:0:(ldlm_lib.c:2404:target_send_reply_msg()) @@@ dropping reply  req@ffff88002f42e050 x1415880068039071/t4294967315(0) o36->4782b42e-2e27-d7f7-10c2-d93bd62d8964@0@lo:0/0 lens 512/440 e 0 to 0 dl 1350288594 ref 1 fl Interpret:/0/0 rc 0/0
Lustre: *** cfs_fail_loc=123, val=2147483648***
Lustre: *** cfs_fail_loc=119, val=2147483648***
LustreError: 7253:0:(ldlm_lib.c:2404:target_send_reply_msg()) @@@ dropping reply  req@ffff88003c9be850 x1415880068039093/t4294967317(0) o36->4782b42e-2e27-d7f7-10c2-d93bd62d8964@0@lo:0/0 lens 512/456 e 0 to 0 dl 1350288608 ref 1 fl Interpret:/0/0 rc 0/0
Lustre: DEBUG MARKER: == recovery-small test 8: touch: drop rep (bug 1423) == 16:10:09 (1350288609)
Lustre: DEBUG MARKER: == recovery-small test 9: pause bulk on OST (bug 1420) == 16:10:16 (1350288616)
LustreError: 7378:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id 214 sleeping for 5000ms
LustreError: 7378:0:(fail.c:137:__cfs_fail_timeout_set()) cfs_fail_timeout id 214 awake
Lustre: DEBUG MARKER: == recovery-small test 10: finish request on server after client eviction (bug 1521) == 16:10:22 (1350288622)
Lustre: *** cfs_fail_loc=305, val=0***
Lustre: *** cfs_fail_loc=305, val=0***
Lustre: *** cfs_fail_loc=305, val=0***
Lustre: *** cfs_fail_loc=305, val=0***
Lustre: Skipped 1 previous similar message
Lustre: 7253:0:(client.c:1909:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1350288622/real 1350288622]  req@ffff88000f6dac00 x1415880068039182/t0(0) o104->lustre-MDT0000@0@lo:15/16 lens 296/224 e 0 to 1 dl 1350288633 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Lustre: 7253:0:(client.c:1909:ptlrpc_expire_one_request()) Skipped 4 previous similar messages
LustreError: 138-a: lustre-MDT0000: A client on nid 0@lo was evicted due to a lock blocking callback time out: rc -107
LustreError: 7253:0:(mdt_handler.c:3208:mdt_recovery()) operation 101 on unconnected MDS from 12345-0@lo
LustreError: 11-0: an error occurred while communicating with 0@lo. The ldlm_enqueue operation failed with -107
LustreError: Skipped 3 previous similar messages
Lustre: lustre-MDT0000-mdc-ffff8800110d8800: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: Skipped 4 previous similar messages
LustreError: 167-0: lustre-MDT0000-mdc-ffff8800110d8800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
LustreError: 9229:0:(mdc_locks.c:773:mdc_enqueue()) ldlm_cli_enqueue: -5
LustreError: 9229:0:(client.c:1129:ptlrpc_import_delay_req()) @@@ IMP_INVALID  req@ffff88003dfacc00 x1415880068039208/t0(0) o101->lustre-MDT0000-mdc-ffff8800110d8800@0@lo:12/10 lens 592/1136 e 0 to 0 dl 0 ref 2 fl Rpc:P/0/ffffffff rc 0/-1
LustreError: 9229:0:(mdc_locks.c:773:mdc_enqueue()) ldlm_cli_enqueue: -108
Lustre: lustre-MDT0000-mdc-ffff8800110d8800: Connection restored to lustre-MDT0000 (at 0@lo)
Lustre: Skipped 4 previous similar messages
Lustre: DEBUG MARKER: == recovery-small test 11: wake up a thread waiting for completion after eviction (b=2460) == 16:10:33 (1350288633)
Lustre: DEBUG MARKER: cancel_lru_locks osc start
Lustre: DEBUG MARKER: cancel_lru_locks osc stop
Lustre: DEBUG MARKER: == recovery-small test 12: recover from timed out resend in ptlrpcd (b=2494) == 16:10:33 (1350288633)
Lustre: DEBUG MARKER: multiop /mnt/lustre/f.recovery-small.12 OS_c
Lustre: *** cfs_fail_loc=115, val=2147483648***
Lustre: lustre-MDT0000: Client 4782b42e-2e27-d7f7-10c2-d93bd62d8964 (at 0@lo) reconnecting
Lustre: Skipped 4 previous similar messages
Lustre: DEBUG MARKER: == recovery-small test 13: mdc_readpage restart test (bug 1138) == 16:11:13 (1350288673)
Lustre: *** cfs_fail_loc=104, val=2147483648***
Lustre: DEBUG MARKER: == recovery-small test 14: mdc_readpage resend test (bug 1138) == 16:11:21 (1350288681)
Lustre: *** cfs_fail_loc=106, val=0***
Lustre: Skipped 1 previous similar message
Lustre: DEBUG MARKER: == recovery-small test 15: failed open (-ENOMEM) == 16:11:21 (1350288681)
Lustre: *** cfs_fail_loc=128, val=0***
Lustre: DEBUG MARKER: == recovery-small test 16: timeout bulk put, don't evict client (2732) == 16:11:21 (1350288681)
Lustre: DEBUG MARKER: cancel_lru_locks osc start
Lustre: DEBUG MARKER: cancel_lru_locks osc stop
Lustre: *** cfs_fail_loc=504, val=0***
LustreError: 7378:0:(ldlm_lib.c:2706:target_bulk_io()) @@@ network error on bulk PUT 0(4096)  req@ffff88000f78a000 x1415880068039372/t0(0) o3->4782b42e-2e27-d7f7-10c2-d93bd62d8964@0@lo:0/0 lens 488/432 e 0 to 0 dl 1350288692 ref 1 fl Interpret:/0/0 rc 0/0
Lustre: lustre-OST0001: Bulk IO read error with 4782b42e-2e27-d7f7-10c2-d93bd62d8964 (at 0@lo), client will retry: rc -110
LustreError: 7372:0:(ofd_obd.c:521:ofd_set_info_async()) lustre-OST0001: Unsupported key revimp_update
Lustre: DEBUG MARKER: == recovery-small test 17: timeout bulk get, don't evict client (2732) == 16:11:53 (1350288713)
Lustre: *** cfs_fail_loc=503, val=0***
Lustre: 7377:0:(service.c:1266:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-1), not sending early reply
  req@ffff88003dfacc00 x1415880068039460/t0(0) o4->4782b42e-2e27-d7f7-10c2-d93bd62d8964@0@lo:0/0 lens 488/448 e 1 to 0 dl 1350288739 ref 2 fl Interpret:/0/0 rc 0/0
LustreError: 7378:0:(ldlm_lib.c:2686:target_bulk_io()) @@@ timeout on bulk GET after 26+0s  req@ffff88003dfacc00 x1415880068039460/t0(0) o4->4782b42e-2e27-d7f7-10c2-d93bd62d8964@0@lo:0/0 lens 488/448 e 1 to 0 dl 1350288739 ref 1 fl Interpret:/0/0 rc 0/0
Lustre: lustre-OST0000: Bulk IO write error with 4782b42e-2e27-d7f7-10c2-d93bd62d8964 (at 0@lo), client will retry: rc -110
Lustre: 6241:0:(client.c:1909:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1350288713/real 1350288713]  req@ffff88000f62e400 x1415880068039460/t0(0) o4->lustre-OST0000-osc-ffff8800110d8800@0@lo:6/4 lens 488/448 e 1 to 1 dl 1350288750 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Lustre: 6241:0:(client.c:1909:ptlrpc_expire_one_request()) Skipped 6 previous similar messages
Lustre: lustre-OST0000-osc-ffff8800110d8800: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: Skipped 3 previous similar messages
Lustre: lustre-OST0000: Client 4782b42e-2e27-d7f7-10c2-d93bd62d8964 (at 0@lo) reconnecting
Lustre: Skipped 2 previous similar messages
LustreError: 7372:0:(ofd_obd.c:521:ofd_set_info_async()) lustre-OST0000: Unsupported key revimp_update
Lustre: lustre-OST0000-osc-ffff8800110d8800: Connection restored to lustre-OST0000 (at 0@lo)
Lustre: Skipped 3 previous similar messages
Lustre: DEBUG MARKER: == recovery-small test 18a: manual ost invalidate clears page cache immediately == 16:13:10 (1350288790)
Lustre: DEBUG MARKER: cancel_lru_locks osc start
Lustre: DEBUG MARKER: cancel_lru_locks osc stop
Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request
LustreError: 9216:0:(ofd_obd.c:521:ofd_set_info_async()) lustre-OST0001: Unsupported key revimp_update
LustreError: 167-0: lustre-OST0001-osc-ffff8800110d8800: This client was evicted by lustre-OST0001; in progress operations using this service will fail.
Lustre: DEBUG MARKER: == recovery-small test 18b: eviction and reconnect clears page cache (2766) == 16:13:10 (1350288790)
Lustre: DEBUG MARKER: cancel_lru_locks osc start
Lustre: DEBUG MARKER: cancel_lru_locks osc stop
Lustre: 10292:0:(genops.c:1437:obd_export_evict_by_uuid()) lustre-OST0000: evicting 4782b42e-2e27-d7f7-10c2-d93bd62d8964 at adminstrative request
LustreError: 11-0: an error occurred while communicating with 0@lo. The obd_ping operation failed with -107
LustreError: 167-0: lustre-OST0000-osc-ffff8800110d8800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
Lustre: DEBUG MARKER: == recovery-small test 18c: Dropped connect reply after eviction handing (14755) == 16:13:32 (1350288812)
Lustre: DEBUG MARKER: cancel_lru_locks osc start
Lustre: DEBUG MARKER: cancel_lru_locks osc stop
Lustre: 10435:0:(genops.c:1437:obd_export_evict_by_uuid()) lustre-OST0000: evicting 4782b42e-2e27-d7f7-10c2-d93bd62d8964 at adminstrative request
LustreError: 11-0: an error occurred while communicating with 0@lo. The ost_statfs operation failed with -107
Lustre: *** cfs_fail_loc=225, val=0***
LustreError: 9216:0:(ofd_obd.c:521:ofd_set_info_async()) lustre-OST0000: Unsupported key revimp_update
Lustre: Evicted from lustre-OST0000_UUID (at 192.168.56.4@tcp) after server handle changed from 0x7edcb56c067938a1 to 0x7edcb56c06793a53
LustreError: 167-0: lustre-OST0000-osc-ffff8800110d8800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
Lustre: DEBUG MARKER: == recovery-small test 19a: test expired_lock_main on mds (2867) == 16:13:42 (1350288822)
Lustre: Mounted lustre-client
Lustre: *** cfs_fail_loc=304, val=0***
Lustre: *** cfs_fail_loc=304, val=0***
LustreError: 7383:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -11 from cancel RPC: canceling anyway
LustreError: 7383:0:(ldlm_request.c:1795:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -11
LustreError: 7231:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -11 from cancel RPC: canceling anyway
LustreError: 7231:0:(ldlm_request.c:1795:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -11
LustreError: 7372:0:(ofd_obd.c:521:ofd_set_info_async()) lustre-OST0000: Unsupported key revimp_update
LNet: Service thread pid 7253 was inactive for 40.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Pid: 7253, comm: mdt00_001

Call Trace:
 [<ffffffff814fea12>] schedule_timeout+0x192/0x2e0
 [<ffffffff8107e120>] ? process_timeout+0x0/0x10
 [<ffffffffa0392aa0>] ? ldlm_expired_completion_wait+0x0/0x260 [ptlrpc]
 [<ffffffffa0753751>] cfs_waitq_timedwait+0x11/0x20 [libcfs]
 [<ffffffffa0396b6d>] ldlm_completion_ast+0x48d/0x730 [ptlrpc]
 [<ffffffff81060250>] ? default_wake_function+0x0/0x20
 [<ffffffffa0396366>] ldlm_cli_enqueue_local+0x1e6/0x560 [ptlrpc]
 [<ffffffffa03966e0>] ? ldlm_completion_ast+0x0/0x730 [ptlrpc]
 [<ffffffffa0ba7fe0>] ? mdt_blocking_ast+0x0/0x2a0 [mdt]
 [<ffffffffa0bab460>] mdt_object_lock+0x320/0xb70 [mdt]
 [<ffffffffa0ba7fe0>] ? mdt_blocking_ast+0x0/0x2a0 [mdt]
 [<ffffffffa03966e0>] ? ldlm_completion_ast+0x0/0x730 [ptlrpc]
 [<ffffffffa0763621>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
 [<ffffffffa0bcdacb>] mdt_attr_set+0x8b/0x590 [mdt]
 [<ffffffffa0bce5ed>] mdt_reint_setattr+0x61d/0xea0 [mdt]
 [<ffffffffa0763621>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
 [<ffffffffa0763621>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
 [<ffffffffa0bc7801>] mdt_reint_rec+0x41/0xe0 [mdt]
 [<ffffffffa0bc0be3>] mdt_reint_internal+0x4f3/0x7f0 [mdt]
 [<ffffffffa0bc0f24>] mdt_reint+0x44/0xe0 [mdt]
 [<ffffffffa0bb0322>] mdt_handle_common+0x932/0x1740 [mdt]
 [<ffffffffa0bb1205>] mdt_regular_handle+0x15/0x20 [mdt]
 [<ffffffffa03cf50c>] ptlrpc_server_handle_request+0x41c/0xe00 [ptlrpc]
 [<ffffffffa075365e>] ? cfs_timer_arm+0xe/0x10 [libcfs]
 [<ffffffffa03c68e9>] ? ptlrpc_wait_event+0xa9/0x2a0 [ptlrpc]
 [<ffffffffa0763621>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
 [<ffffffff810533f3>] ? __wake_up+0x53/0x70
 [<ffffffffa03d0afc>] ptlrpc_main+0xc0c/0x19f0 [ptlrpc]
 [<ffffffffa03cfef0>] ? ptlrpc_main+0x0/0x19f0 [ptlrpc]
 [<ffffffff8100c14a>] child_rip+0xa/0x20
 [<ffffffffa03cfef0>] ? ptlrpc_main+0x0/0x19f0 [ptlrpc]
 [<ffffffffa03cfef0>] ? ptlrpc_main+0x0/0x19f0 [ptlrpc]
 [<ffffffff8100c140>] ? child_rip+0x0/0x20

LustreError: dumping log to /tmp/lustre-log.1350288862.7253
LNet: Service thread pid 9216 was inactive for 40.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Pid: 9216, comm: ll_ost00_004

Call Trace:
 [<ffffffff814fea12>] schedule_timeout+0x192/0x2e0
 [<ffffffff8107e120>] ? process_timeout+0x0/0x10
 [<ffffffffa0392aa0>] ? ldlm_expired_completion_wait+0x0/0x260 [ptlrpc]
 [<ffffffffa0753751>] cfs_waitq_timedwait+0x11/0x20 [libcfs]
 [<ffffffffa0396b6d>] ldlm_completion_ast+0x48d/0x730 [ptlrpc]
 [<ffffffff81060250>] ? default_wake_function+0x0/0x20
 [<ffffffffa0396366>] ldlm_cli_enqueue_local+0x1e6/0x560 [ptlrpc]
 [<ffffffffa03966e0>] ? ldlm_completion_ast+0x0/0x730 [ptlrpc]
 [<ffffffffa0395340>] ? ldlm_blocking_ast+0x0/0x180 [ptlrpc]
 [<ffffffffa0cc6b91>] ofd_destroy_by_fid+0x141/0x3d0 [ofd]
 [<ffffffffa0395340>] ? ldlm_blocking_ast+0x0/0x180 [ptlrpc]
 [<ffffffffa03966e0>] ? ldlm_completion_ast+0x0/0x730 [ptlrpc]
 [<ffffffffa0cc80b7>] ofd_destroy+0x187/0x670 [ofd]
 [<ffffffffa06de63d>] ost_handle+0x422d/0x4690 [ost]
 [<ffffffffa075f424>] ? libcfs_id2str+0x74/0xb0 [libcfs]
 [<ffffffffa03cf50c>] ptlrpc_server_handle_request+0x41c/0xe00 [ptlrpc]
 [<ffffffffa075365e>] ? cfs_timer_arm+0xe/0x10 [libcfs]
 [<ffffffffa03c68e9>] ? ptlrpc_wait_event+0xa9/0x2a0 [ptlrpc]
 [<ffffffffa0763621>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
 [<ffffffff810533f3>] ? __wake_up+0x53/0x70
 [<ffffffffa03d0afc>] ptlrpc_main+0xc0c/0x19f0 [ptlrpc]
 [<ffffffffa03cfef0>] ? ptlrpc_main+0x0/0x19f0 [ptlrpc]
 [<ffffffff8100c14a>] child_rip+0xa/0x20
 [<ffffffffa03cfef0>] ? ptlrpc_main+0x0/0x19f0 [ptlrpc]
 [<ffffffffa03cfef0>] ? ptlrpc_main+0x0/0x19f0 [ptlrpc]
 [<ffffffff8100c140>] ? child_rip+0x0/0x20

LustreError: dumping log to /tmp/lustre-log.1350288867.9216
LustreError: 0:0:(ldlm_lockd.c:374:waiting_locks_callback()) ### lock callback timer expired after 339s: evicting client at 0@lo  ns: mdt-ffff88002a762000 lock: ffff88000f701c80/0x7edcb56c06792f2b lrc: 3/0,0 mode: PR/PR res: 50040/2142381559 bits 0x1 rrc: 2 type: IBT flags: 0x4000020 nid: 0@lo remote: 0x7edcb56c06792f1d expref: 10 pid: 7253 timeout: 4319000968
LNet: Service thread pid 7253 completed after 150.07s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources).
Lustre: Unmounted lustre-client
LustreError: 7253:0:(mdt_handler.c:3208:mdt_recovery()) operation 101 on unconnected MDS from 12345-0@lo
LustreError: 11-0: an error occurred while communicating with 0@lo. The ldlm_enqueue operation failed with -107
Lustre: lustre-MDT0000-mdc-ffff8800110d8800: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: Skipped 5 previous similar messages
LustreError: 167-0: lustre-MDT0000-mdc-ffff8800110d8800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
LustreError: 10630:0:(mdc_locks.c:773:mdc_enqueue()) ldlm_cli_enqueue: -5
LustreError: 10630:0:(file.c:2333:ll_inode_revalidate_fini()) failure -5 inode 50040
Lustre: lustre-MDT0000-mdc-ffff8800110d8800: Connection restored to lustre-MDT0000 (at 0@lo)
Lustre: Skipped 5 previous similar messages
Lustre: DEBUG MARKER: == recovery-small test 19b: test expired_lock_main on ost (2867) == 16:16:17 (1350288977)
Lustre: Mounted lustre-client
Lustre: *** cfs_fail_loc=304, val=0***
LustreError: 0:0:(ldlm_lockd.c:374:waiting_locks_callback()) ### lock callback timer expired after 156s: evicting client at 0@lo  ns: filter-ffff8800074a4000 lock: ffff88003bcc2bc0/0x7edcb56c06793a92 lrc: 3/0,0 mode: PR/PR res: 10/0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x10020 nid: 0@lo remote: 0x7edcb56c06793a8b expref: 4 pid: 9216 timeout 4319006006
LNet: Service thread pid 9216 completed after 151.02s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources).
LustreError: 167-0: lustre-OST0000-osc-ffff8800110d8800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
Lustre: 7384:0:(client.c:1909:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1350288977/real 1350288977]  req@ffff88000f6d3000 x1415880068040156/t0(0) o103->lustre-OST0001-osc-ffff8800110d8800@0@lo:17/18 lens 328/224 e 0 to 1 dl 1350288984 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Lustre: 7384:0:(client.c:1909:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
LustreError: 7384:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -11 from cancel RPC: canceling anyway
LustreError: 7384:0:(ldlm_request.c:1795:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -11
Lustre: lustre-OST0001: Client 4782b42e-2e27-d7f7-10c2-d93bd62d8964 (at 0@lo) reconnecting
Lustre: Skipped 4 previous similar messages
LustreError: 9216:0:(ofd_obd.c:521:ofd_set_info_async()) lustre-OST0001: Unsupported key revimp_update
LustreError: 0:0:(ldlm_lockd.c:374:waiting_locks_callback()) ### lock callback timer expired after 151s: evicting client at 0@lo  ns: filter-ffff8800104b4000 lock: ffff88000f70c540/0x7edcb56c06793cbb lrc: 3/0,0 mode: PW/PW res: 9/0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) flags: 0x20 nid: 0@lo remote: 0x7edcb56c06793cb4 expref: 4 pid: 9217 timeout 4319156380
LustreError: 10810:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway
LustreError: 10810:0:(ldlm_request.c:1795:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108
Lustre: Unmounted lustre-client
LustreError: 11-0: an error occurred while communicating with 0@lo. The obd_ping operation failed with -107
LustreError: Skipped 1 previous similar message
LustreError: 167-0: lustre-OST0001-osc-ffff8800110d8800: This client was evicted by lustre-OST0001; in progress operations using this service will fail.
Lustre: DEBUG MARKER: == recovery-small test 20a: ldlm_handle_enqueue error (should return error) == 16:18:53 (1350289133)
Lustre: DEBUG MARKER: cancel_lru_locks osc start
Lustre: DEBUG MARKER: cancel_lru_locks osc stop
Lustre: *** cfs_fail_loc=308, val=0***
Lustre: DEBUG MARKER: == recovery-small test 20b: ldlm_handle_enqueue error (should return error) == 16:18:53 (1350289133)
Lustre: DEBUG MARKER: cancel_lru_locks osc start
Lustre: DEBUG MARKER: cancel_lru_locks osc stop
Lustre: DEBUG MARKER: == recovery-small test 21a: drop close request while close and open are both in flight == 16:18:54 (1350289134)
LustreError: 7253:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id 129 sleeping for 5000ms
Lustre: *** cfs_fail_loc=115, val=2147483648***
Lustre: DEBUG MARKER: cancel_lru_locks mdc start
Lustre: DEBUG MARKER: cancel_lru_locks mdc stop
LustreError: 7253:0:(fail.c:137:__cfs_fail_timeout_set()) cfs_fail_timeout id 129 awake
Lustre: DEBUG MARKER: == recovery-small test 21b: drop open request while close and open are both in flight == 16:19:02 (1350289142)
Lustre: DEBUG MARKER: cancel_lru_locks mdc start
Lustre: DEBUG MARKER: cancel_lru_locks mdc stop
Lustre: 11240:0:(client.c:1909:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1350289142/real 1350289142]  req@ffff88000a13a400 x1415880068040608/t0(0) o36->lustre-MDT0000-mdc-ffff8800110d8800@0@lo:12/10 lens 488/568 e 0 to 1 dl 1350289335 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Lustre: 11240:0:(client.c:1909:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Lustre: lustre-MDT0000-mdc-ffff8800110d8800: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: Skipped 4 previous similar messages
Lustre: lustre-MDT0000: Client 4782b42e-2e27-d7f7-10c2-d93bd62d8964 (at 0@lo) reconnecting
Lustre: Skipped 1 previous similar message
Lustre: lustre-MDT0000-mdc-ffff8800110d8800: Connection restored to lustre-MDT0000 (at 0@lo)
Lustre: Skipped 4 previous similar messages
Lustre: DEBUG MARKER: == recovery-small test 21c: drop both request while close and open are both in flight == 16:22:15 (1350289335)
Lustre: *** cfs_fail_loc=107, val=2147483648***
Lustre: Skipped 1 previous similar message
Lustre: DEBUG MARKER: cancel_lru_locks mdc start
Lustre: DEBUG MARKER: cancel_lru_locks mdc stop
Lustre: DEBUG MARKER: == recovery-small test 21d: drop close reply while close and open are both in flight == 16:22:25 (1350289345)
LustreError: 7253:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id 129 sleeping for 5000ms
Lustre: *** cfs_fail_loc=122, val=2147483648***
LustreError: 7256:0:(ldlm_lib.c:2404:target_send_reply_msg()) @@@ dropping reply  req@ffff8800005d0c50 x1415880068041048/t4294967478(0) o35->4782b42e-2e27-d7f7-10c2-d93bd62d8964@0@lo:0/0 lens 392/456 e 0 to 0 dl 1350289352 ref 1 fl Interpret:/0/0 rc 0/0
LustreError: 7256:0:(ldlm_lib.c:2404:target_send_reply_msg()) Skipped 1 previous similar message
Lustre: DEBUG MARKER: cancel_lru_locks mdc start
Lustre: DEBUG MARKER: cancel_lru_locks mdc stop
LustreError: 7253:0:(fail.c:137:__cfs_fail_timeout_set()) cfs_fail_timeout id 129 awake
Lustre: DEBUG MARKER: == recovery-small test 21e: drop open reply while close and open are both in flight == 16:22:33 (1350289353)
Lustre: *** cfs_fail_loc=119, val=2147483648***
LustreError: 7253:0:(ldlm_lib.c:2404:target_send_reply_msg()) @@@ dropping reply  req@ffff880014964850 x1415880068041098/t4294967495(0) o36->4782b42e-2e27-d7f7-10c2-d93bd62d8964@0@lo:0/0 lens 488/456 e 0 to 0 dl 1350289545 ref 1 fl Interpret:/0/0 rc 0/0
Lustre: DEBUG MARKER: cancel_lru_locks mdc start
Lustre: DEBUG MARKER: cancel_lru_locks mdc stop
Lustre: DEBUG MARKER: == recovery-small test 21f: drop both reply while close and open are both in flight == 16:25:46 (1350289546)
Lustre: *** cfs_fail_loc=119, val=2147483648***
LustreError: 7253:0:(ldlm_lib.c:2404:target_send_reply_msg()) @@@ dropping reply  req@ffff88001dbb0050 x1415880068041478/t4294967512(0) o36->4782b42e-2e27-d7f7-10c2-d93bd62d8964@0@lo:0/0 lens 488/456 e 0 to 0 dl 1350289557 ref 1 fl Interpret:/0/0 rc 0/0
Lustre: DEBUG MARKER: cancel_lru_locks mdc start
Lustre: DEBUG MARKER: cancel_lru_locks mdc stop
Lustre: DEBUG MARKER: == recovery-small test 21g: drop open reply and close request while close and open are both in flight == 16:25:54 (1350289554)
LustreError: 7253:0:(ldlm_lib.c:2404:target_send_reply_msg()) @@@ dropping reply  req@ffff88001fdb0c50 x1415880068041539/t4294967529(0) o36->4782b42e-2e27-d7f7-10c2-d93bd62d8964@0@lo:0/0 lens 488/456 e 0 to 0 dl 1350289565 ref 1 fl Interpret:/0/0 rc 0/0
LustreError: 7253:0:(ldlm_lib.c:2404:target_send_reply_msg()) Skipped 1 previous similar message
Lustre: DEBUG MARKER: cancel_lru_locks mdc start
Lustre: DEBUG MARKER: cancel_lru_locks mdc stop
Lustre: DEBUG MARKER: == recovery-small test 21h: drop open request and close reply while close and open are both in flight == 16:26:03 (1350289563)
Lustre: DEBUG MARKER: cancel_lru_locks mdc start
Lustre: DEBUG MARKER: cancel_lru_locks mdc stop
Lustre: *** cfs_fail_loc=122, val=2147483648***
Lustre: Skipped 2 previous similar messages
Lustre: DEBUG MARKER: == recovery-small test 22: drop close request and do mknod == 16:26:11 (1350289571)
Lustre: DEBUG MARKER: cancel_lru_locks mdc start
Lustre: DEBUG MARKER: cancel_lru_locks mdc stop
Lustre: DEBUG MARKER: == recovery-small test 23: client hang when close a file after mds crash == 16:26:18 (1350289578)
Lustre: Failing over lustre-MDT0000
Lustre: server umount lustre-MDT0000 complete
LDISKFS-fs (loop0): mounted filesystem with ordered data mode. quota=on. Opts: 
LustreError: 166-1: MGC192.168.56.4@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
Lustre: MGC192.168.56.4@tcp: Reactivating import
Lustre: Found index 0 for lustre-MDT0000, updating log
Lustre: Modifying parameter lustre-MDT0000-mdtlov.lov.stripesize in log lustre-MDT0000
Lustre: lustre-MDT0000: used disk, loading
LustreError: 12284:0:(sec_config.c:1024:sptlrpc_target_local_copy_conf()) missing llog context
LustreError: 12288:0:(mdt_handler.c:3208:mdt_recovery()) operation 400 on unconnected MDS from 12345-0@lo
LustreError: 11-0: an error occurred while communicating with 0@lo. The obd_ping operation failed with -107
LustreError: Skipped 2 previous similar messages
Lustre: 12284:0:(mdt_lproc.c:418:lprocfs_wr_identity_upcall()) lustre-MDT0000: identity upcall set to /root/lustre-master/lustre/tests/../utils/l_getidentity
Lustre: lustre-MDT0000: Temporarily refusing client connection from 0@lo
Lustre: lustre-MDT0000: Temporarily refusing client connection from 0@lo
Lustre: 12250:0:(ldlm_lib.c:2163:target_recovery_init()) RECOVERY: service lustre-MDT0000, 1 recoverable clients, last_transno 4294967564
Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
LustreError: 167-0: lustre-MDT0000-osp-OST0000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
Lustre: 9217:0:(ofd_obd.c:1067:ofd_orphans_destroy()) lustre-OST0000: deleting orphan objects from 22 to 66
Lustre: DEBUG MARKER: SKIP: recovery-small test_24 skipping ALWAYS excluded test 24
Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports == 16:26:39 (1350289599)
Lustre: DEBUG MARKER: SKIP: recovery-small test_26a local OST
Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports == 16:26:39 (1350289599)
Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node
Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's == 16:26:39 (1350289599)
Lustre: Failing over lustre-MDT0000
Lustre: server umount lustre-MDT0000 complete
LustreError: 166-1: MGC192.168.56.4@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
LustreError: 7239:0:(client.c:1129:ptlrpc_import_delay_req()) @@@ IMP_INVALID  req@ffff88003cea4400 x1415880068042635/t0(0) o101->MGC192.168.56.4@tcp@0@lo:26/25 lens 328/384 e 0 to 0 dl 0 ref 2 fl Rpc:/0/ffffffff rc 0/-1
LustreError: 7239:0:(client.c:1129:ptlrpc_import_delay_req()) @@@ IMP_INVALID  req@ffff88003cea4400 x1415880068042636/t0(0) o101->MGC192.168.56.4@tcp@0@lo:26/25 lens 328/384 e 0 to 0 dl 0 ref 2 fl Rpc:/0/ffffffff rc 0/-1
LDISKFS-fs (loop0): mounted filesystem with ordered data mode. quota=on. Opts: 
Lustre: MGC192.168.56.4@tcp: Reactivating import
Lustre: Skipped 1 previous similar message
Lustre: Found index 0 for lustre-MDT0000, updating log
Lustre: Modifying parameter lustre-MDT0000-mdtlov.lov.stripesize in log lustre-MDT0000
Lustre: Skipped 4 previous similar messages
Lustre: lustre-MDT0000: used disk, loading
LustreError: 12873:0:(sec_config.c:1024:sptlrpc_target_local_copy_conf()) missing llog context
Lustre: lustre-MDT0000: Temporarily refusing client connection from 0@lo
Lustre: Skipped 1 previous similar message
LustreError: 7372:0:(ldlm_resource.c:1103:ldlm_resource_get()) lvbo_init failed for resource 68: rc -2
LustreError: 12687:0:(ldlm_resource.c:1103:ldlm_resource_get()) lvbo_init failed for resource 69: rc -2
Lustre: 12873:0:(mdt_lproc.c:418:lprocfs_wr_identity_upcall()) lustre-MDT0000: identity upcall set to /root/lustre-master/lustre/tests/../utils/l_getidentity
Lustre: 12839:0:(ldlm_lib.c:2163:target_recovery_init()) RECOVERY: service lustre-MDT0000, 1 recoverable clients, last_transno 8589935008
Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
LustreError: 7239:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id 407 sleeping for 10000ms
Lustre: 7372:0:(ofd_obd.c:1067:ofd_orphans_destroy()) lustre-OST0000: deleting orphan objects from 139 to 163
Lustre: 7372:0:(ofd_obd.c:1067:ofd_orphans_destroy()) Skipped 1 previous similar message
LustreError: 7239:0:(fail.c:137:__cfs_fail_timeout_set()) cfs_fail_timeout id 407 awake
Lustre: Failing over lustre-MDT0000
Lustre: server umount lustre-MDT0000 complete
LDISKFS-fs (loop0): mounted filesystem with ordered data mode. quota=on. Opts: 
LustreError: 166-1: MGC192.168.56.4@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
Lustre: MGC192.168.56.4@tcp: Reactivating import
Lustre: Skipped 1 previous similar message
Lustre: Found index 0 for lustre-MDT0000, updating log
Lustre: Modifying parameter lustre-MDT0000-mdtlov.lov.stripesize in log lustre-MDT0000
Lustre: Skipped 4 previous similar messages
Lustre: lustre-MDT0000: used disk, loading
LustreError: 13667:0:(sec_config.c:1024:sptlrpc_target_local_copy_conf()) missing llog context
LustreError: 12716:0:(ldlm_resource.c:1103:ldlm_resource_get()) lvbo_init failed for resource 4448: rc -2
LustreError: 12716:0:(ldlm_resource.c:1103:ldlm_resource_get()) Skipped 135 previous similar messages
Lustre: 13667:0:(mdt_lproc.c:418:lprocfs_wr_identity_upcall()) lustre-MDT0000: identity upcall set to /root/lustre-master/lustre/tests/../utils/l_getidentity
Lustre: lustre-MDT0000: Temporarily refusing client connection from 0@lo
Lustre: Skipped 1 previous similar message
LustreError: 11-0: an error occurred while communicating with 0@lo. The mds_connect operation failed with -11
LustreError: Skipped 4 previous similar messages
Lustre: 13634:0:(ldlm_lib.c:2163:target_recovery_init()) RECOVERY: service lustre-MDT0000, 1 recoverable clients, last_transno 12884934143
Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
Lustre: 13317:0:(ofd_obd.c:1067:ofd_orphans_destroy()) lustre-OST0000: deleting orphan objects from 5541 to 5572
Lustre: 13317:0:(ofd_obd.c:1067:ofd_orphans_destroy()) Skipped 1 previous similar message
LustreError: 13672:0:(mdt_handler.c:3208:mdt_recovery()) operation 400 on unconnected MDS from 12345-0@lo
LustreError: 13672:0:(mdt_handler.c:3208:mdt_recovery()) Skipped 1 previous similar message
LustreError: 167-0: lustre-MDT0000-osp-OST0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
LustreError: Skipped 3 previous similar messages
Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) == 16:28:46 (1350289726)
Lustre: *** cfs_fail_loc=305, val=0***
Lustre: *** cfs_fail_loc=305, val=0***
Lustre: Skipped 1 previous similar message
LustreError: 138-a: lustre-MDT0000: A client on nid 0@lo was evicted due to a lock blocking callback time out: rc -107
LustreError: 13672:0:(mdt_handler.c:3208:mdt_recovery()) operation 41 on unconnected MDS from 12345-0@lo
LustreError: 13672:0:(mdt_handler.c:3208:mdt_recovery()) Skipped 1 previous similar message
Lustre: *** cfs_fail_loc=12f, val=0***
LustreError: 13736:0:(target.c:670:lut_client_new()) lustre-MDT0000: no room for 0 clients - fix LR_MAX_CLIENTS
LustreError: 13838:0:(lmv_obd.c:1191:lmv_statfs()) can't stat MDS #0 (lustre-MDT0000-mdc-ffff8800110d8800), error -5
LustreError: 13838:0:(llite_lib.c:1558:ll_statfs_internal()) md_statfs fails: rc = -5
Lustre: Failing over lustre-MDT0000
Lustre: server umount lustre-MDT0000 complete
LDISKFS-fs (loop0): mounted filesystem with ordered data mode. quota=on. Opts: 
LustreError: 166-1: MGC192.168.56.4@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
Lustre: MGC192.168.56.4@tcp: Reactivating import
Lustre: Skipped 1 previous similar message
Lustre: Found index 0 for lustre-MDT0000, updating log
Lustre: Modifying parameter lustre-MDT0000-mdtlov.lov.stripesize in log lustre-MDT0000
Lustre: Skipped 4 previous similar messages
Lustre: lustre-MDT0000: used disk, loading
LustreError: 13996:0:(sec_config.c:1024:sptlrpc_target_local_copy_conf()) missing llog context
Lustre: 13996:0:(mdt_lproc.c:418:lprocfs_wr_identity_upcall()) lustre-MDT0000: identity upcall set to /root/lustre-master/lustre/tests/../utils/l_getidentity
Lustre: lustre-MDT0000: Temporarily refusing client connection from 0@lo
Lustre: 13962:0:(ldlm_lib.c:2163:target_recovery_init()) RECOVERY: service lustre-MDT0000, 1 recoverable clients, last_transno 17179869189
LustreError: 14001:0:(mdt_handler.c:3208:mdt_recovery()) operation 41 on unconnected MDS from 12345-0@lo
Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
Lustre: 7372:0:(ofd_obd.c:1067:ofd_orphans_destroy()) lustre-OST0000: deleting orphan objects from 5575 to 5605
Lustre: 7372:0:(ofd_obd.c:1067:ofd_orphans_destroy()) Skipped 1 previous similar message
LustreError: 13317:0:(ldlm_resource.c:1103:ldlm_resource_get()) lvbo_init failed for resource 5540: rc -2
LustreError: 13317:0:(ldlm_resource.c:1103:ldlm_resource_get()) Skipped 2015 previous similar messages
Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) == 16:29:29 (1350289769)
Lustre: Failing over lustre-MDT0000
Lustre: server umount lustre-MDT0000 complete
LDISKFS-fs (loop0): mounted filesystem with ordered data mode. quota=on. Opts: 
Lustre: *** cfs_fail_loc=711, val=0***
LustreError: 14236:0:(mdt_handler.c:6134:mdt_iocontrol()) Aborting recovery for device lustre-MDT0000
LustreError: 14236:0:(ldlm_lib.c:2122:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery
Lustre: 14301:0:(ldlm_lib.c:1789:target_recovery_overseer()) recovery is aborted, evict exports in recovery
Lustre: lustre-MDT0000: disconnecting 1 stale clients
LustreError: 14335:0:(lmv_obd.c:1191:lmv_statfs()) can't stat MDS #0 (lustre-MDT0000-mdc-ffff8800110d8800), error -5
LustreError: 14335:0:(llite_lib.c:1558:ll_statfs_internal()) md_statfs fails: rc = -5
Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) == 16:29:33 (1350289773)
LustreError: 14274:0:(mdt_handler.c:3208:mdt_recovery()) operation 39 on unconnected MDS from 12345-0@lo
LustreError: 14274:0:(mdt_handler.c:3208:mdt_recovery()) Skipped 1 previous similar message
LustreError: 14431:0:(osp_dev.c:217:osp_disconnect()) lustre-MDT0000-osp-OST0000: can't disconnect: rc = -107
LustreError: 14431:0:(obd_mount.c:2110:server_put_super()) lustre-OST0000: Fail to disconnect osp-on-ost!
LDISKFS-fs (loop1): mounted filesystem with ordered data mode. quota=on. Opts: 
Lustre: Found index 0 for lustre-OST0000, updating log
Lustre: Skipped 1 previous similar message
Lustre: 14534:0:(ofd_fs.c:271:ofd_groups_init()) lustre-OST0000: 1 groups initialized
Lustre: 14514:0:(ldlm_lib.c:2163:target_recovery_init()) RECOVERY: service lustre-OST0000, 1 recoverable clients, last_transno 4294972782
Lustre: 14514:0:(ldlm_lib.c:2163:target_recovery_init()) Skipped 1 previous similar message
LustreError: 14514:0:(ofd_obd.c:1358:ofd_iocontrol()) lustre-OST0000: aborting recovery
LustreError: 14514:0:(ldlm_lib.c:2122:target_stop_recovery_thread()) lustre-OST0000: Aborting recovery
Lustre: 14539:0:(ldlm_lib.c:1789:target_recovery_overseer()) recovery is aborted, evict exports in recovery
Lustre: 14539:0:(ldlm_lib.c:1789:target_recovery_overseer()) Skipped 2 previous similar messages
Lustre: lustre-OST0000: disconnecting 1 stale clients
Lustre: *** cfs_fail_loc=711, val=0***
Lustre: 13317:0:(ofd_obd.c:1067:ofd_orphans_destroy()) lustre-OST0000: deleting orphan objects from 5575 to 5638
Lustre: 13317:0:(ofd_obd.c:1067:ofd_orphans_destroy()) Skipped 2 previous similar messages
LustreError: 13317:0:(ldlm_resource.c:1103:ldlm_resource_get()) lvbo_init failed for resource 5605: rc -2
LustreError: 13317:0:(ldlm_resource.c:1103:ldlm_resource_get()) Skipped 94 previous similar messages
Lustre: DEBUG MARKER: == recovery-small test 50: failover MDS under load == 16:29:37 (1350289777)
Lustre: Failing over lustre-MDT0000
Lustre: Skipped 1 previous similar message
Lustre: server umount lustre-MDT0000 complete
Lustre: Skipped 1 previous similar message
LDISKFS-fs (loop0): mounted filesystem with ordered data mode. quota=on. Opts: 
LustreError: 166-1: MGC192.168.56.4@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
LustreError: Skipped 1 previous similar message
Lustre: MGC192.168.56.4@tcp: Reactivating import
Lustre: Skipped 3 previous similar messages
Lustre: Found index 0 for lustre-MDT0000, updating log
Lustre: Modifying parameter lustre-MDT0000-mdtlov.lov.stripesize in log lustre-MDT0000
Lustre: Skipped 9 previous similar messages
Lustre: lustre-MDT0000: used disk, loading
Lustre: Skipped 1 previous similar message
LustreError: 14811:0:(sec_config.c:1024:sptlrpc_target_local_copy_conf()) missing llog context
LustreError: 14811:0:(sec_config.c:1024:sptlrpc_target_local_copy_conf()) Skipped 1 previous similar message
LustreError: 14815:0:(mdt_handler.c:3208:mdt_recovery()) operation 400 on unconnected MDS from 12345-0@lo
LustreError: 14815:0:(mdt_handler.c:3208:mdt_recovery()) Skipped 1 previous similar message
LustreError: 12747:0:(ldlm_resource.c:1103:ldlm_resource_get()) lvbo_init failed for resource 5930: rc -2
LustreError: 12747:0:(ldlm_resource.c:1103:ldlm_resource_get()) Skipped 30 previous similar messages
Lustre: 14811:0:(mdt_lproc.c:418:lprocfs_wr_identity_upcall()) lustre-MDT0000: identity upcall set to /root/lustre-master/lustre/tests/../utils/l_getidentity
Lustre: 14811:0:(mdt_lproc.c:418:lprocfs_wr_identity_upcall()) Skipped 1 previous similar message
Lustre: lustre-MDT0000: Temporarily refusing client connection from 0@lo
Lustre: Skipped 1 previous similar message
Lustre: 14776:0:(ldlm_lib.c:2163:target_recovery_init()) RECOVERY: service lustre-MDT0000, 1 recoverable clients, last_transno 25769807259
LustreError: 167-0: lustre-MDT0000-osp-OST0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
LustreError: Skipped 5 previous similar messages
Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
Lustre: 7373:0:(ofd_obd.c:1067:ofd_orphans_destroy()) lustre-OST0000: deleting orphan objects from 6222 to 6247
Lustre: Failing over lustre-MDT0000
Lustre: server umount lustre-MDT0000 complete
LDISKFS-fs (loop0): mounted filesystem with ordered data mode. quota=on. Opts: 
LustreError: 166-1: MGC192.168.56.4@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
Lustre: MGC192.168.56.4@tcp: Reactivating import
Lustre: Skipped 1 previous similar message
Lustre: Found index 0 for lustre-MDT0000, updating log
Lustre: Modifying parameter lustre-MDT0000-mdtlov.lov.stripesize in log lustre-MDT0000
Lustre: Skipped 4 previous similar messages
Lustre: lustre-MDT0000: used disk, loading
LustreError: 15042:0:(sec_config.c:1024:sptlrpc_target_local_copy_conf()) missing llog context
LustreError: 13317:0:(ldlm_resource.c:1103:ldlm_resource_get()) lvbo_init failed for resource 7074: rc -2
LustreError: 13317:0:(ldlm_resource.c:1103:ldlm_resource_get()) Skipped 355 previous similar messages
Lustre: 15042:0:(mdt_lproc.c:418:lprocfs_wr_identity_upcall()) lustre-MDT0000: identity upcall set to /root/lustre-master/lustre/tests/../utils/l_getidentity
Lustre: lustre-MDT0000: Temporarily refusing client connection from 0@lo
LustreError: 11-0: an error occurred while communicating with 0@lo. The mds_connect operation failed with -11
LustreError: Skipped 14 previous similar messages
Lustre: 15009:0:(ldlm_lib.c:2163:target_recovery_init()) RECOVERY: service lustre-MDT0000, 1 recoverable clients, last_transno 30064790465
LustreError: 15047:0:(mdt_handler.c:3208:mdt_recovery()) operation 41 on unconnected MDS from 12345-0@lo
Lustre: lustre-MDT0000-mdc-ffff8800110d8800: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: Skipped 24 previous similar messages
Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
Lustre: lustre-MDT0000-mdc-ffff8800110d8800: Connection restored to lustre-MDT0000 (at 0@lo)
Lustre: Skipped 24 previous similar messages
Lustre: 12687:0:(ofd_obd.c:1067:ofd_orphans_destroy()) lustre-OST0000: deleting orphan objects from 9475 to 9512
Lustre: 12687:0:(ofd_obd.c:1067:ofd_orphans_destroy()) Skipped 1 previous similar message
Lustre: 6242:0:(client.c:1909:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1350289884/real 1350289884]  req@ffff88000718e800 x1415880068159482/t0(0) o400->lustre-MDT0000-mdc-ffff8800110d8800@0@lo:12/10 lens 224/224 e 0 to 1 dl 1350289906 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Lustre: 6242:0:(client.c:1909:ptlrpc_expire_one_request()) Skipped 22 previous similar messages
Lustre: Failing over lustre-MDT0000
LustreError: 15048:0:(mdt_handler.c:3208:mdt_recovery()) operation 35 on unconnected MDS from 12345-0@lo
LustreError: 15048:0:(mdt_handler.c:3208:mdt_recovery()) Skipped 2 previous similar messages
Lustre: server umount lustre-MDT0000 complete
LDISKFS-fs (loop0): mounted filesystem with ordered data mode. quota=on. Opts: 
LustreError: 166-1: MGC192.168.56.4@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
Lustre: MGC192.168.56.4@tcp: Reactivating import
Lustre: Skipped 1 previous similar message
Lustre: Found index 0 for lustre-MDT0000, updating log
Lustre: Modifying parameter lustre-MDT0000-mdtlov.lov.stripesize in log lustre-MDT0000
Lustre: Skipped 4 previous similar messages
Lustre: lustre-MDT0000: used disk, loading
LustreError: 15270:0:(sec_config.c:1024:sptlrpc_target_local_copy_conf()) missing llog context
Lustre: 15270:0:(mdt_lproc.c:418:lprocfs_wr_identity_upcall()) lustre-MDT0000: identity upcall set to /root/lustre-master/lustre/tests/../utils/l_getidentity
LustreError: 7549:0:(ldlm_resource.c:1103:ldlm_resource_get()) lvbo_init failed for resource 12537: rc -2
LustreError: 7549:0:(ldlm_resource.c:1103:ldlm_resource_get()) Skipped 1582 previous similar messages
Lustre: lustre-MDT0000: Temporarily refusing client connection from 0@lo
Lustre: 15237:0:(ldlm_lib.c:2163:target_recovery_init()) RECOVERY: service lustre-MDT0000, 1 recoverable clients, last_transno 34359758243
LustreError: 167-0: lustre-MDT0000-osp-OST0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
LustreError: Skipped 3 previous similar messages
Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
Lustre: 9216:0:(ofd_obd.c:1067:ofd_orphans_destroy()) lustre-OST0000: deleting orphan objects from 12827 to 12873
Lustre: 9216:0:(ofd_obd.c:1067:ofd_orphans_destroy()) Skipped 1 previous similar message
Lustre: DEBUG MARKER: == recovery-small test 51: failover MDS during recovery == 16:33:13 (1350289993)
Lustre: Failing over lustre-MDT0000
Lustre: server umount lustre-MDT0000 complete
LDISKFS-fs (loop0): mounted filesystem with ordered data mode. quota=on. Opts: 
LustreError: 166-1: MGC192.168.56.4@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
Lustre: MGC192.168.56.4@tcp: Reactivating import
Lustre: Skipped 1 previous similar message
Lustre: Found index 0 for lustre-MDT0000, updating log
Lustre: Modifying parameter lustre-MDT0000-mdtlov.lov.stripesize in log lustre-MDT0000
Lustre: Skipped 4 previous similar messages
Lustre: lustre-MDT0000: used disk, loading
LustreError: 15579:0:(sec_config.c:1024:sptlrpc_target_local_copy_conf()) missing llog context
LustreError: 13027:0:(ldlm_resource.c:1103:ldlm_resource_get()) lvbo_init failed for resource 13699: rc -2
LustreError: 13027:0:(ldlm_resource.c:1103:ldlm_resource_get()) Skipped 1328 previous similar messages
Lustre: 15579:0:(mdt_lproc.c:418:lprocfs_wr_identity_upcall()) lustre-MDT0000: identity upcall set to /root/lustre-master/lustre/tests/../utils/l_getidentity
Lustre: lustre-MDT0000: Temporarily refusing client connection from 0@lo
Lustre: 15545:0:(ldlm_lib.c:2163:target_recovery_init()) RECOVERY: service lustre-MDT0000, 1 recoverable clients, last_transno 38654712513
LustreError: 15670:0:(ldlm_lib.c:2122:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery
Lustre: 15610:0:(ldlm_lib.c:1789:target_recovery_overseer()) recovery is aborted, evict exports in recovery
Lustre: 15610:0:(ldlm_lib.c:1789:target_recovery_overseer()) Skipped 2 previous similar messages
Lustre: 7373:0:(ofd_obd.c:1067:ofd_orphans_destroy()) lustre-OST0000: deleting orphan objects from 14020 to 14058
Lustre: 7373:0:(ofd_obd.c:1067:ofd_orphans_destroy()) Skipped 1 previous similar message
LDISKFS-fs (loop0): mounted filesystem with ordered data mode. quota=on. Opts: 
LustreError: 7239:0:(client.c:1129:ptlrpc_import_delay_req()) @@@ IMP_INVALID  req@ffff88000fd83400 x1415880068218745/t0(0) o101->MGC192.168.56.4@tcp@0@lo:26/25 lens 328/384 e 0 to 0 dl 0 ref 2 fl Rpc:/0/ffffffff rc 0/-1
LustreError: 7239:0:(client.c:1129:ptlrpc_import_delay_req()) Skipped 4 previous similar messages
LustreError: 15804:0:(mdt_handler.c:3208:mdt_recovery()) operation 400 on unconnected MDS from 12345-0@lo
LustreError: 15804:0:(mdt_handler.c:3208:mdt_recovery()) Skipped 2 previous similar messages
Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
LDISKFS-fs (loop0): mounted filesystem with ordered data mode. quota=on. Opts: 
LustreError: 166-1: MGC192.168.56.4@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
LustreError: Skipped 1 previous similar message
Lustre: MGC192.168.56.4@tcp: Reactivating import
Lustre: Skipped 3 previous similar messages
Lustre: Modifying parameter lustre-MDT0000-mdtlov.lov.stripesize in log lustre-MDT0000
Lustre: Skipped 9 previous similar messages
Lustre: lustre-MDT0000: used disk, loading
Lustre: Skipped 1 previous similar message
LustreError: 16023:0:(sec_config.c:1024:sptlrpc_target_local_copy_conf()) missing llog context
LustreError: 16023:0:(sec_config.c:1024:sptlrpc_target_local_copy_conf()) Skipped 1 previous similar message
Lustre: 16023:0:(mdt_lproc.c:418:lprocfs_wr_identity_upcall()) lustre-MDT0000: identity upcall set to /root/lustre-master/lustre/tests/../utils/l_getidentity
Lustre: 16023:0:(mdt_lproc.c:418:lprocfs_wr_identity_upcall()) Skipped 1 previous similar message
LDISKFS-fs (loop0): mounted filesystem with ordered data mode. quota=on. Opts: 
Lustre: Failing over lustre-MDT0000
Lustre: Skipped 3 previous similar messages
LustreError: 16345:0:(ldlm_lib.c:2122:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery
Lustre: 16279:0:(ldlm_lib.c:1789:target_recovery_overseer()) recovery is aborted, evict exports in recovery
Lustre: 16279:0:(ldlm_lib.c:1789:target_recovery_overseer()) Skipped 2 previous similar messages
Lustre: 7549:0:(ofd_obd.c:1067:ofd_orphans_destroy()) lustre-OST0000: deleting orphan objects from 14192 to 14221
Lustre: 7549:0:(ofd_obd.c:1067:ofd_orphans_destroy()) Skipped 5 previous similar messages
Lustre: server umount lustre-MDT0000 complete
Lustre: Skipped 3 previous similar messages
LDISKFS-fs (loop0): mounted filesystem with ordered data mode. quota=on. Opts: 
Lustre: Found index 0 for lustre-MDT0000, updating log
Lustre: Skipped 3 previous similar messages
Lustre: lustre-MDT0000: Temporarily refusing client connection from 0@lo
Lustre: Skipped 4 previous similar messages
Lustre: 16437:0:(ldlm_lib.c:2163:target_recovery_init()) RECOVERY: service lustre-MDT0000, 1 recoverable clients, last_transno 47244640652
Lustre: 16437:0:(ldlm_lib.c:2163:target_recovery_init()) Skipped 3 previous similar messages
LustreError: 16475:0:(mdt_handler.c:3208:mdt_recovery()) operation 400 on unconnected MDS from 12345-0@lo
LustreError: 16475:0:(mdt_handler.c:3208:mdt_recovery()) Skipped 3 previous similar messages
Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
Lustre: Skipped 1 previous similar message
Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
Lustre: Skipped 1 previous similar message
LustreError: 13083:0:(ldlm_resource.c:1103:ldlm_resource_get()) lvbo_init failed for resource 14221: rc -2
LustreError: 13083:0:(ldlm_resource.c:1103:ldlm_resource_get()) Skipped 839 previous similar messages
LDISKFS-fs (loop0): mounted filesystem with ordered data mode. quota=on. Opts: 
LustreError: 166-1: MGC192.168.56.4@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
LustreError: Skipped 2 previous similar messages
Lustre: MGC192.168.56.4@tcp: Reactivating import
Lustre: Skipped 5 previous similar messages
Lustre: Modifying parameter lustre-MDT0000-mdtlov.lov.stripesize in log lustre-MDT0000
Lustre: Skipped 14 previous similar messages
Lustre: lustre-MDT0000: used disk, loading
Lustre: Skipped 2 previous similar messages
LustreError: 16697:0:(sec_config.c:1024:sptlrpc_target_local_copy_conf()) missing llog context
LustreError: 16697:0:(sec_config.c:1024:sptlrpc_target_local_copy_conf()) Skipped 2 previous similar messages
Lustre: 16697:0:(mdt_lproc.c:418:lprocfs_wr_identity_upcall()) lustre-MDT0000: identity upcall set to /root/lustre-master/lustre/tests/../utils/l_getidentity
Lustre: 16697:0:(mdt_lproc.c:418:lprocfs_wr_identity_upcall()) Skipped 2 previous similar messages
LustreError: 11-0: an error occurred while communicating with 0@lo. The ldlm_enqueue operation failed with -19
LustreError: Skipped 25 previous similar messages
LDISKFS-fs (loop0): mounted filesystem with ordered data mode. quota=on. Opts: 
Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
Lustre: Skipped 1 previous similar message
Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
Lustre: Skipped 1 previous similar message
Lustre: DEBUG MARKER: SKIP: recovery-small test_52 skipping ALWAYS excluded test 52
Lustre: DEBUG MARKER: == recovery-small test 53: touch: drop rep == 16:36:28 (1350290188)
Lustre: *** cfs_fail_loc=30c, val=2147483648***
LustreError: 16928:0:(ldlm_lib.c:2404:target_send_reply_msg()) @@@ dropping reply  req@ffff880034054050 x1415880068232711/t60129542210(0) o101->4782b42e-2e27-d7f7-10c2-d93bd62d8964@0@lo:0/0 lens 592/632 e 0 to 0 dl 1350290228 ref 1 fl Interpret:/0/0 rc 301/0
LustreError: 16928:0:(ldlm_lib.c:2404:target_send_reply_msg()) Skipped 1 previous similar message
Lustre: lustre-MDT0000: Client 4782b42e-2e27-d7f7-10c2-d93bd62d8964 (at 0@lo) reconnecting
Lustre: Skipped 7 previous similar messages
Lustre: DEBUG MARKER: == recovery-small test 54: back in time == 16:37:29 (1350290249)
Lustre: Mounted lustre-client
Lustre: Failing over lustre-MDT0000
Lustre: Skipped 2 previous similar messages
Lustre: server umount lustre-MDT0000 complete
Lustre: Skipped 2 previous similar messages
LDISKFS-fs (loop0): mounted filesystem with ordered data mode. quota=on. Opts: 
LustreError: 166-1: MGC192.168.56.4@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
LustreError: Skipped 1 previous similar message
Lustre: MGC192.168.56.4@tcp: Reactivating import
Lustre: Skipped 3 previous similar messages
Lustre: Found index 0 for lustre-MDT0000, updating log
Lustre: Skipped 2 previous similar messages
Lustre: Modifying parameter lustre-MDT0000-mdtlov.lov.stripesize in log lustre-MDT0000
Lustre: Skipped 9 previous similar messages
Lustre: lustre-MDT0000: used disk, loading
Lustre: Skipped 1 previous similar message
LustreError: 17353:0:(sec_config.c:1024:sptlrpc_target_local_copy_conf()) missing llog context
LustreError: 17353:0:(sec_config.c:1024:sptlrpc_target_local_copy_conf()) Skipped 1 previous similar message
Lustre: 17353:0:(mdt_lproc.c:418:lprocfs_wr_identity_upcall()) lustre-MDT0000: identity upcall set to /root/lustre-master/lustre/tests/../utils/l_getidentity
Lustre: 17353:0:(mdt_lproc.c:418:lprocfs_wr_identity_upcall()) Skipped 1 previous similar message
Lustre: lustre-MDT0000: Temporarily refusing client connection from 0@lo
Lustre: Skipped 2 previous similar messages
Lustre: 17320:0:(ldlm_lib.c:2163:target_recovery_init()) RECOVERY: service lustre-MDT0000, 2 recoverable clients, last_transno 60129542217
Lustre: 17320:0:(ldlm_lib.c:2163:target_recovery_init()) Skipped 2 previous similar messages
LustreError: 17356:0:(mdt_handler.c:3208:mdt_recovery()) operation 41 on unconnected MDS from 12345-0@lo
LustreError: 17356:0:(mdt_handler.c:3208:mdt_recovery()) Skipped 5 previous similar messages
LustreError: 167-0: lustre-MDT0000-osp-OST0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
LustreError: Skipped 11 previous similar messages
Lustre: 9216:0:(ofd_obd.c:1067:ofd_orphans_destroy()) lustre-OST0000: deleting orphan objects from 15229 to 15249
Lustre: 9216:0:(ofd_obd.c:1067:ofd_orphans_destroy()) Skipped 7 previous similar messages
Lustre: Unmounted lustre-client
Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request == 16:37:55 (1350290275)
LustreError: 17553:0:(ofd_grant.c:250:ofd_grant_space_left()) lustre-OST0000: cli 4782b42e-2e27-d7f7-10c2-d93bd62d8964/ffff880018769800 left 27250688 < tot_grant 35066160 unstable 0 pending 0
LustreError: 17553:0:(ofd_grant.c:250:ofd_grant_space_left()) lustre-OST0000: cli 4782b42e-2e27-d7f7-10c2-d93bd62d8964/ffff880018769800 left 26202112 < tot_grant 34017584 unstable 0 pending 0
Lustre: *** cfs_fail_loc=21d, val=0***
LustreError: 17553:0:(ost_handler.c:1008:ost_brw_write()) Dropping timed-out write from 12345-0@lo because locking object 0x3b93 took 0 seconds (limit was 6).
Lustre: lustre-OST0000: Bulk IO write error with 4782b42e-2e27-d7f7-10c2-d93bd62d8964 (at 0@lo), client will retry: rc -110
Lustre: *** cfs_fail_loc=21d, val=0***
LustreError: 17553:0:(ost_handler.c:1008:ost_brw_write()) Dropping timed-out write from 12345-0@lo because locking object 0x3b93 took 0 seconds (limit was 6).
Lustre: lustre-OST0000: Bulk IO write error with 4782b42e-2e27-d7f7-10c2-d93bd62d8964 (at 0@lo), client will retry: rc -110
LustreError: 13027:0:(ofd_grant.c:250:ofd_grant_space_left()) lustre-OST0000: cli 4782b42e-2e27-d7f7-10c2-d93bd62d8964/ffff880018769800 left 20959232 < tot_grant 28774704 unstable 0 pending 0
LustreError: 13027:0:(ofd_grant.c:250:ofd_grant_space_left()) Skipped 9 previous similar messages
LustreError: 13027:0:(ofd_obd.c:521:ofd_set_info_async()) lustre-OST0000: Unsupported key revimp_update
Lustre: *** cfs_fail_loc=21d, val=0***
Lustre: Skipped 7 previous similar messages
LustreError: 17553:0:(ost_handler.c:1008:ost_brw_write()) Dropping timed-out write from 12345-0@lo because locking object 0x3b93 took 0 seconds (limit was 6).
LustreError: 17553:0:(ost_handler.c:1008:ost_brw_write()) Skipped 7 previous similar messages
Lustre: lustre-OST0000: Bulk IO write error with 4782b42e-2e27-d7f7-10c2-d93bd62d8964 (at 0@lo), client will retry: rc -110
Lustre: Skipped 7 previous similar messages
LustreError: 13027:0:(ofd_grant.c:250:ofd_grant_space_left()) lustre-OST0000: cli 4782b42e-2e27-d7f7-10c2-d93bd62d8964/ffff880018769800 left 20959232 < tot_grant 28774704 unstable 0 pending 0
LustreError: 13027:0:(ofd_obd.c:521:ofd_set_info_async()) lustre-OST0000: Unsupported key revimp_update
Lustre: *** cfs_fail_loc=21d, val=0***
Lustre: Skipped 8 previous similar messages
LustreError: 17553:0:(ost_handler.c:1008:ost_brw_write()) Dropping timed-out write from 12345-0@lo because locking object 0x3b93 took 0 seconds (limit was 6).
LustreError: 17553:0:(ost_handler.c:1008:ost_brw_write()) Skipped 8 previous similar messages
Lustre: lustre-OST0000: Bulk IO write error with 4782b42e-2e27-d7f7-10c2-d93bd62d8964 (at 0@lo), client will retry: rc -110
Lustre: Skipped 8 previous similar messages
LustreError: 13027:0:(ofd_grant.c:250:ofd_grant_space_left()) lustre-OST0000: cli 4782b42e-2e27-d7f7-10c2-d93bd62d8964/ffff880018769800 left 20959232 < tot_grant 28774704 unstable 0 pending 0
LustreError: 13027:0:(ofd_grant.c:250:ofd_grant_space_left()) Skipped 1 previous similar message
LustreError: 13027:0:(ofd_obd.c:521:ofd_set_info_async()) lustre-OST0000: Unsupported key revimp_update
Lustre: *** cfs_fail_loc=21d, val=0***
Lustre: Skipped 8 previous similar messages
LustreError: 17553:0:(ost_handler.c:1008:ost_brw_write()) Dropping timed-out write from 12345-0@lo because locking object 0x3b93 took 0 seconds (limit was 6).
LustreError: 17553:0:(ost_handler.c:1008:ost_brw_write()) Skipped 8 previous similar messages
Lustre: lustre-OST0000: Bulk IO write error with 4782b42e-2e27-d7f7-10c2-d93bd62d8964 (at 0@lo), client will retry: rc -110
Lustre: Skipped 8 previous similar messages
LustreError: 13027:0:(ofd_grant.c:250:ofd_grant_space_left()) lustre-OST0000: cli 4782b42e-2e27-d7f7-10c2-d93bd62d8964/ffff880018769800 left 20959232 < tot_grant 28774704 unstable 0 pending 0
LustreError: 13027:0:(ofd_obd.c:521:ofd_set_info_async()) lustre-OST0000: Unsupported key revimp_update
Lustre: *** cfs_fail_loc=21d, val=0***
Lustre: Skipped 8 previous similar messages
LustreError: 17553:0:(ost_handler.c:1008:ost_brw_write()) Dropping timed-out write from 12345-0@lo because locking object 0x3b93 took 0 seconds (limit was 6).
LustreError: 17553:0:(ost_handler.c:1008:ost_brw_write()) Skipped 8 previous similar messages
Lustre: lustre-OST0000: Bulk IO write error with 4782b42e-2e27-d7f7-10c2-d93bd62d8964 (at 0@lo), client will retry: rc -110
Lustre: Skipped 8 previous similar messages
LustreError: 13027:0:(ofd_grant.c:250:ofd_grant_space_left()) lustre-OST0000: cli 4782b42e-2e27-d7f7-10c2-d93bd62d8964/ffff880018769800 left 20959232 < tot_grant 28774704 unstable 0 pending 0
LustreError: 13027:0:(ofd_grant.c:250:ofd_grant_space_left()) Skipped 1 previous similar message
LustreError: 13027:0:(ofd_obd.c:521:ofd_set_info_async()) lustre-OST0000: Unsupported key revimp_update
LustreError: 13027:0:(ofd_obd.c:521:ofd_set_info_async()) Skipped 1 previous similar message
Lustre: *** cfs_fail_loc=21d, val=0***
Lustre: Skipped 17 previous similar messages
LustreError: 17550:0:(ost_handler.c:1008:ost_brw_write()) Dropping timed-out write from 12345-0@lo because locking object 0x3b93 took 0 seconds (limit was 6).
LustreError: 17550:0:(ost_handler.c:1008:ost_brw_write()) Skipped 17 previous similar messages
Lustre: lustre-OST0000: Bulk IO write error with 4782b42e-2e27-d7f7-10c2-d93bd62d8964 (at 0@lo), client will retry: rc -110
Lustre: Skipped 17 previous similar messages
LustreError: 13027:0:(ofd_grant.c:250:ofd_grant_space_left()) lustre-OST0000: cli 4782b42e-2e27-d7f7-10c2-d93bd62d8964/ffff880018769800 left 20959232 < tot_grant 28774704 unstable 0 pending 0
LustreError: 13027:0:(ofd_grant.c:250:ofd_grant_space_left()) Skipped 3 previous similar messages
LustreError: 13027:0:(ofd_obd.c:521:ofd_set_info_async()) lustre-OST0000: Unsupported key revimp_update
LustreError: 13027:0:(ofd_obd.c:521:ofd_set_info_async()) Skipped 2 previous similar messages
Lustre: *** cfs_fail_loc=21d, val=0***
Lustre: Skipped 26 previous similar messages
LustreError: 17547:0:(ost_handler.c:1008:ost_brw_write()) Dropping timed-out write from 12345-0@lo because locking object 0x3b93 took 0 seconds (limit was 6).
LustreError: 17547:0:(ost_handler.c:1008:ost_brw_write()) Skipped 26 previous similar messages
Lustre: lustre-OST0000: Bulk IO write error with 4782b42e-2e27-d7f7-10c2-d93bd62d8964 (at 0@lo), client will retry: rc -110
Lustre: Skipped 26 previous similar messages
LustreError: 17566:0:(vvp_io.c:1038:vvp_io_commit_write()) Write page 10240 of inode ffff88000060f378 failed -28
LustreError: 17566:0:(vvp_io.c:1038:vvp_io_commit_write()) Write page 10240 of inode ffff88000060f378 failed -28
Lustre: DEBUG MARKER: == recovery-small test 56: do not allow reconnect to busy exports == 16:39:13 (1350290353)
LustreError: 17358:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id 136 sleeping for 40000ms
LustreError: 17358:0:(fail.c:137:__cfs_fail_timeout_set()) cfs_fail_timeout id 136 awake
Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash == 16:39:54 (1350290394)
LustreError: 18290:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id b00 sleeping for 10000ms
LustreError: 18308:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway
LustreError: 18308:0:(ldlm_request.c:1795:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108
Lustre: Unmounted lustre-client
LustreError: 18290:0:(fail.c:137:__cfs_fail_timeout_set()) cfs_fail_timeout id b00 awake
LDISKFS-fs (loop0): mounted filesystem with ordered data mode. quota=on. Opts: 
LustreError: 9216:0:(ldlm_resource.c:1103:ldlm_resource_get()) lvbo_init failed for resource 15186: rc -2
LustreError: 9216:0:(ldlm_resource.c:1103:ldlm_resource_get()) Skipped 1207 previous similar messages
LustreError: 18412:0:(mdt_handler.c:6134:mdt_iocontrol()) Aborting recovery for device lustre-MDT0000
Lustre: Mounted lustre-client
Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing == 16:40:08 (1350290408)
LustreError: 18621:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id 801 sleeping for 20000ms
Lustre: *** cfs_fail_loc=305, val=0***
Lustre: *** cfs_fail_loc=305, val=0***
LustreError: 138-a: lustre-MDT0000: A client on nid 0@lo was evicted due to a lock blocking callback time out: rc -107
LustreError: 18621:0:(fail.c:137:__cfs_fail_timeout_set()) cfs_fail_timeout id 801 awake
LustreError: 18621:0:(file.c:2333:ll_inode_revalidate_fini()) failure -2 inode 144115305919021057
Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction == 16:40:28 (1350290428)
Lustre: Mounted lustre-client
Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request
Lustre: Unmounted lustre-client
LustreError: 138-a: lustre-OST0000: A client on nid 0@lo was evicted due to a lock blocking callback time out: rc -107
Lustre: DEBUG MARKER: SKIP: recovery-small test_60 skipping ALWAYS excluded test 60
Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 == 16:40:28 (1350290428)
LustreError: 18968:0:(osd_handler.c:1040:osd_ro()) *** setting lustre-MDT0000 read-only ***
Turning device loop0 (0x700000) read-only
Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
Removing read-only on unknown block (0x700000)
LDISKFS-fs (loop0): recovery complete
LDISKFS-fs (loop0): mounted filesystem with ordered data mode. quota=on. Opts: 
LustreError: 19089:0:(mdt_handler.c:6134:mdt_iocontrol()) Aborting recovery for device lustre-MDT0000
LustreError: 19089:0:(ldlm_lib.c:2122:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery
Lustre: 19153:0:(ldlm_lib.c:1789:target_recovery_overseer()) recovery is aborted, evict exports in recovery
Lustre: 19153:0:(ldlm_lib.c:1789:target_recovery_overseer()) Skipped 2 previous similar messages
Lustre: lustre-MDT0000: disconnecting 2 stale clients
LustreError: 19187:0:(lmv_obd.c:1191:lmv_statfs()) can't stat MDS #0 (lustre-MDT0000-mdc-ffff88001255f800), error -5
LustreError: 19187:0:(llite_lib.c:1558:ll_statfs_internal()) md_statfs fails: rc = -5
Lustre: DEBUG MARKER: == recovery-small test 100: IR: Make sure normal recovery still works w/o IR == 16:40:31 (1350290431)
LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target)
LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target)
LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target)
LDISKFS-fs (loop1): mounted filesystem with ordered data mode. quota=on. Opts: 
Lustre: 19621:0:(ofd_fs.c:271:ofd_groups_init()) lustre-OST0000: 1 groups initialized
Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect
Lustre: Skipped 1 previous similar message
Lustre: lustre-OST0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted.
Lustre: Skipped 1 previous similar message
Lustre: DEBUG MARKER: == recovery-small test 101: IR: Make sure IR works w/o normal recovery == 16:40:55 (1350290455)
LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target)
LustreError: Skipped 1 previous similar message
LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target)
LDISKFS-fs (loop1): mounted filesystem with ordered data mode. quota=on. Opts: 
Lustre: 19912:0:(ofd_fs.c:271:ofd_groups_init()) lustre-OST0000: 1 groups initialized
Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-90
Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart == 16:41:14 (1350290474)
Lustre: lustre-OST0001: haven't heard from client 7286ee97-312c-8e6e-0c9e-f38aa52a1a79 (at 0@lo) in 47 seconds. I think it's dead, and I am evicting it. exp ffff880004fa3400, cur 1350290475 expire 1350290445 last 1350290428
LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target)
LDISKFS-fs (loop1): mounted filesystem with ordered data mode. quota=on. Opts: 
Lustre: 20205:0:(ofd_fs.c:271:ofd_groups_init()) lustre-OST0000: 1 groups initialized
Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-90
Lustre: lustre-OST0000-osc-ffff88001255f800: Connection restored to lustre-OST0000 (at 0@lo)
Lustre: Skipped 45 previous similar messages
LustreError: 20349:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway
LustreError: 20349:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Skipped 1 previous similar message
LustreError: 20349:0:(ldlm_request.c:1795:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108
LustreError: 20349:0:(ldlm_request.c:1795:ldlm_cli_cancel_list()) Skipped 1 previous similar message
Lustre: Unmounted lustre-client
LDISKFS-fs (loop0): mounted filesystem with ordered data mode. quota=on. Opts: 
LustreError: 20532:0:(osp_dev.c:217:osp_disconnect()) lustre-MDT0000-osp-OST0000: can't disconnect: rc = -107
LustreError: 20532:0:(obd_mount.c:2110:server_put_super()) lustre-OST0000: Fail to disconnect osp-on-ost!
Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: Skipped 47 previous similar messages
LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target)
LustreError: Skipped 1 previous similar message
LDISKFS-fs (loop1): mounted filesystem with ordered data mode. quota=on. Opts: 
Lustre: 20646:0:(ofd_fs.c:271:ofd_groups_init()) lustre-OST0000: 1 groups initialized
Lustre: lustre-OST0000: Denying connection for new client 08124359-1c3b-3dc8-c281-b62a916c4a2f (at 0@lo), waiting for all 1 known clients (0 recovered, 0 in progress, and 1 unseen) to recover in 1:00
Lustre: Mounted lustre-client
Lustre: DEBUG MARKER: == recovery-small test 103: IR: MDS can start w/o MGS and get updated nidtbl later == 16:41:44 (1350290504)
Lustre: DEBUG MARKER: SKIP: recovery-small test_103 mgs and mds on the same target
Lustre: DEBUG MARKER: == recovery-small test 104: IR: ost can disable IR voluntarily == 16:41:44 (1350290504)
LustreError: 20916:0:(ldlm_lib.c:2122:target_stop_recovery_thread()) lustre-OST0000: Aborting recovery
Lustre: 20650:0:(ldlm_lib.c:1789:target_recovery_overseer()) recovery is aborted, evict exports in recovery
Lustre: 20650:0:(ldlm_lib.c:1789:target_recovery_overseer()) Skipped 2 previous similar messages
LDISKFS-fs (loop1): mounted filesystem with ordered data mode. quota=on. Opts: 
Lustre: lustre-OST0000: Denying connection for new client 08124359-1c3b-3dc8-c281-b62a916c4a2f (at 0@lo), waiting for all 1 known clients (0 recovered, 0 in progress, and 1 unseen) to recover in 1:00
Lustre: DEBUG MARKER: == recovery-small test 105: IR: NON IR clients support == 16:41:54 (1350290514)
Lustre: DEBUG MARKER: SKIP: recovery-small test_105 Needs multiple clients
Lustre: DEBUG MARKER: == recovery-small test 106: lightweight connection support == 16:41:54 (1350290514)
Lustre: *** cfs_fail_loc=805, val=0***
Lustre: Mounted lustre-client
LustreError: 21286:0:(osd_handler.c:1040:osd_ro()) *** setting lustre-MDT0000 read-only ***
Turning device loop0 (0x700000) read-only
Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
Removing read-only on unknown block (0x700000)
LDISKFS-fs (loop0): recovery complete
LDISKFS-fs (loop0): mounted filesystem with ordered data mode. quota=on. Opts: 
LustreError: 21510:0:(mdc_locks.c:773:mdc_enqueue()) ldlm_cli_enqueue: -5
Lustre: DEBUG MARKER: recovery-small test_106: @@@@@@ FAIL: lightweight client not evicted by mds
Lustre: Unmounted lustre-client
Lustre: 9216:0:(ofd_obd.c:1067:ofd_orphans_destroy()) lustre-OST0001: deleting orphan objects from 15221 to 15317
Lustre: 9216:0:(ofd_obd.c:1067:ofd_orphans_destroy()) Skipped 11 previous similar messages
